Every organisation has knowledge trapped in PDFs, SharePoint folders and email threads. Retrieval-augmented generation (RAG) is the most practical way to make it usable: when someone asks a question, the system first finds the relevant passages in your documents, then asks a language model to answer using only those passages.
How a RAG system works
- Ingest: collect documents from their sources and convert them to clean text, keeping headings and tables.
- Chunk: split documents into passages that each make sense on their own.
- Index: store passages with embeddings (for meaning) and keywords (for exact terms like part numbers).
- Retrieve: for each question, fetch the most relevant passages and re-rank them.
- Generate: the model answers from those passages and cites them.
The five mistakes behind bad answers
1. Poor document conversion
Scanned PDFs, multi-column layouts and tables often turn into scrambled text. If the text is wrong, every later step is wrong. Invest in proper extraction and spot-check the output.
2. Chunks that lose context
A passage that says “the limit is ₹50,000” is useless without knowing which policy it belongs to. Keep the document title and section heading with every chunk.
3. Meaning-only search
Embeddings are great at meaning but weak at exact codes, names and numbers. Hybrid search (embeddings plus keyword) with a re-ranker fixes most “it couldn't find it” complaints.
4. No permissions
If HR salary documents are indexed, the assistant will happily quote them to anyone. Carry access rights from the source system into retrieval so people only get answers from documents they could already open.
5. No evaluation
Without a fixed test set, every change is a guess. Collect real questions with known good answers and score each release against them.
Choosing where it runs
Hosted model APIs give the best quality with the least effort. Where data must stay in your environment, open-weight models running on your own cloud or servers are now good enough for many document Q&A tasks. Both can meet DPDP requirements if designed for it.
Keeping it up to date
Documents change: a policy is revised, a product manual gets a new version, a price list is replaced. Build incremental sync from day one so new and changed files are re-indexed automatically and deleted files disappear from answers. Show the document date beside every citation so people can judge how current an answer is.
Measuring whether it works
| Measure | How to check it |
|---|---|
| Answer accuracy | Score answers against your test set of real questions |
| Retrieval quality | Was the right passage among the top results? |
| Groundedness | Is every claim in the answer supported by a cited passage? |
| Usage and satisfaction | Thumbs up/down, repeat users, questions per week |
| Deflection | Fewer tickets or emails to HR, IT or support on covered topics |
Review the thumbs-down answers weekly. They show you which documents are missing, unclear or contradictory, which is valuable even before the assistant is fixed.
A trustworthy assistant cites its source and admits when it doesn't know. Users forgive “I couldn't find that”; they don't forgive a confident wrong answer.
