RAG & Knowledge
Retrieval quality beats model choice
Teams spend weeks comparing models and an afternoon on how documents are prepared. The second decision usually determines whether the answers are any good.
The model is downstream of the retrieval
A language model can only reason over what it is given. If the retrieval step hands it three irrelevant paragraphs, no amount of model capability recovers the answer, and the more capable models will simply produce a more fluent version of the wrong thing.
This is why an assistant that felt impressive in testing degrades once it meets the real document set. Nothing about the model changed. The corpus got messier.
Where retrieval actually goes wrong
Chunking that severs context. Splitting on a fixed character count cuts tables in half and separates a clause from the heading that qualifies it. Chunk on document structure instead.
No source hierarchy. When the current policy and its three superseded versions are all in the index with equal standing, the system will confidently cite the 2019 one. Recency and authority have to be modelled, not hoped for.
Ignoring permissions. If retrieval does not filter by what the asker is allowed to see, the assistant becomes a way around your access controls. This is the failure that ends projects.
Scanned documents nobody checked. A PDF of a photograph of a page contains no text. It will sit in the index contributing nothing, and nobody will notice until someone asks about it.
What to build instead
Define the approved sources and exclude everything else. A smaller, curated corpus outperforms a larger, unmaintained one almost every time.
Enforce permissions at retrieval, so a user cannot reach through the assistant to content they could not open directly. Attribute every answer to its source and make the citation clickable, so a reader can check rather than trust.
Then write an evaluation set from the questions people actually ask, including the ones where the right answer is that the documents do not say.
A cheaper first step
Before any of this, take twenty real questions to whoever currently answers them and record what they reach for. Half the time the answer lives in one document nobody had indexed, and the project shrinks from a platform to a single well-scoped assistant.
Next
Automate the exception, not just the happy path
Most of the cost in a manual process is not the standard case. It is the twenty percent that does not fit, and that is usually the part automation projects leave behind.