The retrieval pipeline behind an AI agent platform's knowledge base

An AI agent platform's core promise is that agents answer from the customer's own uploaded documents, with a citation on every claim. That promise is kept or broken in the retrieval pipeline — and customers upload native PDFs, scans, and exports with tables and nested clauses, not clean text.

We were building one subsystem inside a live multi-tenant platform, to the client's interfaces, while their internal team continued on the workflow engine. Citations had to resolve to an exact source span, which constrains every upstream stage of the pipeline.

We worked backwards from the citation guarantee. Provenance — document, page, offset — is attached at extraction and never dropped. Chunking follows document structure rather than a fixed token count, with tables extracted as coherent units. Hybrid lexical and semantic retrieval, and a structural abstain path so weak grounding returns an honest non-answer instead of passing thin context to the model.

A production ingestion and retrieval pipeline against the client's interfaces, citations resolving to document-page-span, a calibrated abstention path, and an evaluation set that scores citation accuracy separately from answer accuracy.

Our first chunking pass was length-based. Aggregate retrieval scores looked fine, but citations on table and clause content landed on fragments that didn't support the claim — and tables carry most of the precise content customers care about. Average-case metrics hid a failure concentrated exactly where the product's promise matters. We rebuilt chunking to be structure-aware, which cost about a week.