How it works
ClauseRAG: precise citations for clause-structured documents
A RAG design for answers that return to complete source clauses without losing scope.
During a recent customer follow-up, just as the meeting was wrapping up, the customer made a request: “Could the Agent's answers link back to the exact clauses in the source document? It would make checking regulatory details much easier.”
The meeting had gone well. I had relaxed a little, thought “that's just RAG,” and said yes.
Back at the office, I realized it was not that simple.
In vertical AI agent applications, clause-structured documents are common. Many of the documents we handle are SOPs, company policies, standards, and regulations. They have numbered chapters, sections, clauses, and subclauses. I will call them clause-structured documents in this article. A broadly correct answer is not enough for these users. They want each requirement in that answer to point back to the right clause, with nothing important missing and little irrelevant material attached.
What are we trying to solve?
We first looked at several widely used chatbots. In the products we tested at the time, general-purpose tools such as ChatGPT and Doubao could read the files and answer the questions well. Their citations, however, were mostly at the file level. They showed which document supported an answer, but did not take the reader back to a specific clause.
That tradeoff makes sense for everyday questions. It is simpler and faster, and most users will not inspect every sentence against the source.
The tradeoff changes when the wording of the source matters. A broad citation creates two problems. The reader still has to search inside it, so the citation has merely handed the retrieval work back to the user. A wide context can also confuse the model when it contains several numbers, similar headings, conditions, or exceptions.
The problem became:
Return only the clauses needed to answer the question completely, while ensuring that every citation leads back to the exact supporting clause in the source document and that the original scope is preserved.
Then we noticed something useful. In an expert-written, well-structured SOP or regulation, a clause is usually small enough for a person to verify. We could treat the clause as the minimum review unit. In this article, a precise citation means finding every complete clause needed by the answer while avoiding unrelated clauses.
We can write the objective as a constrained optimization problem. Let q be the question, R(q) = {r1, …, rm} the requirements that the answer must cover, C(q) the retrieved candidate clauses, and E the final set of cited clauses:
subject to:
In plain language, every requirement in the answer must be supported by the selected clauses, and each citation must lead back to the source. Once that is true, fewer clauses and less unrelated text make the answer cheaper to review.
Is there already a product for this?
Tencent ima and Google NotebookLM can both link citations back to source passages. We uploaded the same documents and asked the same questions that we used for ClauseRAG.
I preferred ima in these tests. Its answers felt more natural and its citations were narrower. On several questions that required multiple pieces of evidence, though, the displayed citations did not cover every part of the answer. NotebookLM usually got the main answer right, but its citation window was much wider and sometimes included neighboring clauses.
We also looked at specialized Agent products such as Harvey and Glean. Their websites showed similar source-indexing features, but we could not sign in to the demos or obtain accounts, so we could not test them.
After that detour, we still had to build it ourselves.
Breaking down RAG
RAG predates ChatGPT. The term appeared in a 2020 NeurIPS paper and became much more common in products after ChatGPT.
The basic idea has stayed the same: retrieve relevant material from an external source before the model answers, then send that material to the model along with the question.
Basic RAG
A basic pipeline splits a document into chunks and converts each chunk into a vector. At query time, it sends the most semantically similar chunks to the model. This is cheap, direct, and easy to build. The problem is that chunk boundaries have no necessary relationship to clause boundaries.
Advanced RAG
Production systems often combine lexical and semantic search. Lexical search, commonly BM25, is good at exact clause numbers, names, and figures that appear in the source. Dense retrieval handles questions that use different words for the same idea. The two result sets are merged and reranked. Microsoft's RAG information retrieval guide presents these as composable stages.
Anthropic's Contextual Retrieval adds a short piece of context to each chunk before indexing it. Its example uses a financial report. A chunk that says only “The company's revenue grew by 3% over the previous quarter” does not identify the company or the quarter. The contextualized version adds ACME, the second quarter of 2023, and the previous quarter's revenue of $314 million before the chunk enters the lexical and semantic indexes.
Hierarchical RAG
Hierarchical RAG does not treat every chunk as a flat peer. It also uses sections and parent-child relationships. PageIndex is a representative example of this direction.
Agentic RAG
Agentic RAG gives the model more control. It may rewrite the question, choose a source, call retrieval tools, and decide whether to keep searching. That helps with multi-hop and dynamic tasks, but the number of calls, latency, and failure states become harder to control.
These approaches can be combined. An Advanced RAG pipeline can use a hierarchical index, and an agent can call both lexical and semantic retrieval.
From broad chunks to precise citations
The most direct way to turn broad chunks into precise citations is to tell the model in the prompt to select the clauses that should be cited. This removes a lot of irrelevant text, but it sometimes drops required clauses and does not reliably preserve complete clause boundaries. The results are in Experiment 1.
So the question was: how could we consistently find every required clause without returning extra ones?
Clause-structured documents already have a useful shape:
Document → Chapter → Section → Clause → Subclause / ItemThe author has already decided the hierarchy, headings, and text. Parent headings constrain scope, while children under the same parent often express parallel conditions. That structure is human-authored prior knowledge.
We built a fixed record for each clause. It stores the exact source text, number, heading path, parent-child relationships, and source position.
When the LLM answers, it selects one of these prebuilt records rather than generating a passage on the fly. We call that record a clause node.
Why not keep searching down the tree?
Once we had a clause tree, our first idea was to search it from the root downward. At each branch, the system would compare the question with the child nodes, keep the highest-scoring branches, and continue to the next level.
At first, the retrieval representation for each node contained only its own heading and text. That works for leaves because the source clause is right there. It is much weaker for a chapter or section node, whose own text may be little more than an introductory sentence while the useful details sit several levels below. A question about one of those details may have low similarity to the parent. If the parent is pruned, every clause below it disappears from the search.
The obvious fix was to put information about the whole subtree into its parent. We generated subtree summaries from the leaves upward. A leaf used its source text; an internal node used summaries already generated for its children.
Subtree summaries improved routing at the upper levels, but recursive compression blurred the differences between neighboring rules. Top-K pruning is also irreversible. Once the traversal takes a wrong turn, the skipped clauses do not get another chance. The results improved, but not enough. See Experiment 2.
If similarity is not accurate enough, the next idea is to let an LLM choose. At each branch, the model reads the question and node content, then decides which branches to expand. This is an LLM Tree Planner. It traverses the same tree as similarity Top-K; only the selection function changes.
The input cost of an LLM Tree Planner grows quickly. Layer-by-layer calls must run in sequence, and the original question is repeated at every level. Longer questions and deeper trees add both tokens and waiting time. We did not choose it as the default. The calculation is in Experiment 3.
This brought us back to the earlier RAG approaches. Hybrid retrieval is already good at finding relevant material across a document, so the tree did not need to take over that job. The tree was better suited to preserving clause boundaries and parent-child relationships, and to adding child clauses when retrieval hit a parent. The LLM no longer needed to navigate the tree. It only had to make one final selection from a much smaller candidate set.
That division of work led to the final design.
The final design
During the offline build, ClauseRAG turns the document's existing structure into a clause tree and records where every clause appears in the source.
At query time, lexical and semantic search retrieve candidate clauses. The tree adds relevant child clauses to the candidate set when needed, but it is no longer used for layer-by-layer routing. One LLM call then selects the minimum set of clauses required for the answer.
The answer can use only those clauses. Citation locations come directly from the offline build.
The full comparison with the RAG baseline is in Experiment 4.
If the tree does not route the search, why keep it?
When coarse retrieval hits a parent such as “the following conditions must be met,” the system follows the tree to add its child clauses to the candidate set. During evidence selection, the LLM can also see the chapter, section, and parent headings for each clause.
The tree therefore stores clause nodes and supports candidate expansion and disambiguation. Layer-by-layer search is only one possible use of a tree.
In technical terms, ClauseRAG is still a form of RAG that uses document hierarchy. Most of the difference appears after retrieval: the system selects clauses already bound to source positions and removes unnecessary ones before answering.
Where ClauseRAG fits
ClauseRAG is designed for clause-structured documents written by experts and organized consistently. It assumes that a clause is already a practical unit for human review. Unstructured prose and one-off, low-risk questions may not justify the extra offline build.
We have not tested ClauseRAG on a large document repository yet.
If you want to see what it feels like when an answer links back to complete clauses, try the ClauseRAG demo.
Experiments
Readers who do not care about the numbers can stop here.
We used two fixed documents: one Chinese safety standard and one English regulation. The evaluation set had 40 questions. Thirty-seven were answerable and three should be rejected. Before running the systems, we marked the 82 clauses that the answers should cite.
Experiment 1: selecting clauses from retrieved chunks
This experiment tested the most direct approach: retrieve broad chunks, then ask an LLM to select the clauses that should be cited.
Clause-level evidence precision is the proportion of returned clauses that belong to the minimum required evidence set.
| Method | Clause-level evidence precision | Questions with complete clause recall |
|---|---|---|
| Return retrieved chunks directly | 2.9% | 36 / 37 |
| Ask an LLM to select clauses from the chunks | 66.4% | 24 / 37 |
The LLM removed most irrelevant text, but it also started dropping required clauses. A prompt alone did not give us both precise and complete citations.
Experiment 2: subtree summaries
The control representation contained the node's heading, its own text, and the headings of its direct children. The experiment added subtree summaries generated from the leaves upward. Both sides used the same Top-K, traversal depth, and candidate limit.
| Metric | Original node representation | With subtree summaries |
|---|---|---|
| Questions with all required clauses found | 18 / 37 | 21 / 37 |
| Recall of required clauses | 59.0% | 73.6% |
| Questions pruned incorrectly at the top level | 15 | 10 |
Across the two documents, we generated 202 summaries. Including truncation repairs, the model made 57 calls. Across all 57 calls, it used 80,440 tokens in total, counting both input and output, and took 167 seconds.
Most of the gain came from the Chinese document, which improved from 9 / 17 to 12 / 17. The English document stayed at 9 / 20, and several questions regressed. Subtree summaries helped upper-level routing but did not fix the underlying problem with irreversible pruning.
Experiment 3: estimating LLM Tree Planner input cost
We did not run a layer-by-layer LLM traversal for this experiment. We calculated the input sizes from the two complete trees with the same tokenizer.
| Path | Average input per question |
|---|---|
| Give the whole tree to the LLM once | about 105,000 tokens |
| Call the LLM at each level and choose every branch correctly | about 43,000 tokens |
| Call the LLM at each level and eventually inspect the whole tree | about 118,000 tokens |
Each layer depends on the previous one, so the calls cannot run in parallel. They also repeat the original question. This table compares input volume only. Actual latency will vary with tree depth, pruning decisions, and model response time.
Experiment 4: final comparison
The baseline was not a bare vector search. It also parsed the documents, combined lexical and semantic retrieval, asked an LLM to select clauses from chunks, and checked citation positions. ClauseRAG used prebuilt clause nodes, added child-clause expansion, and made one final clause selection.
| Metric | RAG baseline | ClauseRAG |
|---|---|---|
| Questions with all required clauses found | 24 / 37 | 37 / 37 |
| Complete answers with complete citations | 23 / 37 | 37 / 37 |
| Citation precision / clause recall | 0.679 / 0.829 | 0.971 / 1.000 |
| Total model usage for 40 questions | 194,489 tokens | 161,157 tokens |
| Mean end-to-end latency | 6.26s | 6.54s |
In the latest test, the two documents took about three minutes and three and a half minutes to build. ClauseRAG is better suited to documents that will be queried repeatedly and whose answers need human verification.