Consider a hypothetical support assistant for a software company. A customer asks how long a download link will work. The assistant answers “30 days” and cites a help article that says exactly that. The link opens. The quoted sentence is real. For an export created after the policy change, the answer is wrong: a newer policy reduced the lifetime to seven days.
A citation lets someone inspect the source. The product also needs to establish which source governs this question, preserving the relationship between a document, its versions and the circumstances in which each version applies.
The missing condition
Suppose the change took effect on October 1. Exports created before then keep their original 30-day links; exports created on or after October 1 receive seven-day links. The customer created an export on October 4. These dates and rules are invented for this example.
The old help article contains the customer’s wording, including “download link.” A newer release note calls the same thing an “export URL.” A search can find the old article because it matches the question well. The assistant can then faithfully repeat a rule that no longer applies to this export.
This is one way retrieval-augmented generation can fail. Retrieval-augmented generation, usually shortened to RAG, supplies a model with material found through search. Microsoft’s RAG overview describes that pattern and the work involved in finding relevant material. Relevance answers which documents concern the question; the product must also establish which policy applies.
Changing the prompt to “use the latest document” leaves an ambiguity. A recently edited FAQ might describe the old rule for earlier exports. A release note published in September might announce the October policy. Publication date, editing date and effective date answer different questions.
A record of what applies
For this support assistant, a document record needs a stable identity, a version, the period or conditions it governs, and a link to the source. Those details distinguish the two policies. A superseded version can remain available for historical questions while being excluded from answers about new exports.
| Source | Rule | Applies to |
|---|---|---|
| Help article, version 1 | 30-day link | Exports created before October 1 |
| Policy, version 2 | Seven-day link | Exports created on or after October 1 |
| FAQ edited October 3 | Explains earlier exports | The earlier-export exception |
The October 3 FAQ is the newest file in this small collection. Sorting by editing date would put it first, although it does not establish the lifetime of the customer’s October 4 export. Explicit conditions let the application select the relevant policy and retain the exception without asking the model to infer that relationship from scattered prose.
The same distinction matters when one rule applies to a particular plan or product version. If the customer’s plan is unknown and the available rules differ by plan, the answer needs that missing fact or a clear statement of the alternatives. Confidence in the wording cannot resolve the missing plan.
Follow the change into search
A corrected source does not by itself show what the assistant received. Search indexes hold representations of documents, often split into passages. A source update needs to reach those passages and their metadata before the revised rule can appear in an answer.
Microsoft’s Azure Storage indexing guidance illustrates a specific failure to check: its indexers detect changed storage objects, but removal requires a deletion strategy. Deleting a source object can otherwise leave a search document behind. That behavior is specific to the documented indexers; other retrieval systems need their own update and removal checks.
For the export policy, a useful trace connects the source version to the indexed passage and then to the text sent to the model. If the source is correct but search returns version 1 for a new export, the investigation has a concrete target. If search returns version 2 and the assistant still answers “30 days,” the failure occurred after retrieval.
Freshness ranking can help surface recent material. Azure AI Search’s preview freshness feature, for example, boosts newer content while allowing older relevant documents to appear. Its documentation distinguishes that preference from a filter that excludes documents. For this example, a preference for newer documents cannot replace the October 1 applicability rule.
An answer test that crosses the date
The example suggests a small test collection with a useful boundary. One question concerns a September 30 export and expects the 30-day rule. Another concerns an October 4 export and expects seven days. A third omits the creation date and expects the assistant to resolve the ambiguity before giving one lifetime. Each expected answer includes the source version that supports it.
Those cases distinguish a genuine correction from an assistant that now says “seven days” to everything. A further case can present both versions with their conditions removed. If the documents conflict and nothing establishes which applies, a useful response explains the conflict or routes it for clarification. Selecting the newer-looking file would conceal the missing evidence.
For a real retrieval product, choose one recent rule change and follow it through that same path. Check an example on each side of the boundary, the version search returns, and the passage the answer cites. Record the condition that selects the rule alongside the expected response. A correct answer to an old question should remain correct after the new policy arrives.