It was 11pm. The AI assistant had been returning slightly wrong answers for three days and nobody could figure out why. Not wrong enough to obviously break anything. Wrong enough that two engineers had opened tickets saying "I think the AI is getting worse?"
I started where I always start: what actually got retrieved.
Added five lines of logging to dump the retrieval results before they hit the LLM. Ran the same query that had been producing bad answers.
The top result was from a document dated 14 months ago.
The current document, the one with the right information, was ranked fourth.
Similarity scores: 0.89 for the old document, 0.81 for the new one. The old document won because it was written more cleanly and the semantic match was slightly stronger. The model did exactly what it was supposed to do. It used the best matching document. The best matching document was outdated.
Not a model problem. Not a prompt problem. A data problem that looked like a model problem for three days.
The fix was two parts. Metadata filtering so that documents tagged as superseded never enter the retrieval pool. And a freshness signal in the ranking so that when two documents match similarly, the newer one gets a small boost.
# What we added to the retrieval call
results = vectorstore.similarity_search(
query=query,
k=10,
filter={"status": {"$ne": "superseded"}}
)
# Re-rank by blending similarity score with freshness
def freshness_score(doc, max_age_days=365):
age = (datetime.now() - doc.metadata["last_modified"]).days
return max(0, 1 - (age / max_age_days))
def rerank(results):
return sorted(results, key=lambda r: (
0.8 * r[1] + # similarity
0.2 * freshness_score(r[0]) # freshness
), reverse=True)The fix took forty minutes once I understood the actual problem.
The lesson I keep relearning: when an AI system gives bad answers, the instinct is to look at the model or the prompt. Start with the retrieval. Most of the time the model is doing exactly what you told it to do. The question is whether what you told it to do was right.