Query Rewriting: Letting the Agent Improve Its Own Search Terms
A pattern where the model rewrites a bad query based on the first round of weak results.
How it works
Rather than treating the first search query as final, a query-rewriting loop evaluates whether the returned results actually look relevant (via the model's own judgment, or a simple heuristic like result count or snippet keyword overlap) and, if not, has the model generate a revised query — often broader, narrower, or using different terminology — before trying again, up to a small retry cap.
Example
First query AI search tool returns generic, unhelpful results. The agent, evaluating the snippets as too broad, rewrites to LLM function calling search API tool schema — a more specific reformulation informed by what the first round's weak results revealed was missing — and the second call returns materially more relevant hits.
Pitfalls
- Without a retry cap, a query that's fundamentally unanswerable via search (an opinion question, a request for information that doesn't exist online) can loop through rewrites indefinitely without ever succeeding.
- A rewritten query that overcorrects — going from too broad to overly narrow — can produce zero results instead of improving relevance, so rewriting needs to be a considered adjustment, not a random variation.
- Evaluating 'relevance' purely by result count is a weak heuristic — five irrelevant results and five highly relevant ones look identical by that measure alone, so a better implementation checks actual snippet content.