Most retrieval improvements focus on the index: better embeddings, reranking, hybrid fusion. But there's an entire class of gains available before retrieval runs, by fixing the query itself. Users ask questions in a form that often doesn't match how the answer is written, and that mismatch quietly caps your recall. Query expansion and HyDE are two techniques that close it — and they're some of the highest-leverage, lowest-infrastructure changes you can make.
The vocabulary mismatch problem
Retrieval works by similarity, and similarity assumes the query and the answer live near each other in embedding space. Often they don't:
- A user asks a terse question: "reset password." The answer is a detailed passage: "To restore access to your account, navigate to the account recovery page and follow the identity verification steps..." Short question, long procedural answer — different length, different vocabulary, different distance in embedding space.
- A user asks imprecisely: "the thing where the app logs you out randomly." The doc describes it precisely: "session timeout due to token expiration." No shared words, low similarity.
The passage that answers the question exists. It's just not near the query as the user phrased it. You can keep improving the embedding model, but you're fighting a mismatch that lives in the query, not the index.
Query expansion: ask in more ways
Query expansion reformulates or augments the original query into several variants that cover more of the ways the answer might be phrased, then retrieves for all of them and merges the results.
def expand(query: str) -> list[str]:
return llm(f"""Generate 3 alternative phrasings of this search query
that use different vocabulary but preserve the intent.
Query: {query}""").splitlines()
# "reset password" →
# "how to recover my account access"
# "steps to change a forgotten password"
# "restore login credentials procedure"
You retrieve for each variant and fuse the candidate sets. The effect is a wider net cast in the right directions: if any phrasing lands near the answer passage, you retrieve it. It directly attacks the "the answer uses different words than the question" failure by generating the different words yourself.
HyDE: search with a hypothetical answer
HyDE — Hypothetical Document Embeddings — takes a cleverer angle. Instead of trying to match a question to an answer, it first asks the model to write a hypothetical answer, then searches with that.
def hyde(query: str):
hypothetical = llm(f"Write a short passage that would answer: {query}")
return retrieve(embed(hypothetical), k=10)
The insight: an answer looks like an answer. A generated hypothetical passage — even if some of its facts are wrong — has the shape, length, and vocabulary of a real answer document, so it sits much closer in embedding space to the real answer than the terse question ever did. You're matching answer-to-answer instead of question-to-answer, and answers cluster with answers.
It sounds backwards to search with a possibly-wrong generated passage. It works because you're not using the hypothetical's facts — you're using its form to navigate to the region of embedding space where real answers live. The real retrieved documents supply the actual facts.
The trade-offs
Neither technique is free, and the cost is the same for both: an extra LLM call before retrieval, which adds latency and cost to every query.
| Query expansion | HyDE | |
|---|---|---|
| Extra LLM calls | One (generate variants) | One (generate hypothetical) |
| Best for | Vocabulary/phrasing gaps | Terse queries, question↔answer shape gap |
| Failure mode | Variants drift off-intent | Hypothetical hallucinates a misleading direction |
| Added latency | One generation + N retrievals | One generation + one retrieval |
The failure modes are worth respecting. Query expansion can generate variants that drift from the user's actual intent, pulling in off-topic results. HyDE can generate a hypothetical that's confidently wrong in a way that points retrieval at the wrong neighborhood entirely. Both are most dangerous on queries where the model has weak priors about the domain.
When to use them
These are recall boosters for hard queries, not a tax to put on every query:
- Reach for them when recall is your bottleneck — when you can see the answer is in the corpus but retrieval isn't finding it, and the cause is a query-vs-answer mismatch rather than a ranking problem. (If retrieval finds the right doc but ranks it poorly, that's a reranking problem, not a query problem.)
- Consider applying them selectively. Short or vague queries benefit most; a query that's already detailed and well-phrased gains little and pays the latency cost anyway. Some systems route only low-confidence or short queries through expansion/HyDE.
- Combine, don't replace. These operate before retrieval; reranking operates after. They stack — expand or HyDE to improve recall (get the right passage into the candidate set), then rerank to improve precision (get it to the top). Pre- and post-retrieval upgrades are complementary, not competing.
The mental model: retrieval quality isn't only about a better index. Sometimes the highest-leverage fix is to stop searching with the user's raw question and start searching with something that looks more like the answer. Query expansion and HyDE are two ways to do exactly that — and they cost you an LLM call, not a re-index.