Prompt rewriter
A prompt rewriter is a layer — often a small LLM — that takes the user's raw query and rewrites it into a form that downstream retrieval or generation handles better.
User queries are messy: typos, missing context, ambiguous wording, implicit references to earlier turns. A prompt rewriter normalises them: resolve coreferences from chat history, expand abbreviations, fill in implicit context, decompose multi-part questions. Used heavily in production RAG and search systems in 2026 because retrieval quality is highly sensitive to query phrasing. Implementations: a small LLM with a tight system prompt, a fine-tuned classifier, or a semantic rewriter (HyDE-style hypothetical document generation).
When to use prompt rewriter
- RAG over user queries that miss context.
- Multi-turn chat where queries reference earlier turns.
- Voice agents where ASR adds noise.
Common mistakes
- Rewriter that's stronger than the answer model — adds cost without proportional quality.
- No eval on the rewriter — drift goes unnoticed.
FAQ
What is prompt rewriter?
A prompt rewriter is a layer — often a small LLM — that takes the user's raw query and rewrites it into a form that downstream retrieval or generation handles better.
When should I use prompt rewriter?
RAG over user queries that miss context. Multi-turn chat where queries reference earlier turns. Voice agents where ASR adds noise.
What are the most common mistakes with prompt rewriter?
Rewriter that's stronger than the answer model — adds cost without proportional quality. No eval on the rewriter — drift goes unnoticed.
Related terms
- Retrieval-augmented generation (RAG) — Retrieval-augmented generation (RAG) injects relevant documents into the prompt at query time so the model answers from your data instead of its training memory.
- Semantic routing — Semantic routing classifies an incoming query by meaning — via embedding similarity to predefined route prototypes — and dispatches it to the right model, agent, or sub-system.
- Router LLM — A router LLM is a small fast language model whose only job is to classify or rewrite an incoming request — deciding which downstream model, agent, or tool should handle it.
- AI agent — An AI agent is a system where a language model autonomously plans and executes a sequence of tool calls to accomplish a goal.
Last updated: 2026-06-01. Raw markdown: https://promtable.com/glossary/prompt-rewriter.md.