The difference between a RAG chatbot that feels sluggish and one that feels responsive is almost entirely streaming. An answer that pops in fully-formed after a three-second silence reads as slow and slightly broken, even if it's fast in absolute terms. The same answer streamed word-by-word, appearing as it's generated, feels immediate and alive — because the user sees progress within a few hundred milliseconds. Server-sent events (SSE) are the clean, standard way to build this. Here's how, and where retrieval fits in the streaming flow.
Why streaming, and why SSE
LLMs generate text token by token. Without streaming, your server waits for the entire generation to finish before sending anything to the browser — so the user waits for the full answer with no feedback. With streaming, you forward each token as it's produced, and the answer materializes in real time. The perceived latency drops dramatically because what matters to users is time to first token, not time to last.
Why SSE specifically? For this job it's the right tool:
- It's one-directional server→client, which is exactly the shape of streaming an answer (the server pushes tokens; the client just listens). You don't need the bidirectional complexity of WebSockets.
- It's built on plain HTTP, so it works through standard infrastructure, proxies, and load balancers without special handling.
- It has native browser support via
EventSource, and it auto-reconnects.
For streaming AI responses, SSE hits the sweet spot: simpler than WebSockets, purpose-built for server-push text.
The flow: retrieve first, then stream
The crucial architectural detail is what streams. In a RAG answer, retrieval happens before generation and is not streamed — you run search, get your passages, and only then start the model. So the flow is:
- Retrieve (not streamed). Run hybrid search over the knowledge base for the question. This is a quick request/response; the user might see a brief "searching..." state.
- Start generation with the retrieved context. Build the grounded prompt (passages + question + grounding instructions) and call the model with streaming enabled.
- Stream tokens over SSE. As the model emits tokens, forward each to the client over the SSE connection.
- Signal completion, and optionally send the sources at the end so the UI can render citations.
Retrieval is a discrete step; generation is the streamed step. Keeping this clear avoids the confusion of "how do I stream search results?" — you don't; you stream the answer built from them.
Server side
A server-side route runs retrieval, then streams the model's output as SSE events. In a Next.js-style server route:
export async function POST(req) {
const { question } = await req.json()
// 1. Retrieve (not streamed)
const passages = await hybridSearch(question) // calls the KB search API
const context = passages
.map(p => `[Source: ${p.source}]\n${p.content}`)
.join("\n\n")
// 2. Stream the model's grounded generation as SSE
const stream = new ReadableStream({
async start(controller) {
const enc = new TextEncoder()
const modelStream = await llm.stream({
system: "Answer only from the provided sources; cite each claim; " +
"if the sources don't cover it, say so.",
messages: [{ role: "user",
content: `Sources:\n${context}\n\nQuestion: ${question}` }],
})
for await (const token of modelStream) {
controller.enqueue(enc.encode(`data: ${JSON.stringify({ token })}\n\n`))
}
// send sources, then a done signal
controller.enqueue(enc.encode(`data: ${JSON.stringify({ sources: passages })}\n\n`))
controller.enqueue(enc.encode(`data: [DONE]\n\n`))
controller.close()
},
})
return new Response(stream, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
},
})
}
The text/event-stream content type and the data: ...\n\n framing are the SSE protocol. Each token goes out as its own event the moment it's generated.
Client side
The browser consumes the stream and appends tokens as they arrive:
const res = await fetch("/api/answer", {
method: "POST",
body: JSON.stringify({ question }),
})
const reader = res.body.getReader()
const decoder = new TextDecoder()
let answer = ""
while (true) {
const { value, done } = await reader.read()
if (done) break
for (const line of decoder.decode(value).split("\n\n")) {
if (!line.startsWith("data: ")) continue
const payload = line.slice(6)
if (payload === "[DONE]") break
const data = JSON.parse(payload)
if (data.token) { answer += data.token; render(answer) } // append & repaint
if (data.sources) renderCitations(data.sources)
}
}
Each token appends to the answer and repaints, so the user watches it type out. When the sources arrive at the end, you render the citations.
Details that matter in production
- Show a retrieval state. Since retrieval precedes streaming, a brief "searching..." indicator covers that gap so the user isn't staring at nothing before the first token.
- Stream sources at the end, not the start. You want the answer to begin appearing immediately; citations can render once generation completes.
- Handle disconnects. SSE auto-reconnects, but a dropped stream mid-answer needs graceful handling so the user isn't left with a half-sentence and no recovery.
- Flush promptly. Make sure your server and any proxies don't buffer the stream — buffering defeats the entire purpose by holding tokens until the end.
The takeaway
Streaming is the cheapest, highest-impact upgrade to a RAG answer UI: the answer is exactly as good, but it feels dramatically faster because users see it building in real time. SSE is the right transport — HTTP-native, one-directional, browser-supported — and the key mental model is that retrieval is a discrete pre-step and generation is what you stream. Run search, build the grounded prompt, then push tokens as the model produces them. A few hundred milliseconds to first token instead of three seconds to full answer is the whole difference between a bot that feels responsive and one that feels broken.