LLM Streaming SSE Implementation: The Complete Guide

Why LLM Streaming SSE Is Harder Than It Looks

Anyone who has used a modern LLM chat interface knows the difference streaming makes. Tokens appear one by one instead of making you stare at a blank box for eight seconds. That feeling is real — and so is the production complexity behind it.

Most teams hit the same three problems when they move streaming to production: SSE connections that drop unexpectedly, tokens that arrive out of order or get lost entirely, and a frontend that visually tears as it tries to keep up. Entry-level tutorials show you how to open a stream. They almost never explain what to do when it breaks.

This guide walks through the full chain — SSE protocol basics, backend setup, reconnection strategy, and incremental rendering — with concrete, production-ready approaches at each step.

---

SSE Protocol Basics and Backend Setup

Server-Sent Events is an HTML5 standard that runs over plain HTTP. The server pushes data; the client only reads. Compared to WebSockets, SSE is lighter, works naturally with HTTP/2 multiplexing, and punches through corporate firewalls more reliably. OpenAI-compatible streaming responses follow the standard SSE format:

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: [DONE]

Each message starts with data: and events are separated by \n\n. [DONE] is an OpenAI convention, not part of the SSE spec — parse it separately.

Two things will silently break your stream if you miss them.

Disable buffering. Nginx buffers responses by default and will hold your tokens until it has enough data to send in one batch. Add this to your location block:

proxy_buffering off;
proxy_cache off;
X-Accel-Buffering: no;

In Node.js, call res.flushHeaders() to send response headers immediately, then use res.write() for each chunk. Don't wait for res.end().

Set the right Content-Type. The response header must be Content-Type: text/event-stream; charset=utf-8. Without it, the browser skips the EventSource parsing path entirely.

On the client side, fetch with ReadableStream is more flexible than the native EventSource API. EventSource doesn't support POST or custom headers, which rules out bearer token auth:

const response = await fetch('/api/chat', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${token}`
  },
  body: JSON.stringify({ messages, stream: true }),
});

const reader = response.body!.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const chunk = decoder.decode(value, { stream: true });
  // parse SSE lines here
}

---

Reconnection: The Edge Case You Can't Skip in Production

Networks are unreliable, and LLM inference is slow. A complex Gemini CLI task can run for 30 seconds, and the probability of a dropped connection over that window isn't negligible.

SSE has a built-in reconnect mechanism: the server can include id: <event-id> fields, and the browser will send Last-Event-ID on reconnect. The problem is that native EventSource reconnection doesn't know how many tokens the client already received. If the server doesn't deduplicate, the user sees repeated content.

The reliable approach is to own the reconnect logic at the application layer:

1. Generate a stream_id for each streaming response. Include a seq counter (starting at 0) on every SSE event.
2. The client tracks the last received seq. On reconnect, it sends both stream_id and last_seq to the server.
3. The server reads the cached token sequence from Redis (TTL of five minutes works well) and resumes from last_seq + 1.

Use exponential backoff with jitter for retry delays so a wave of reconnecting clients doesn't pile onto the backend at once:

const delay = Math.min(1000 * 2 ** retryCount + Math.random() * 1000, 30000);

Cap retries at five. After that, surface an error to the user — don't leave them watching a spinner indefinitely.

One more thing that gets overlooked: load balancer idle timeouts. AWS ALB defaults to 60 seconds, which is too short for long inference jobs. Either raise it to 300 seconds, or have the server emit a keepalive comment every 15 seconds:

: ping

Lines starting with : are comments in the SSE spec — clients ignore them — but they reset the TCP idle timer and keep the connection alive through the load balancer.

---

Frontend Rendering: From Token Stream to Readable Text

Tokens arrive one at a time, but you cannot trigger a React re-render for each one. At 100 tokens per second, calling setState on every token will saturate the main thread. The fix is to batch updates, flushing the UI once per animation frame (roughly 16ms at 60fps):

let buffer = '';
let rafId: number;

function onToken(token: string) {
  buffer += token;
  cancelAnimationFrame(rafId);
  rafId = requestAnimationFrame(() => {
    setContent(prev => prev + buffer);
    buffer = '';
  });
}

Markdown rendering during streaming is its own problem. Passing incomplete text to marked or react-markdown on every frame causes constant DOM reconstruction and visible flicker in code blocks. Two approaches depending on your use case:

• Defer Markdown rendering: show plain text during streaming, convert to Markdown once the stream closes. Simple and effective for shorter responses.
• Incremental Markdown: use marked's streaming lexer (built into v9+) to render only completed blocks (paragraphs, full code blocks) while keeping the incomplete trailing text as plain.

Code blocks need special handling either way. When you see an opening ``` but haven't received the closing one yet, wrap the raw text in a <pre> tag. Once the closing delimiter arrives, run syntax highlighting (highlight.js or shiki). Don't call hljs.highlightAuto() on every incoming token — the cost adds up fast.

The streaming cursor is a CSS job, not a JavaScript one:

.streaming-content::after {
  content: '▋';
  animation: blink 1s step-end infinite;
}
@keyframes blink { 50% { opacity: 0; } }

Remove the .streaming-content class when the stream ends and the cursor disappears automatically. No cleanup timer needed.

---

Quick Reference

Scenario	Recommended approach	Watch out for
Basic chat UI	fetch + ReadableStream	Nginx proxy_buffering
Resume after disconnect	App-layer seq + Redis cache	TTL ≥ max inference time
High-frequency token render	rAF batching at 16ms	Per-token setState
Streaming Markdown	marked v9+ stream lexer	Keep incomplete code blocks as plain text
Load balancer keepalive	SSE : ping every 15s	ALB idle timeout → 300s

---

Once this pipeline is solid, the next natural challenge is switching between models — GPT, Claude, DeepSeek — without rewriting your streaming integration every time. I use XycAi for that: one OpenAI-compatible API that covers 200+ models, with Claude Code, Codex, and Gemini CLI all connecting via a single config change. The pricing starts at 14% of list price, which makes multi-model testing genuinely affordable rather than something you budget carefully.