Stream LLM responses so your app doesn't look frozen

You hit send, the spinner spins, and the reply takes a few seconds, then a few more. Your user taps the button again or closes the tab, and nothing on the screen tells them whether it worked. Streaming can show useful progress before the complete reply is ready. It does not guarantee a short wait: model startup, buffering and the network still matter.

Streaming means consuming incremental response data instead of waiting for the entire response. Network chunks are not necessarily model tokens or complete events. Streaming can change perceived latency; measure time to first useful text and time to completion separately rather than promising either is unchanged. The hard part is doing it without breaking cancel, errors, and the half-finished output you already showed.

Why waiting for the full reply feels broken

People judge an app by what is on the screen. A blank box that just sits there gives them nothing to react to, so they start guessing. Is it stuck, did my click land, should I press the button again? Every one of those guesses is a chance to lose them.

The full reply might take a few seconds, and that wait is real. What is not real is the silence. When data arrives, streaming lets the person see progress. A delayed first event or a buffering proxy can still leave a wait. Show an honest waiting state until useful text is available.

There is a second win. Once text is on screen, the user can start reading, and can decide they already have enough. That turns a "wait for it" screen into something closer to a conversation.

How the stream actually travels

Many LLM APIs stream their replies as server-sent events, a plain text format the browser already knows how to read. If you have not met it before, the MDN guide to using server-sent events explains what arrives and how to shape it. The core idea is simple: the connection stays open, and the server keeps pushing lines of data until it says it is done.

On the client, you read that connection like any other stream. Call fetch, take the response body, and pull from it with a reader instead of calling .json() on the whole thing. The MDN walkthrough on using readable streams shows the getReader() pattern and the TextDecoder step that turns bytes back into text. Each chunk you read is a small piece of the answer, sometimes mid-word, sometimes mid-sentence.

That last detail matters more than it sounds. What you get are fragments, not tidy objects, so your code has to treat the reply as text that is still being written.

Keep the API key on your server

Your API key is a payment method. If it sits in browser code, anyone can open dev tools, copy it, and spend on your account. For the server-held API-key design described here, keep the model request behind your server. Other architectures need their own credential and authorization design; never embed a reusable secret in public client code.

Instead, three hops: the browser calls your server, your server calls the model with the key, and your server relays the stream back. Your server authenticates the caller, enforces limits, adds the provider credential and relays the supported event format. It must parse or frame events correctly rather than treating each network read as a complete message. Buffer them and you have rebuilt the frozen screen you were trying to kill.

This also gives you one place to set the rules: how long a request may run, what a user is allowed to send, and what you log.

The parts your coding agent usually skips

A coding agent can wire up streaming and stop at the happy path unless you ask for more. These are the pieces that separate a demo from something you would charge money for.

A Stop button. Users need a way out of a long answer. Wire fetch to an AbortController and call abort() when they tap Stop. Handle cancellation as its own UI state. Propagate cancellation through the relay where supported and verify upstream behavior. Aborting the browser request alone does not establish that provider work or billing stopped.

Errors in the middle, not just at the start. On a good connection the stream finishes cleanly. On a phone switching between networks it can die halfway. Your code has to tell the difference between done and dropped, and show something honest instead of a spinner that never ends.

What to save if they leave. If the user closes the tab mid-answer, decide up front what you keep. Choose retention by sensitivity and user expectations. Keeping partial output may help continuity, but do not persist confidential content automatically. Scope any saved text to the authorized session and define expiry.

Do not parse until the end. The fragments are not valid JSON while they stream, because a single object arrives in pieces. For one whole JSON document, accumulate it until successful protocol completion, then parse and validate it. A dropped connection is not completion. A deliberate event-by-event protocol can parse complete framed events as they arrive. If you need structured output rather than plain text, schema-validated retries are their own problem, and this post on getting JSON you can trust from an LLM walks through it.

The prompt to hand your coding agent

Paste this and edit anything in brackets. Wiring up a stream with all these edge cases is a good job for a coding agent running in a harness that can edit files and run your app, and if you want to watch one work, consider a coding-agent workspace, after checking its access and current terms.

Add streaming to the LLM call in this app.

Requirements:
- Keep the API key on the server. The browser must never see it.
- The server relays the stream to the client and does not buffer the full reply.
- The client reads the fetch response body with a reader and appends chunks as they arrive.
- Do not parse the output until the stream ends.

Also handle:
- A Stop button that cancels with AbortController and keeps the text received so far.
- Mid-stream errors: show the partial text plus an honest message, and allow a retry.
- A user who leaves mid-answer: apply the stated session retention policy, with no automatic sensitive persistence.
- Parse complete protocol events, handle split UTF-8 sequences, and distinguish normal completion from truncation.
- Verify whether cancellation reaches the provider; do not promise stopped billing without evidence.

Show me the diff and explain each change in one line.

Test it before you ship

Run these by hand. They take a few minutes and cover the failures users actually hit.

1. Slow network in dev tools: pick a slow throttling preset and confirm text appears
   before the answer finishes, not all at once.
2. Stop mid-answer: does the stream stop without errors in the console?
3. Kill the server mid-answer: does the UI show a clear error, or hang forever?
4. Close and reopen: does partial-output retention match the stated privacy policy?
5. Send a very long answer: does the screen keep scrolling and stay readable?