Reference·Reference

Streaming

Server-sent events, how bytes are relayed, and how usage is captured.

A streaming request returns text/event-stream. Each event is a data: line containing a JSON object, terminated by a blank line.

Reading a stream

const res = await fetch(url, { method: 'POST', headers, body });
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  let i;
  while ((i = buffer.indexOf('\n')) >= 0) {
    const line = buffer.slice(0, i).trim();
    buffer = buffer.slice(i + 1);
    if (!line.startsWith('data:')) continue;
    const payload = line.slice(5).trim();
    if (!payload || payload === '[DONE]') continue;
    process(JSON.parse(payload));
  }
}

What the gateway does

• Relays bytes as they arrive; nothing is buffered.
• Never rewrites event payloads.
• Watches each chunk only to read usage figures, after forwarding it.
• Cancels the upstream request when you disconnect, and stops billing at that point.

Where usage appears

OpenAI-shaped streams put usage in a final event when stream_options.include_usage is set. Anthropic reports it on message_start and message_delta. Gemini reports usageMetadata on its last chunk.