# Streaming a model's answer

When what a route runs returns an `AiStream`, the route sends a model's
answer as it is written, piece by piece. An `AiStream` is the text
`ai.stream` asks a model for ([Streaming](streaming.md)). It is sent
as server-sent events, the HTML standard's `text/event-stream`, which is
the shape OpenAI's and Anthropic's APIs stream in. Jadpo has no WebSockets
([AI](ai.md)).

The action asks for the stream. As with any outside call, it maps each way
the call can fail before its first piece to one of its own failures:

```jadpo
use support/types { Asked }
use support/failures { AssistantUnavailable }
use support/prompts { AnswerQuestion }

/// Streams an answer to a customer's question as it is written.
action stream_answer(asked: Asked) fails AssistantUnavailable -> AiStream {
    return attempt ai.stream(
        AnswerQuestion {
            question: asked.customer_question
        },
        max_output: 600,
        timeout: 8s
    ) {
        refused: AssistantUnavailable
        timed_out: AssistantUnavailable
        over_budget: AssistantUnavailable
        too_long: AssistantUnavailable
    }
}
```

The route names `output: AiStream`, exactly what the action returns:

```jadpo
use support/types { Asked }
use support/actions { stream_answer }

/// Customers' questions.
routes HelpDesk {
    prefix: /help_desk

    /// Streams the assistant's reply as it is written.
    POST / {
        authentication: RouteAuthentication.none
        input: Asked
        output: AiStream
        run: stream_answer(input)
    }
}
```

What a client reads, once the first piece has come:

```text
HTTP/1.1 200 OK
content-type: text/event-stream
cache-control: no-store
x-request-id: req_0b1e…

event: piece
data: {"text":"Our sourdough is "}

event: piece
data: {"text":"baked beside sesame."}

event: done
data: {}

```

A stream that stopped part-way ends like this:

```text
event: piece
data: {"text":"Our sourdough is "}

event: error
data: {"error":{"code":"unavailable","message":"A dependency is temporarily unavailable.","request_id":"req_0b1e…"}}

```

| Event | Data | When |
|---|---|---|
| `piece` | `{"text": "..."}`: the next part of the answer, as the model sent it | As each part comes; none for an empty answer |
| `done` | `{}` | The answer is whole; the last event |
| `error` | The failure envelope ([Failures](failures.md)), with `unavailable`, or `internal_fault` for a defect | The answer stopped before it was whole; the last event |

## Before and after the first piece

- **Before the first piece** nothing has been sent, so the route answers
  as any other does. A failure the action's `attempt` mapped from
  `ai.stream`'s `refused`, `timed_out`, `over_budget` or `too_long`
  answers with its envelope and its kind's status, and the deadline
  passing answers 504. The action has committed by the time the first
  piece goes out.
- **After it**, the 200 has been sent, so whatever stops the answer
  part-way ends the stream with an `error` event, `unavailable`. That
  covers the model declining to go on, a cut-off at `max_output:`, the
  call's `timeout:` or the route's deadline passing, and the provider's
  connection breaking or sending something that isn't a chat completion.
  The app's mapping can't run then, and the client only needs to know the
  answer is incomplete, so the code and its wording are the runtime's own.
- **Every stream ends** with exactly one `done` or `error`, and then the
  response ends.

## The events

- **Each event** is an `event:` line, one `data:` line of compact JSON and
  a blank line, in UTF-8. The JSON escapes the text's newlines, so they
  never split an event. A piece is what the provider sent, never cut or
  joined, and empty ones are left out.
- **No resuming.** A model's answer can't be asked for again from where it
  stopped, so events carry no `id:`, the server sends no `retry:` and
  `Last-Event-ID` isn't read. A client that loses a stream sends the
  request again.
- **Never a `GET`.** Only an action asks a model, so a streaming route is
  never a `GET`. A browser's `EventSource` only sends `GET` and reconnects
  by itself, so it can't call one. A client reads the stream with `fetch`
  and a stream reader, as it reads OpenAI's and Anthropic's.

## Time, and a client that goes away

- **Time.** The stream is held to whichever comes first: the call's
  `timeout:`, counted from when the action asked, or the route's deadline,
  counted from when the request was matched. Past it, the stream ends with
  `error`.
- **Its timeout inside the deadline.** The check that each `timeout:` an
  action reaches leaves room inside its route's deadline
  ([Time limits](time-limits.md)) counts a stream once, since it asks once.
  A stream's timeout is 25 seconds by default, so a streaming route sets
  its own `deadline:`.
- **A client that goes away** stops the stream. The runtime stops reading
  the model's answer and closes its connection to the provider at once, so
  nothing more is spent. What the action committed stays committed. This
  holds on every host: natively, the body is chunked and fed as each event
  is read; on Workers, it is a `ReadableStream` pulled as the client
  reads, and a client that goes away ends the request, which ends the
  model's call with it.

## Status, repeats, OpenAPI and the record

- **It answers 200 and waits for nothing.** A streaming route takes no
  other `status:` and no `settle:` (`route.stream_invalid`).
- **A stream is never kept for a repeat.** An `Idempotency-Key` keeps a
  repeat from running alongside the first until its action has committed.
  A repeat after the stream has started runs again, as for a response that
  starts a session (`keeps: request` in `x-jadpo-idempotency`, see
  [OpenAPI](openapi.md)).
- **OpenAPI** lists the 200 as `text/event-stream`, its schema a string,
  described with its events. Each event's data schema is in
  `x-jadpo-stream` (`events: { piece:, done:, error: }`), since plain
  OpenAPI 3.1 has no field for it. The failures before the first piece are
  listed as any route's are.
- **The API record** writes the route's output as
  `output AiStream (server-sent events)`:

```jadpo
// Generated by Jadpo from the routes of API main; update it with `jadpo api update`, never by hand.
record api 1

POST /help_desk
    authentication none
    input Asked (body)
    output AiStream (server-sent events)
    status 200
    fails assistant_unavailable

GET /openapi.json
    generated openapi
    authentication none
    status 200

input Asked
    customer_question: Text { max_length: 2000 }

failure assistant_unavailable
    status 503
    message "The assistant can't answer now; a person will reply."
```

## In a route test

In a route test, a streaming route's `Response` has `pieces`, the text of
each `piece` event in order, in place of `output`. Its `code` is the
`error` event's code when one ended the stream, and `none` after `done`.
Its status is 200 once the first piece came. The `ai` fake cuts a scripted
text into pieces of a few words, and `partial("...", <condition>)` scripts
a stream that stops part-way ([Testing AI calls](ai-tests.md)):

```jadpo
use support/types { Asked }

test "a reply comes piece by piece" {
    fakes: {
        ai.answer_question: ["Our sourdough is baked beside sesame."]
    }
    var response: Response = request {
        method: HttpMethod.post
        path: "/help_desk"
        input: Asked {
            customer_question: "Does your sourdough contain sesame?"
        }
    }
    assert response.status == 200
    assert response.pieces == ["Our sourdough is ", "baked beside sesame."]
    assert response.code == none
}

test "a failure after the first piece arrives as an error event" {
    fakes: {
        ai.answer_question: [partial("Our sourdough is baked", timed_out)]
    }
    var response: Response = request {
        method: HttpMethod.post
        path: "/help_desk"
        input: Asked {
            customer_question: "Does your sourdough contain sesame?"
        }
    }
    assert response.status == 200
    assert response.pieces == ["Our sourdough is ", "baked"]
    assert response.code == "unavailable"
}

test "a refusal before the first piece is the desk's own failure" {
    fakes: {
        ai.answer_question: [refused]
    }
    var response: Response = request {
        method: HttpMethod.post
        path: "/help_desk"
        input: Asked {
            customer_question: "Help me pick a lock"
        }
    }
    assert response.status == 503
    assert response.code == "assistant_unavailable"
}
```