Streaming a model's answer

When what a route runs returns an AiStream, the route sends a model's answer as it is written, piece by piece. An AiStream is the text ai.stream asks a model for (Streaming). It is sent as server-sent events, the HTML standard's text/event-stream, which is the shape OpenAI's and Anthropic's APIs stream in. Jadpo has no WebSockets (AI).

The action asks for the stream. As with any outside call, it maps each way the call can fail before its first piece to one of its own failures:

use support/types { Asked }use support/failures { AssistantUnavailable }use support/prompts { AnswerQuestion }/// Streams an answer to a customer's question as it is written.action stream_answer(asked: Asked) fails AssistantUnavailable -> AiStream {    return attempt ai.stream(        AnswerQuestion {            question: asked.customer_question        },        max_output: 600,        timeout: 8s    ) {        refused: AssistantUnavailable        timed_out: AssistantUnavailable        over_budget: AssistantUnavailable        too_long: AssistantUnavailable    }}

The route names output: AiStream, exactly what the action returns:

use support/types { Asked }use support/actions { stream_answer }/// Customers' questions.routes HelpDesk {    prefix: /help_desk    /// Streams the assistant's reply as it is written.    POST / {        authentication: RouteAuthentication.none        input: Asked        output: AiStream        run: stream_answer(input)    }}

What a client reads, once the first piece has come:

HTTP/1.1 200 OKcontent-type: text/event-streamcache-control: no-storex-request-id: req_0b1e…event: piecedata: {"text":"Our sourdough is "}event: piecedata: {"text":"baked beside sesame."}event: donedata: {}

A stream that stopped part-way ends like this:

event: piecedata: {"text":"Our sourdough is "}event: errordata: {"error":{"code":"unavailable","message":"A dependency is temporarily unavailable.","request_id":"req_0b1e…"}}

Before and after the first piece

• Before the first piece nothing has been sent, so the route answers as any other does. A failure the action's attempt mapped from ai.stream's refused, timed_out, over_budget or too_long answers with its envelope and its kind's status, and the deadline passing answers 504. The action has committed by the time the first piece goes out.
• After it, the 200 has been sent, so whatever stops the answer part-way ends the stream with an error event, unavailable. That covers the model declining to go on, a cut-off at max_output:, the call's timeout: or the route's deadline passing, and the provider's connection breaking or sending something that isn't a chat completion. The app's mapping can't run then, and the client only needs to know the answer is incomplete, so the code and its wording are the runtime's own.
• Every stream ends with exactly one done or error, and then the response ends.

The events

• Each event is an event: line, one data: line of compact JSON and a blank line, in UTF-8. The JSON escapes the text's newlines, so they never split an event. A piece is what the provider sent, never cut or joined, and empty ones are left out.
• No resuming. A model's answer can't be asked for again from where it stopped, so events carry no id:, the server sends no retry: and Last-Event-ID isn't read. A client that loses a stream sends the request again.
• Never a GET. Only an action asks a model, so a streaming route is never a GET. A browser's EventSource only sends GET and reconnects by itself, so it can't call one. A client reads the stream with fetch and a stream reader, as it reads OpenAI's and Anthropic's.

Time, and a client that goes away

• Time. The stream is held to whichever comes first: the call's timeout:, counted from when the action asked, or the route's deadline, counted from when the request was matched. Past it, the stream ends with error.
• Its timeout inside the deadline. The check that each timeout: an action reaches leaves room inside its route's deadline (Time limits) counts a stream once, since it asks once. A stream's timeout is 25 seconds by default, so a streaming route sets its own deadline:.
• A client that goes away stops the stream. The runtime stops reading the model's answer and closes its connection to the provider at once, so nothing more is spent. What the action committed stays committed. This holds on every host: natively, the body is chunked and fed as each event is read; on Workers, it is a ReadableStream pulled as the client reads, and a client that goes away ends the request, which ends the model's call with it.

Status, repeats, OpenAPI and the record

• It answers 200 and waits for nothing. A streaming route takes no other status: and no settle: (route.stream_invalid).
• A stream is never kept for a repeat. An Idempotency-Key keeps a repeat from running alongside the first until its action has committed. A repeat after the stream has started runs again, as for a response that starts a session (keeps: request in x-jadpo-idempotency, see OpenAPI).
• OpenAPI lists the 200 as text/event-stream, its schema a string, described with its events. Each event's data schema is in x-jadpo-stream (events: { piece:, done:, error: }), since plain OpenAPI 3.1 has no field for it. The failures before the first piece are listed as any route's are.
• The API record writes the route's output as output AiStream (server-sent events):

// Generated by Jadpo from the routes of API main; update it with `jadpo api update`, never by hand.record api 1POST /help_desk    authentication none    input Asked (body)    output AiStream (server-sent events)    status 200    fails assistant_unavailableGET /openapi.json    generated openapi    authentication none    status 200input Asked    customer_question: Text { max_length: 2000 }failure assistant_unavailable    status 503    message "The assistant can't answer now; a person will reply."

In a route test

In a route test, a streaming route's Response has pieces, the text of each piece event in order, in place of output. Its code is the error event's code when one ended the stream, and none after done. Its status is 200 once the first piece came. The ai fake cuts a scripted text into pieces of a few words, and partial("...", <condition>) scripts a stream that stops part-way (Testing AI calls):

use support/types { Asked }test "a reply comes piece by piece" {    fakes: {        ai.answer_question: ["Our sourdough is baked beside sesame."]    }    var response: Response = request {        method: HttpMethod.post        path: "/help_desk"        input: Asked {            customer_question: "Does your sourdough contain sesame?"        }    }    assert response.status == 200    assert response.pieces == ["Our sourdough is ", "baked beside sesame."]    assert response.code == none}test "a failure after the first piece arrives as an error event" {    fakes: {        ai.answer_question: [partial("Our sourdough is baked", timed_out)]    }    var response: Response = request {        method: HttpMethod.post        path: "/help_desk"        input: Asked {            customer_question: "Does your sourdough contain sesame?"        }    }    assert response.status == 200    assert response.pieces == ["Our sourdough is ", "baked"]    assert response.code == "unavailable"}test "a refusal before the first piece is the desk's own failure" {    fakes: {        ai.answer_question: [refused]    }    var response: Response = request {        method: HttpMethod.post        path: "/help_desk"        input: Asked {            customer_question: "Help me pick a lock"        }    }    assert response.status == 503    assert response.code == "assistant_unavailable"}