Streaming

ai.stream sends a model's text to the client as it is written, rather than waiting for the whole answer, as a chat app does. The action returns the stream to its route, which sends it as server-sent events (Streaming a model's answer).

• ai.stream(prompt) takes the prompt and model:, max_output: and timeout:, written out and checked as ai.generate's are. as: and attempts: are type.named_argument_unknown, with the reason. Only an action calls it: in a function it is effect.function_persistence, and in a test semantic.not_callable. It goes through the ai: connection (ai.connection_required) and gives an AiStream, a built-in type with no fields and no JSON form, which an action returns to its route. Through a package's connection, it goes through the package's stream: (Streams, embeddings and the OpenAI and Anthropic packages).

• Its four conditions each need an arm naming a failure of kind unavailable, as generate's do:

  • refused: a refusal, or a content_filter finish, before any text;
  • timed_out: no first piece within timeout:;
  • over_budget: a length finish before any text;
  • too_long: the server's context_length_exceeded.

  Text isn't decoded, so a stream is never malformed. A server that can't be reached is sent the request again, as generate's is, until the answer starts to come; then it is unavailable.

• The request is generate's, with "stream": true and no response_format. It asks for OpenAI's streamed chat completion: server-sent events of chat.completion.chunks, each piece a choices.0.delta.content, ended by data: [DONE]. A finish of stop with no [DONE] is whole. A body that ends without either is broken, and so is an error event. A server that ignores stream and answers one whole chat completion is read as one piece. Usage chunks are skipped, since a stream is charged when it is opened (How budgets are counted).

• After the first piece, a refusal, a cut-off, the time running out or a broken connection ends the route's stream with an error event. It is unavailable, the runtime's own failure, since the action has returned and its mapping can't run.

• Answers reused on a rerun. A stream takes the invocation's next outside-call step. An action run again after a concurrent change is given back the stream its first run opened, which nothing has read from yet, for the same request. A stream is never a step of a record (Several changing calls in one action): its answer goes to the client, not the record, and only a route's action returns one.

• The fake answers a streamed prompt from its script, in the built-in driver's own form, with one of:

  • a text (a value of Text, or raw("...")), cut into pieces of three words, each keeping the space after it, then stop and [DONE];
  • a condition, before the first piece;
  • partial("...", <condition>): the text's pieces, then refused (a content_filter finish), over_budget (a length finish) or timed_out (the stream stops), each an error event at the route.

  partial(...) of a prompt no action streams, or with too_long, which is found before anything is sent, is semantic.fake_invalid. A prompt both generated and streamed takes the five conditions, raw(...) and partial(...), just as a prompt asked for as two types takes no value. An unscripted stream gives a generated text naming its prompt.

• jadpo dev streams from the model's server as it answers, each piece reaching the client as it is read. The call log keeps the request in full, as generate's, and not the streamed answer.