# The `ai` capability Your code asks a model through `ai`, never through a provider's own operations. So moving to another provider changes `app.jadpo` and nothing else. `ai` is a capability: a job Jadpo does itself and covers whole, with a provider package filling in the part that talks to the provider ([Capabilities](capabilities.md)). Jadpo renders the prompt from a checked file, asks the provider, enforces the budget, decodes the answer into a type and records the call. The provider only turns text into text or numbers. ## Operations | Operation | Gives | Fails (typed, mappable) | Limits | |---|---|---|---| | `ai.generate(prompt, as: T)` | A `T`, validated | `refused`, `malformed`, `timed_out`, `over_budget`, `too_long` | `max_output:` (default 1,000 tokens), `timeout:` (default 25s), `attempts:` (default 2, at most 3) | | `ai.stream(prompt)` | An `AiStream` of text, handed to a streaming route | `refused`, `timed_out`, `over_budget`, `too_long`, before the first piece arrives (text isn't decoded, so never `malformed`); after it, the stream ends with an error event ([Streaming a model's answer](streaming-a-model-s-answer.md)) | As `generate`, `attempts:` always 1 | | `ai.embed(text, as: V)` | A `V`, a vector type with `dimensions:` | `timed_out`, `over_budget`, `too_long` | `timeout:` (default 5s) | | `ai.embed_all(texts, as: V)` | A `List` in the same order | As `embed` | Up to 100 texts per call; the package splits them into the provider's batch size | - **Options.** Each operation takes the named options the table lists: `model:`, `max_output:`, `timeout:` and `attempts:`. `model:` is a name from `ai: { models: }`. Leave it out and generation uses the model called `default`, and embeddings the one called `embedding`. There are no other options: temperature and other sampling settings stay at the provider's defaults. - **Only actions call them,** including a subscriber's and a schedule's, as with every outside call ([Calling](calling.md)). A function or a query can't. A route's deadline must leave room for the call's `timeout:` times its attempts. `ai.generate`'s defaults come to 50 seconds, so a route that asks a model sets its `deadline:` to up to 60s. Longer work runs in a subscriber, which the route either waits for with `settle:` ([Waiting for follow-on work](waiting-for-follow-on-work.md)) or streams. - **Generating changes nothing at the provider,** so it counts as a call that changes nothing. It is retried on a timeout, can go anywhere in an action, and holds no transaction open ([Outside calls hold no transaction](all-or-nothing-actions-and-concurrent-changes.md#outside-calls-hold-no-transaction)). When an action runs again after a concurrent change, it gets the answers from its first run, kept in the same step records that keep a provider's answers ([Several changing calls in one action](several-changing-calls-in-one-action.md)). So a rerun neither spends twice nor answers differently. - **A problem you don't map is a fault**, a failure the runtime raises itself ([Failures](failures.md)): `misconfigured` for a rejected key or an unknown model, and `unavailable` for rate limits and server errors once the connection's retries are spent. These are the ways a call can fail. Your action maps each one a call can raise to a failure of its own. `ai.generate` can raise all five; the table above lists which the others raise. | Failure | When | Retried | |---|---|---| | `refused` | The provider says the model declined (a refusal stop reason or refusal field) | Never | | `malformed` | The answer still doesn't decode into `T` after every attempt; carries the last `DecodeProblem` ([`json`](json.md)) | Up to `attempts:` | | `timed_out` | No complete answer within `timeout:` | Within `timeout:`, by the connection's `retry:` | | `over_budget` | The call would pass a limit (checked before sending), or the answer was cut off at `max_output:` | Never | | `too_long` | The rendered prompt is longer than the model's context, as the package declares it per model | Never | ## Connecting a provider You connect a provider in `app.jadpo`: a connection to its package, and an `ai:` block naming that connection, the models your app uses, its limits and the personal data it may send. Here an app connects to OpenAI, with each model, the key and the daily spend as settings in `config.jadpo`: ```jadpo // config.jadpo config { ai_daily_spend: Money { default: "GBP 20.00" } drafting_model: ModelId { default: "openai/gpt-6-luna" } openai_api_key: ApiKey { secret: true } openai_host: Url { default: "https://api.openai.com/v1" } search_model: ModelId { default: "openai/text-embedding-3-small" } } // app.jadpo application { connections: { openai: { package: OpenAi.v1 domains: [support] credential: config.openai_api_key host: config.openai_host } } ai: { connection: openai models: { default: config.drafting_model embedding: config.search_model } embedding_dimensions: 1536 limits: { per_request: 20000 per_day: { tokens: 2000000 cost: config.ai_daily_spend } per_caller_per_day: 60000 } personal_data: { allowed: [PersonalData.identifying, PersonalData.health] } } } ``` - **Models are settings.** Each entry under `models:` is a `ModelId` setting ([Settings](settings-config-jadpo.md)), so moving to a newer model is a configuration change, reviewed like any other. A `ModelId` written anywhere but a setting's `default:` is `semantic.ai_model_literal`. `default` is required, and so is `embedding` when your app makes embeddings. Any other names, such as `drafting` or `cheap`, are your app's own. - **`embedding_dimensions:`** is how many numbers each embedding has. Jadpo asks the provider for that many, where it can shorten its vectors. Every vector type you pass as `ai.embed`'s `as:` must have the same `dimensions:` (`semantic.ai_dimensions_mismatch`), so a model change that alters the length is caught before a column holds vectors of two lengths. - **The credential** is an ordinary `secret: true` setting. Code never reads it ([Secrets](secrets.md)). - **One `ai` connection per app.** All of an app's models, such as a cheap one and a strong one, are reached through the same provider. ## How a provider fills it You only need this if you write a provider package. A package fills `ai` with a file, `provides/ai.jadpo`, that maps each of `ai`'s operations onto the provider's own operations. The mapping is data, not code, as a package's mapping for email is ([Capabilities](capabilities.md)). Jadpo hands the mapping a request it has already rendered: `request.instructions`, `request.messages` (each a `ChatMessage`), `request.model`, `request.max_output` and, for `generate`, `request.schema`, the JSON Schema of `T`. The field names below follow OpenAI's Chat Completions API. Each package's names are checked against its provider's documentation. ```jadpo // packages/openai/v1/provides/ai.jadpo provide ai { generate: { call: chat.completions.create input: { model: request.model messages: { each: request.messages as: { role: item.role content: item.content } } max_completion_tokens: request.max_output } answer: response.choices.first.message.content refused_when: response.choices.first.message.refusal cut_off_when: { path: "choices.0.finish_reason" equals: "length" } usage: { input: response.usage.prompt_tokens output: response.usage.completion_tokens cached: response.usage.prompt_tokens_details.cached_tokens } schema: { mode: AiSchema.strict_subset at: "response_format" } } stream: { call: chat.completions.stream pieces: event.choices.first.delta.content usage: event.usage } embed: { call: embeddings.create input: { model: request.model input: request.texts dimensions: settings.embedding_dimensions } vectors: response.data vector: item.embedding batch: 100 usage: { input: response.usage.prompt_tokens output: 0 } } models: { context: models.context_windows prices: models.prices } } ``` The example is a sketch. OpenAI's package as shipped names its operations `chat_completions.create`, `chat_completions.stream` and `embeddings.create`, and marks a refusal and a stream as [Streams, embeddings and the OpenAI and Anthropic packages](streams-embeddings-and-the-openai-and-anthropic-packages.md) describes. - **`schema:`** says how much of a JSON Schema the provider's structured mode enforces, and where it goes. Its `mode:` is one of: - `AiSchema.strict_subset`: the provider enforces part of a schema. Jadpo sends that part and checks the rest by decoding. - `AiSchema.json_only`: the provider promises JSON, but not its shape. - `AiSchema.none`: the provider promises nothing. `at:` is where in the request the provider takes the schema. Jadpo writes there the wrapper that mode needs, such as OpenAI's `{"type": "json_schema", ...}`. See [Typed output](typed-output.md). - **`models:`** points at two tables in the package's `libraries/`: each model's context window, and its dated price per million input, cached and output tokens. The context window decides `too_long`, and the prices feed the cost estimate. Your app can override a price with a setting. - **The built-in driver**, `Driver.openai_compatible { url: }`, fills `ai` for any server that speaks OpenAI's API, such as Ollama, llama.cpp's server, vLLM or LM Studio. It is the local driver every capability has, so in development you can generate for real at no cost. It accepts `http://` only for an address on this machine. - **Conformance.** `jadpo package conform ai` runs the capability's tests against the provider: a typed answer, a refusal, a cut-off, a stream, an embedding of the declared length and the usage reported. The registry lists a package as filling a capability only with a passing run. By default the suite runs against the package's recordings and a local fake. With `--live` it runs against the provider's API through your app's connection, and is refused if no credential is set for it ([The conformance suite](the-conformance-suite.md)). ## A model on this machine The quickest way to generate for real is a model running on your own machine, with no key and no cost. The built-in driver speaks OpenAI's Chat Completions API to a server that runs a model, here vLLM. Its `url:` is the address the API is under, as a `Url` setting or written out. Plain HTTP reaches only this machine: ```jadpo application { connections: { local_model: { driver: Driver.openai_compatible url: config.model_url } } ai: { connection: local_model models: { default: config.drafting_model } limits: { per_day: { tokens: 2000000 } } } } ``` ```jadpo config { drafting_model: ModelId { default: "meta-llama/Llama-3.1-8B-Instruct" } model_url: Url { default: "http://localhost:8000/v1" } } ``` An action asks the model with a prompt, and names with `as:` the type the answer is decoded into. It maps each of the five failures to one of its own, each of kind `unavailable`. The failure that `malformed` maps to receives the last `DecodeProblem`, which says what didn't decode: ```jadpo use support/types { CustomerQuestion, SupportAnswer } use support/failures { AnswerUnreadable, AssistantUnavailable } use support/prompts { AnswerQuestion } /// Answers a customer's question, in English for now. action answer_customer( customer_question: CustomerQuestion ) fails AnswerUnreadable, AssistantUnavailable -> SupportAnswer { return attempt ai.generate( AnswerQuestion { question: customer_question reader_locale: Locale.en_gb }, as: SupportAnswer, max_output: 600 ) { refused: AssistantUnavailable malformed: AnswerUnreadable timed_out: AssistantUnavailable over_budget: AssistantUnavailable too_long: AssistantUnavailable } } ``` This is what the server is sent for "Does your sourdough contain sesame?". The schema is cut to what the server's structured mode enforces, as [Typed output](typed-output.md) describes: ```text POST http://localhost:8000/v1/chat/completions {"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "system", "content": "You answer questions from customers of an online bakery. ... Write in\nthe language of en-GB. ..."}, {"role": "user", "content": "\nDoes your sourdough contain sesame?\n"}], "max_tokens": 600, "response_format": {"type": "json_schema", "json_schema": {"name": "answer_question", "strict": true, "schema": {...}}}} ``` In `jadpo dev` each call goes to the server, and `jadpo calls` shows each attempt with what was sent and what came back. In a test, the `ai` fake answers instead ([Testing AI calls](ai-tests.md)).