Jadpo is experimental and not ready for production. These docs follow the compiler's main branch as of 30 Oct 2026. What's built · Install 0.1.0
jadpo
Reference: AI

AI

The conformance suite

jadpo package conform <package> ai checks that a provider package fills ai as the capability says, before the registry lists it.

jadpo package conform <package> ai checks that a provider package fills ai as the capability says, before the registry lists it. By default it runs against the package's recordings, with nothing leaving your machine.

  • Where it runs. In an app, naming a package the project holds as jadpo package show does: its folder's name, its name in code in any case (openai for OpenAi), or with its version. The capability is ai. A capability the package doesn't fill is cli.package_arguments.
  • The suite is an app of its own. A scratch app holds the package as your app installed it, copied into packages/ and locked to its digest there. It is connected as provider, with ai: going through it, and a conformance domain whose actions make each case's call, with a first migration made for jadpo_ai_usage. Each case asks through the program's own ai.generate, ai.stream and ai.embed paths, outside any running app and counted against no limit, as jadpo ai record asks. So a case goes through the mapping, its request, the package driver's HTTP and the package's own types, as an app's call does.
  • The six cases:
    • a typed answer: ConformanceAnswer, a city and the letters in its name, asked in the provider's structured mode and decoded, its schema sent where schema: { at: } says;
    • a refusal: each refused_when: mark read as refused;
    • a cut-off: a long essay asked with max_output: 16, read as over_budget;
    • a stream: its pieces read as sent, then whole;
    • an embedding: one text embedded as a Vector of 256 dimensions, what the suite's embedding_dimensions: declares. It is skipped when the package maps no embed:, as Anthropic's doesn't;
    • usage: the typed answer's tokens read as usage: says.
  • Against the recordings and a local fake, by default. A server on this machine's loopback address, on a thread of its own, answers every request with the answer set for the case, again on every send. Each answer is the package's recording of the operation, its placeholders filled as jadpo dev's fake fills them. It is marked where the mapping says the provider marks it:
    • the typed answer's text where answer: reads it, and the case passes only when that text is what decodes;
    • a refusal at each mark, and a cut-off at cut_off_when:;
    • three pieces where pieces: reads them, and the end where done_when: marks it or data: [DONE];
    • a vector of 256 numbers at vector:, with its index: 0;
    • 12 input, 7 output and 3 cached tokens where usage: reads them (cached 0 when it reads none), which the usage case expects exactly.
  • How the marks are placed. A place named by field names is written by its spelling on the wire, through the package's types. A mark goes only where the recording already has the member, since a recording keeps what the provider sends. A refused_when: { holds: } field may be made, as a provider leaves it out until it refuses. A missing recording, or a member a mark needs that it doesn't have, fails the case (conform.failed), naming jadpo package record.
  • What each run proves. The fake proves the mapping reads the package's own recorded answers as an app's call reads them. The live run proves the provider still sends them. Without --live, nothing leaves the machine, and nothing goes to the connection's address even with its key set.
  • The models. The app's ai: models, default and embedding, when ai: goes through a connection to the package. Otherwise, in the run against the recordings, both are the first model of the package's context table, after the package's folder name (open_ai/gpt-6-astra).
  • --live runs against the provider's API through your app's own connection to the package. It uses the connection's credential and its host: as the app reads them, handed to the suite's app in memory and never written to disk, and the models the app's ai: names. It is refused (conform.refused) when the app has no connection to the package, when nothing sets its credential or address (a stand-in is never sent), when it gives no host:, or when ai: doesn't go through it. So the live run proves what the app will send. A provider can't be made to refuse on request, so the live run skips the refusal, which the run against the recordings checks. Its usage case expects input and output tokens above zero, and its embedding is skipped when ai: names no embedding model. Jadpo's own tests never make the live run; a person runs it with the provider's keys.
  • The screen follows the publishing bar's (jadpo package check): each failed case's conform.failed first, then Conformance suite with a row per case. Each row has its state as a symbol and a word (passed, failed, skipped) and a few words after. Then the footer, ai conforms or ai doesn't conform, with the package, the counts and what it ran against. After a passing run against the recordings, one hint names the --live run. --json gives package, capability, against (recordings or live), host, conforms and each case's case, state and detail. Errors: conform.failed (a case, or the suite, didn't run as the capability says) and conform.refused (the live run's refusals).

Not built yet:

  • Another capability the package fills is toolchain.not_built_yet, since ai's is the only suite.
  • A package with fixed hosts: is toolchain.not_built_yet too. The suite reaches the provider through its connection's host:, which such a package refuses; both first-party packages take host:.
  • A passing run pinned to the package's digest, and the registry's listing, come with the hosted registry. Running the suite for an app's integrations in scheduled CI comes with the CI work.

Markdown for agents: /docs/the-conformance-suite.md.Built from Jadpo main at a707efb.