# The conformance suite

`jadpo package conform <package> ai` checks that a provider package
fills `ai` as the capability says, before the registry lists it. By
default it runs against the package's recordings, with nothing leaving
your machine.

- **Where it runs.** In an app, naming a package the project holds as
  `jadpo package show` does: its folder's name, its name in code in any
  case (`openai` for `OpenAi`), or with its version. The capability is
  `ai`. A capability the package doesn't fill is `cli.package_arguments`.
- **The suite is an app of its own.** A scratch app holds the package as
  your app installed it, copied into `packages/` and locked to its digest
  there. It is connected as `provider`, with `ai:` going through it, and
  a `conformance` domain whose actions make each case's call, with a
  first migration made for `jadpo_ai_usage`. Each case asks through the
  program's own `ai.generate`, `ai.stream` and `ai.embed` paths, outside
  any running app and counted against no limit, as `jadpo ai record`
  asks. So a case goes through the mapping, its request, the package
  driver's HTTP and the package's own types, as an app's call does.
- **The six cases:**
  - a *typed answer*: `ConformanceAnswer`, a city and the letters in its
    name, asked in the provider's structured mode and decoded, its schema
    sent where `schema: { at: }` says;
  - a *refusal*: each `refused_when:` mark read as `refused`;
  - a *cut-off*: a long essay asked with `max_output: 16`, read as
    `over_budget`;
  - a *stream*: its pieces read as sent, then whole;
  - an *embedding*: one text embedded as a `Vector` of 256 dimensions,
    what the suite's `embedding_dimensions:` declares. It is skipped when
    the package maps no `embed:`, as Anthropic's doesn't;
  - *usage*: the typed answer's tokens read as `usage:` says.
- **Against the recordings and a local fake, by default.** A server on
  this machine's loopback address, on a thread of its own, answers every
  request with the answer set for the case, again on every send. Each
  answer is the package's recording of the operation, its placeholders
  filled as `jadpo dev`'s fake fills them. It is marked where the mapping
  says the provider marks it:
  - the typed answer's text where `answer:` reads it, and the case passes
    only when that text is what decodes;
  - a refusal at each mark, and a cut-off at `cut_off_when:`;
  - three pieces where `pieces:` reads them, and the end where
    `done_when:` marks it or `data: [DONE]`;
  - a vector of 256 numbers at `vector:`, with its `index:` 0;
  - 12 input, 7 output and 3 cached tokens where `usage:` reads them
    (cached 0 when it reads none), which the usage case expects exactly.
- **How the marks are placed.** A place named by field names is written
  by its spelling on the wire, through the package's types. A mark goes
  only where the recording already has the member, since a recording
  keeps what the provider sends. A `refused_when: { holds: }` field may be
  made, as a provider leaves it out until it refuses. A missing
  recording, or a member a mark needs that it doesn't have, fails the
  case (`conform.failed`), naming `jadpo package record`.
- **What each run proves.** The fake proves the mapping reads the
  package's own recorded answers as an app's call reads them. The live
  run proves the provider still sends them. Without `--live`, nothing
  leaves the machine, and nothing goes to the connection's address even
  with its key set.
- **The models.** The app's `ai:` models, `default` and `embedding`, when
  `ai:` goes through a connection to the package. Otherwise, in the run
  against the recordings, both are the first model of the package's
  context table, after the package's folder name
  (`open_ai/gpt-6-astra`).
- **`--live`** runs against the provider's API through your app's own
  connection to the package. It uses the connection's credential and its
  `host:` as the app reads them, handed to the suite's app in memory and
  never written to disk, and the models the app's `ai:` names. It is
  refused (`conform.refused`) when the app has no connection to the
  package, when nothing sets its credential or address (a stand-in is
  never sent), when it gives no `host:`, or when `ai:` doesn't go through
  it. So the live run proves what the app will send. A provider can't be
  made to refuse on request, so the live run skips the refusal, which the
  run against the recordings checks. Its usage case expects input and
  output tokens above zero, and its embedding is skipped when `ai:` names
  no `embedding` model. Jadpo's own tests never make the live run; a
  person runs it with the provider's keys.
- **The screen** follows the publishing bar's (`jadpo package check`):
  each failed case's `conform.failed` first, then `Conformance suite`
  with a row per case. Each row has its state as a symbol and a word
  (`passed`, `failed`, `skipped`) and a few words after. Then the footer,
  `ai conforms` or `ai doesn't conform`, with the package, the counts and
  what it ran against. After a passing run against the recordings, one
  hint names the `--live` run. `--json` gives `package`, `capability`, `against` (`recordings` or
  `live`), `host`, `conforms` and each case's `case`, `state` and
  `detail`. Errors: `conform.failed` (a case, or the suite, didn't run as
  the capability says) and `conform.refused` (the live run's refusals).

Not built yet:

- Another capability the package fills is `toolchain.not_built_yet`,
  since `ai`'s is the only suite.
- A package with fixed `hosts:` is `toolchain.not_built_yet` too. The
  suite reaches the provider through its connection's `host:`, which such
  a package refuses; both first-party packages take `host:`.
- A passing run pinned to the package's digest, and the registry's
  listing, come with the hosted registry. Running the suite for an app's
  integrations in scheduled CI comes with the CI work.