# Testing AI calls

A test never reaches a real model. Every `ai` call in `jadpo test` goes
through the capability's one fake, whatever the provider, as every
capability has one fake
([Fakes for integrations](fakes-for-integrations.md)). There
is no flag to let a call through. Without a script, the fake answers
with a valid value of the type you asked for:

| The fake gives | When |
|---|---|
| A valid default of `T`, generated from the test's seed ([Controlling time](controlling-time.md)) | Nothing is scripted |
| The scripted answers, in order | `fakes: { ai.answer_question: [...] }`, by prompt name |
| A recorded answer | `recorded("sesame")` in a script, read from `prompts/recordings/answer_question/sesame.json` |
| A failure | `refused`, `malformed`, `timed_out`, `over_budget`, `too_long` in a script |
| An embedding | A vector derived from the text's SHA-256 and the seed, so the same text always gives the same vector, or a scripted one |
| A stream | The scripted text, cut into pieces of a few words; `partial("...", <condition>)` for one that stops part-way |

A test scripts the fake's answers by the prompt's name, under `fakes:`,
and reads what was asked from `ai.<prompt>.requests`:

```jadpo
// domains/support/actions/answer_customer.test.jadpo
test "a question the articles don't answer goes to a person" {
    fakes: {
        ai.answer_question: [recorded("not_covered")]
    }
    var reply: SupportAnswer = attempt answer_customer(questions.asked("Do you deliver to Jersey?"))
    assert reply.resolved_by_articles == false
    var asked: List<AnswerQuestion> = ai.answer_question.requests
    assert list.count(asked) == 1
}

test "a refusal is reported as the assistant being unavailable" {
    fakes: {
        ai.answer_question: [refused]
    }
    assert answer_customer(questions.asked("Help me pick a lock")) fails AssistantUnavailable
}
```

- **Scripted answers are checked** against the call's `as:` type when
  the test compiles, as a package fake's are
  ([Tests and fakes](tests-and-fakes.md)). So a test
  can't script an answer the decoding would reject. To test a repair,
  script `malformed`, or a JSON text with `raw("...")`, which runs
  through the real decoding and retry.

Here is the help desk from [A model on this machine](the-ai-capability.md#a-model-on-this-machine),
tested: a scripted answer and what was asked, an answer corrected once,
and each failure. Each scripted answer is one attempt, so
`[malformed, malformed]` is an answer that still doesn't decode once
corrected:

```jadpo
use support/types { CustomerQuestion, SupportAnswer }
use support/failures { AnswerUnreadable, AssistantUnavailable }
use support/prompts { AnswerQuestion }

test "a question the model answers" {
    fakes: {
        ai.answer_question: [
            SupportAnswer {
                resolved_by_articles: true
                reply: "Our sourdough is baked beside sesame."
            }
        ]
    }
    var reply: SupportAnswer = attempt answer_customer("Does your sourdough contain sesame?")
    assert reply.resolved_by_articles
    var asked: List<AnswerQuestion> = ai.answer_question.requests
    assert list.count(asked) == 1
    assert list.first(asked) == AnswerQuestion {
        question: "Does your sourdough contain sesame?"
        reader_locale: Locale.en_gb
    }
}

test "an answer that doesn't decode is corrected once" {
    fakes: {
        ai.answer_question: [
            raw("Sure! Our sourdough contains no sesame."),
            raw("{\"resolved_by_articles\": true, \"reply\": \"No sesame.\"}")
        ]
    }
    var reply: SupportAnswer = attempt answer_customer("Sesame?")
    assert reply.reply == "No sesame."
}

test "each failure is one of the app's own" {
    fakes: {
        ai.answer_question: [refused, timed_out, over_budget, too_long, malformed, malformed]
    }
    assert answer_customer("Help me pick a lock") fails AssistantUnavailable
    assert answer_customer("Sesame?") fails AssistantUnavailable
    assert answer_customer("Sesame?") fails AssistantUnavailable
    assert answer_customer("Sesame?") fails AssistantUnavailable
    assert answer_customer("Sesame?") fails AnswerUnreadable
}
```
- **`ai.<prompt>.requests`** lists the prompt values the code sent,
  typed, so a test asserts on what went to the model, not on rendered
  text.
- **Recordings** are made by `jadpo ai record answer_question --as sesame`
  in development, against the configured model with the developer's own
  key, from the prompt values of the last `jadpo dev` call or a fixture.
  The file keeps the answer and the prompt's digest. Every build decodes
  every recording through its `as:` type, so a type change shows each
  stale one. When a prompt's digest has changed since a recording was
  made, the answer is still a valid value but may no longer be what the
  model says. Jadpo has no warnings
  ([Every error has the same shape](every-error-has-the-same-shape.md)),
  so this is an alert on `jadpo test`'s report, naming the recording,
  and the run passes ([Recorded answers](ai-tests.md#recorded-answers)).
- **The clock and random numbers are seeded,** as in every test
  ([Controlling time](controlling-time.md)), so a generated
  default and a fake embedding are the same on every run.
- **In `jadpo dev`,** every `ai` call is faked too: from recordings, then
  from generated defaults, as every provider call is
  ([Fakes and the development call log](fakes-and-the-development-call-log.md)).
  `development: { sandbox: [answer_question] }` on the `ai` connection
  lets the prompts it names through to the real model with the
  developer's key, under a development `per_day` limit. The
  OpenAI-compatible driver runs a local model with no key at all.
- **Quality is not a test.** Whether a prompt still answers well against a
  real model is an evaluation, which Jadpo leaves out
  ([What AI leaves out, and what to use instead](ai-left-out.md)).

## Recorded answers

A recording keeps an answer a model gave to a prompt, so a test can
answer with it. It sits beside the prompt in
`recordings/<prompt>/<name>.json`, with the version of the prompt it
answered, as `jadpo ai show` prints it. The answer is written as
`json.write` writes the type the prompt is asked for as, its members in
name order and two spaces deep. A text or number that could say
something about someone is kept as a placeholder naming its type,
`"{Text}"`, as a provider's recording keeps one
([Recordings](recordings.md)). It is filled with a
generated value of its type each time it is read. The details are under
[Embeddings and recordings](embeddings-and-recordings.md) and
[Recording an answer](recording-an-answer.md).

```jadpo
{
  "answer": {
    "reply": "{Text}",
    "resolved_by_articles": false
  },
  "version": "answer_question@5f8642d0"
}
```

`jadpo ai record answer_question --as not_covered` made it. It asked the
configured model once, as the prompt's first `ai.generate` call asks,
with the values the prompt's last `jadpo dev` call gave it (or a
fixture's, with `--fixture <fixture>`). Then it kept the answer with the
prompt's version:

```text
✓ Recorded `answer_question`'s answer as `not_covered`
    asked     meta-llama/Llama-3.1-8B-Instruct through local_model
    from      the last `jadpo dev` call, call_3
    answer    decodes as SupportAnswer
    values    1 placeholder
    file      domains/support/prompts/recordings/answer_question/not_covered.json

✓ answer recorded  answer_question@5f8642d0 · not_covered

→ jadpo test  decodes the recording and answers tests with it
```

A test answers a call with it as `recorded("<name>")`. It counts as one
attempt, and runs through the call's real reading and decoding, as any
answer does:

```jadpo
use support/types { SupportAnswer }

test "a question the articles don't answer goes to a person" {
    fakes: {
        ai.answer_question: [recorded("not_covered")]
    }
    var reply: SupportAnswer = attempt answer_customer("Do you deliver to Jersey?")
    assert reply.resolved_by_articles == false
}
```

- **Every recording is decoded on every run** of `jadpo test`, through
  each type a call asks its prompt for as, every rule included. Each is a
  test of the `recording` kind, named `<prompt>'s <name> recording decodes`.
  It fails as `test.recording_undecodable`, before a test answers with
  the recording, when:
  - the type changed under the recording, or the recording is written
    wrong;
  - the recording's folder names no prompt beside it;
  - it isn't one JSON object with an `answer` and a `version`;
  - no call asks its prompt.
- **A changed prompt.** A recording made from an older version of its
  prompt still decodes, so its test passes, but the run says so, because
  the model may no longer answer that way:

```text
▲ alert  `answer_question`'s recording `not_covered` was made from an older
  prompt (1a2b3c4d, now 5f8642d0): its answer may no longer be what the
  model says

✓ 2 passed  1 action test · 1 recording test

→ jadpo ai record answer_question --as not_covered  records it again from the prompt now
```

Here a recording says the model answered `"yes"` where
`resolved_by_articles` is a `Bool`, so it doesn't decode:

```jadpo
{
  "answer": {
    "reply": "{Text}",
    "resolved_by_articles": "yes"
  },
  "version": "answer_question@5f8642d0"
}
```

```text
FAIL  answer_question's sesame recording decodes
 --> domains/support/prompts/recordings/answer_question/sesame.json:1

  decode: `answer_question`'s recording `sesame` doesn't decode: `type` on `resolved_by_articles`
  reason: Every build decodes every recording through each type its prompt is asked for as, every rule included, so an answer no call could read, or one its type has changed under since it was recorded, shows here before a test answers with it.
  next: Correct its answer to the type, or record it again (agent)

  rule: test.recording_undecodable
  rerun: jadpo test domains/support/prompts/recordings/answer_question/sesame.json:1 --seed 1

test failed: 1 failed, 1 recording test
```