Testing AI calls

A test never reaches a real model. Every ai call in jadpo test goes through the capability's one fake, whatever the provider, as every capability has one fake (Fakes for integrations). There is no flag to let a call through. Without a script, the fake answers with a valid value of the type you asked for:

The fake gives	When
A valid default of `T`, generated from the test's seed (Controlling time)	Nothing is scripted
The scripted answers, in order	`fakes: { ai.answer_question: [...] }`, by prompt name
A recorded answer	`recorded("sesame")` in a script, read from `prompts/recordings/answer_question/sesame.json`
A failure	`refused`, `malformed`, `timed_out`, `over_budget`, `too_long` in a script
An embedding	A vector derived from the text's SHA-256 and the seed, so the same text always gives the same vector, or a scripted one
A stream	The scripted text, cut into pieces of a few words; `partial("...", <condition>)` for one that stops part-way

A test scripts the fake's answers by the prompt's name, under fakes:, and reads what was asked from ai.<prompt>.requests:

example

// domains/support/actions/answer_customer.test.jadpotest "a question the articles don't answer goes to a person" {    fakes: {        ai.answer_question: [recorded("not_covered")]    }    var reply: SupportAnswer = attempt answer_customer(questions.asked("Do you deliver to Jersey?"))    assert reply.resolved_by_articles == false    var asked: List<AnswerQuestion> = ai.answer_question.requests    assert list.count(asked) == 1}test "a refusal is reported as the assistant being unavailable" {    fakes: {        ai.answer_question: [refused]    }    assert answer_customer(questions.asked("Help me pick a lock")) fails AssistantUnavailable}

Scripted answers are checked against the call's as: type when the test compiles, as a package fake's are (Tests and fakes). So a test can't script an answer the decoding would reject. To test a repair, script malformed, or a JSON text with raw("..."), which runs through the real decoding and retry.

Here is the help desk from A model on this machine, tested: a scripted answer and what was asked, an answer corrected once, and each failure. Each scripted answer is one attempt, so [malformed, malformed] is an answer that still doesn't decode once corrected:

example

use support/types { CustomerQuestion, SupportAnswer }use support/failures { AnswerUnreadable, AssistantUnavailable }use support/prompts { AnswerQuestion }test "a question the model answers" {    fakes: {        ai.answer_question: [            SupportAnswer {                resolved_by_articles: true                reply: "Our sourdough is baked beside sesame."            }        ]    }    var reply: SupportAnswer = attempt answer_customer("Does your sourdough contain sesame?")    assert reply.resolved_by_articles    var asked: List<AnswerQuestion> = ai.answer_question.requests    assert list.count(asked) == 1    assert list.first(asked) == AnswerQuestion {        question: "Does your sourdough contain sesame?"        reader_locale: Locale.en_gb    }}test "an answer that doesn't decode is corrected once" {    fakes: {        ai.answer_question: [            raw("Sure! Our sourdough contains no sesame."),            raw("{\"resolved_by_articles\": true, \"reply\": \"No sesame.\"}")        ]    }    var reply: SupportAnswer = attempt answer_customer("Sesame?")    assert reply.reply == "No sesame."}test "each failure is one of the app's own" {    fakes: {        ai.answer_question: [refused, timed_out, over_budget, too_long, malformed, malformed]    }    assert answer_customer("Help me pick a lock") fails AssistantUnavailable    assert answer_customer("Sesame?") fails AssistantUnavailable    assert answer_customer("Sesame?") fails AssistantUnavailable    assert answer_customer("Sesame?") fails AssistantUnavailable    assert answer_customer("Sesame?") fails AnswerUnreadable}

ai.<prompt>.requests lists the prompt values the code sent, typed, so a test asserts on what went to the model, not on rendered text.
Recordings are made by jadpo ai record answer_question --as sesame in development, against the configured model with the developer's own key, from the prompt values of the last jadpo dev call or a fixture. The file keeps the answer and the prompt's digest. Every build decodes every recording through its as: type, so a type change shows each stale one. When a prompt's digest has changed since a recording was made, the answer is still a valid value but may no longer be what the model says. Jadpo has no warnings (Every error has the same shape), so this is an alert on jadpo test's report, naming the recording, and the run passes (Recorded answers).
The clock and random numbers are seeded, as in every test (Controlling time), so a generated default and a fake embedding are the same on every run.
In jadpo dev, every ai call is faked too: from recordings, then from generated defaults, as every provider call is (Fakes and the development call log). development: { sandbox: [answer_question] } on the ai connection lets the prompts it names through to the real model with the developer's key, under a development per_day limit. The OpenAI-compatible driver runs a local model with no key at all.
Quality is not a test. Whether a prompt still answers well against a real model is an evaluation, which Jadpo leaves out (What AI leaves out, and what to use instead).

Recorded answers

A recording keeps an answer a model gave to a prompt, so a test can answer with it. It sits beside the prompt in recordings/<prompt>/<name>.json, with the version of the prompt it answered, as jadpo ai show prints it. The answer is written as json.write writes the type the prompt is asked for as, its members in name order and two spaces deep. A text or number that could say something about someone is kept as a placeholder naming its type, "{Text}", as a provider's recording keeps one (Recordings). It is filled with a generated value of its type each time it is read. The details are under Embeddings and recordings and Recording an answer.

example

{  "answer": {    "reply": "{Text}",    "resolved_by_articles": false  },  "version": "answer_question@5f8642d0"}

jadpo ai record answer_question --as not_covered made it. It asked the configured model once, as the prompt's first ai.generate call asks, with the values the prompt's last jadpo dev call gave it (or a fixture's, with --fixture <fixture>). Then it kept the answer with the prompt's version:

text

✓ Recorded `answer_question`'s answer as `not_covered`    asked     meta-llama/Llama-3.1-8B-Instruct through local_model    from      the last `jadpo dev` call, call_3    answer    decodes as SupportAnswer    values    1 placeholder    file      domains/support/prompts/recordings/answer_question/not_covered.json✓ answer recorded  answer_question@5f8642d0 · not_covered→ jadpo test  decodes the recording and answers tests with it

A test answers a call with it as recorded("<name>"). It counts as one attempt, and runs through the call's real reading and decoding, as any answer does:

example

use support/types { SupportAnswer }test "a question the articles don't answer goes to a person" {    fakes: {        ai.answer_question: [recorded("not_covered")]    }    var reply: SupportAnswer = attempt answer_customer("Do you deliver to Jersey?")    assert reply.resolved_by_articles == false}

Every recording is decoded on every run of jadpo test, through each type a call asks its prompt for as, every rule included. Each is a test of the recording kind, named <prompt>'s <name> recording decodes. It fails as test.recording_undecodable, before a test answers with the recording, when:
- the type changed under the recording, or the recording is written wrong;
- the recording's folder names no prompt beside it;
- it isn't one JSON object with an answer and a version;
- no call asks its prompt.
A changed prompt. A recording made from an older version of its prompt still decodes, so its test passes, but the run says so, because the model may no longer answer that way:

text

▲ alert  `answer_question`'s recording `not_covered` was made from an older  prompt (1a2b3c4d, now 5f8642d0): its answer may no longer be what the  model says✓ 2 passed  1 action test · 1 recording test→ jadpo ai record answer_question --as not_covered  records it again from the prompt now

Here a recording says the model answered "yes" where resolved_by_articles is a Bool, so it doesn't decode:

example

{  "answer": {    "reply": "{Text}",    "resolved_by_articles": "yes"  },  "version": "answer_question@5f8642d0"}

text

FAIL  answer_question's sesame recording decodes --> domains/support/prompts/recordings/answer_question/sesame.json:1  decode: `answer_question`'s recording `sesame` doesn't decode: `type` on `resolved_by_articles`  reason: Every build decodes every recording through each type its prompt is asked for as, every rule included, so an answer no call could read, or one its type has changed under since it was recorded, shows here before a test answers with it.  next: Correct its answer to the type, or record it again (agent)  rule: test.recording_undecodable  rerun: jadpo test domains/support/prompts/recordings/answer_question/sesame.json:1 --seed 1test failed: 1 failed, 1 recording test

Recorded answers

Related