# Several changing calls in one action

Some jobs take several calls that each change something at a provider.
Starting a subscription at a payment provider, for example, means
creating the customer, attaching the card and creating the
subscription. Each takes effect at the provider as soon as it is made.
So an action that makes them must never send one twice, and must never
lose track of one that happened. Jadpo makes this safe by recording each
call, a **step**, as it goes.

`domains/billing/subscribers/start_subscription.jadpo` and
`domains/billing/actions/subscribe.jadpo`:

```jadpo
// domains/billing/subscribers/start_subscription.jadpo
subscriber start_subscription {
    on: PlanChosen
    run: subscribe(event)
}

// domains/billing/actions/subscribe.jadpo
action subscribe(event: PlanChosen) fails CardDeclined -> Unit {
    var customer: Stripe.Customer = attempt stripe.customers.create(...)
    attempt stripe.payment_methods.attach(...)
    var subscription: Stripe.Subscription = attempt stripe.subscriptions.create(...)
    attempt Account.update(...)
}
```

The three Stripe calls are steps 1, 2 and 3. The `Account.update` comes
after the last call, because the action's writes always do.

A **changing call** is a provider operation with `changes: true`, or a
store's `put`. Reads aren't counted.

## What is allowed

- **Up to 8 changing calls**, in order, counted on the action's longest
  path through its code. A package action's calls count too: its steps
  are steps of the calling action, under one record, and a second
  package action carries on the numbering.
- **Each must take an idempotency key, or be marked by its package as
  safe to repeat as it is.** A changing call with neither must be the
  only changing call in its action, even as the last step, since a step
  in doubt with no key always needs an operator.
- **Only where the action's key survives a retry.** That is in actions
  run by subscribers, and by routes whose API requires an
  `Idempotency-Key` ([Idempotent writes](idempotent-writes.md)).
  A route without a required key gets a new request id when the client
  retries, and a schedule's retry is its next run, so neither could
  resume where it stopped. Making several changing calls from either is
  a compile error, which names the fixes: require the key; or answer
  `202` and do the steps in a subscriber, the usual shape for a browser
  form; or, for a schedule, emit an event for each item.
- **No outside call inside `for`**, in any action
  ([Loops](loops.md)), so steps are numbered along a path whose
  length the compiler knows. For a call per item, use `batches.each`.
- **`store.put` is a step** like a provider call. In an action with
  several changing calls, its key is made as any step's is, so a rerun
  writes the same object rather than a second one
  ([Files](files.md)).
- **Large answers stay out.** A package may mark an operation
  `large_output: true`, and the compiler refuses such an operation in an
  action with several changing calls.
- **Writes come after the last call**, with no transaction open during
  the calls ([All-or-nothing actions](all-or-nothing-actions-and-concurrent-changes.md)).
  The record's own writes are the runtime's, each in a short transaction
  of its own.

The compiler checks all of these.

## Keys

Each step's idempotency key is made from the app, the environment, the
connection, the caller's scope, the action's key (the delivery's key, or
the client's `Idempotency-Key`), the step's number and a key generation,
hashed to 36 characters or the package's `max_length:`.

A route's action that makes a changing call keeps its client's key, in
the store it writes or in `events: { store: }`'s, even when it neither
writes nor emits. Its calls' keys come from the client's key whether or
not any store keeps it.

A delivery's attempt counts its steps from 1 wherever it runs, a route
waiting for it with `settle:` included. A sign-in email
([Built-in emails](built-in-emails.md)) is made
with a new link and code on each attempt, so each attempt's calls go
under a key of their own, and it keeps no record.

Hashing also fixes a problem single calls have. Jadpo scopes a client's
key to the caller, but a provider such as Stripe scopes keys to the
account, so two users sending the same key would otherwise be given one
answer.

Each package states how its provider treats keys
([Calling](calling.md)): how long they are kept
(`window:`), their maximum length, the codes for "in use" and "different
request", whether failed answers are replayed, and any "use a new key"
answer.

## The record

Before a step is sent, the runtime records its position, its operation,
a fingerprint of its request (over the typed value) and the time it was
sent. When the answer comes, success or failure, it records the answer,
encrypted. An answer over 1 MB is a fault naming the step. A read from a
provider after the first changing step is recorded the same way;
database reads aren't.

The record lives in the store the action writes, beside the outbox. With
no such store, `events: { store: }` is required. The record is deleted in
the action's commit transaction. Records are listed in the personal-data
report ([Personal data](personal-data.md)).

## A rerun replays

When the action runs again, after a retry, a redelivery or a rerun after
a concurrent change, the answered steps are replayed in order without
being sent. `clock.now` is the time stored in the record, fixed for the
record's whole life. So timestamps written after a resumed action are
from when it started, which is also when the provider acted.

Only your code's `clock.now` is fixed. The runtime's own timestamps use
the host's time: an event's `emitted_at`, from which `within:` counts, a
request's AWS signature, the day an `ai` budget counts, and a received
event's time.

At each step, the operation and the fingerprint must match the record,
and at the end every recorded step must have been replayed. Any
difference stops the action as `diverged` before anything is sent,
naming the step and the fields that differ. A deploy that changed the
code, a fresh database read that changed a request, or changed
configuration can each cause one. A `diverged` action is held for an
operator rather than failed, so deploying a fix and retrying finishes
it.

## A step in doubt

A step in doubt was sent but has no answer recorded. It ends the run, to
be retried later, and the next attempt sends it again under the same key
while the provider's window lasts. It parks as `outcome_unknown` instead
when it has no key, when its window would pass, or when the provider
replays a `5xx` under that key.

A step answered with one of its operation's own failures whose kind is
`unavailable` or `rate_limited` did nothing. It stays in doubt, and its
failure is never kept or replayed.

## One runner per record

A route's key is claimed before its action runs. A delivery's lease
carries a generation number that every write to the record checks. A
provider's "key in use" answer means wait and retry, never a failure.

## Ending for good after a step took effect

After the first changing step, every point that can end the action for
good is a warning unless handled with `match`: a step's failure, a write
that can be refused, a `reject`, or a called action's failure. The
warning names three fixes:

- check first, before the first step;
- record what happened and finish in a subscriber;
- or undo it as another step.

Package surfaces list which failures follow an effect, so the warning
reaches through package actions. It holds for single calls too.

## An unfinished record never disappears quietly

A record that ends without committing (failed, diverged, parked, or a
route whose key lapsed) becomes an operator item. It lists each step
and its answer, and offers retry (`--not-done` for a new key generation)
and abandon ([What an operator sees](what-an-operator-sees.md)). It is kept
at most 30 days; after that only the step list remains.

## Deploys

Each record keeps a digest of its action's step shape. Before a deploy,
an operator runs `jadpo runs unfinished --build <build id>` against
production. It lists the unfinished multi-step runs and batch runs whose
recorded path the new build changes, so they can be let finish first. A
run resumed under a changed build still stops as `diverged`.

## Tests and tools

Fakes honour keys and can answer `lost` and `in_use`. `jadpo test` runs
every multi-step test again with a crash at each step boundary, on by
default ([Fakes for integrations](fakes-for-integrations.md)).
`jadpo dev`, `jadpo watch` and `jadpo deliveries show` list each step's
state ([Debugging and review](debugging-and-review.md)).

## Providers that take no key

Calls to providers that take no key, such as GitHub, Twilio's messages
and Postmark, stay one per action. A Stripe `500` at a later step is
stored under its key for a day, so it parks the action for an operator
rather than being retried.

## What `jadpo check` checks

- up to 8 changing calls on one way through an action
  ([`step.too_many`](../errors/step.too_many.md));
- a changing call with no key is the only one in its action
  ([`step.unkeyed`](../errors/step.unkeyed.md));
- several only from a subscriber, or a route whose API requires
  `Idempotency-Key` ([`step.context`](../errors/step.context.md));
- no `large_output: true` operation among them
  ([`step.large_output`](../errors/step.large_output.md));
- a store for their record
  ([`step.store_required`](../errors/step.store_required.md));
- no outside call inside `for`
  ([`effect.call_in_loop`](../errors/effect.call_in_loop.md)).

Each way through an `if` or a `match` counts on its own, a `return` ends
its way, and an action called counts the changing calls of the longest
way through it.

## The late-failure warning

After the first changing call on a way through an action, each point
that can end it for good is
[`step.late_failure`](../errors/step.late_failure.md). There is one
finding for the action, at the first such point, with the others beside
it. You may decline it, since the condition may be one that can't
change.

A point is:

- an `attempt` whose call or write passes on a failure that isn't
  `unavailable` or `rate_limited`: an operation's or a called action's
  failure, a write's `conflict:` or not-found, or a construction's;
- a `reject`;
- a `propagate`.

A call whose failures a `match` handles isn't a point, though a `reject`
in one of its arms is. Where two ways meet, a changing call on either
counts. A package's action is never warned itself. Instead its surface
lists `after_effect` for each failure that can follow its own changing
call, and the app's `attempt` of it is the point, even as the app's
first call. The rule holds for single calls too.

The same walk adds up how long a route's outside calls can take:
[`route.calls_past_deadline`](../errors/route.calls_past_deadline.md)
([Time limits](time-limits.md)).

## How the record is kept

An action whose longest way makes several changing calls keeps its steps
when it is run by a subscriber's delivery, or by a route whose API
requires `Idempotency-Key`, under the key it claimed. The steps go in
`jadpo_step_records` and `jadpo_steps`
([System tables](system-tables.md)), in the store it writes or
in `events: { store: }`'s.

- Each step (each changing call, and a provider read after the first) is
  written before it goes out, with its operation, an HMAC of each
  top-level field of its request, and when it was sent. Its answer is
  written after, sealed.
- A rerun (a delivery's retry, a client's retry under its key, or a
  rerun after a concurrent change) replays an answered step without
  sending it once its operation and fields match, with `clock.now` fixed
  at the first run's start.
- A step in doubt goes again under its key within the provider's
  `window:`, less a margin. It parks as `outcome_unknown` after the
  window, with no key, or on a `5xx` from a provider that
  `replays_failures: true`.
- The provider's `in_use:` answer is retried later; its `reused:` answer
  is `diverged`.
- A request that differs, a run that makes another operation at a step,
  and a run that ends before replaying every recorded step all stop as
  `diverged` before sending anything. The delivery is held in that state
  with the step, the operation and the fields, never a value.
- A record written by a newer runner (a delivery's later attempt, or a
  route's later claim) takes no write from an older one.
- A `put` in such an action keeps its object under a key made from its
  step.
- The record is deleted in the action's commit.

Answers are sealed with `application { steps: { encryption_key: } }`
([`step.encryption_key_required`](../errors/step.encryption_key_required.md),
[`step.settings_invalid`](../errors/step.settings_invalid.md)).

## An example

A bakery pays an approved invoice and keeps a copy of it, two changing
calls each under its own key, and then records the payment, all in a
subscriber. Its settings, the Acme Pay package and the types it uses are
hidden here; the subscriber and the action follow them.

```jadpo
use bakery/actions { pay_invoice }
use bakery/events { InvoiceApproved }

/// Pays each invoice once it is approved.
subscriber pay_approved_invoice {
    on: InvoiceApproved
    run: pay_invoice(event)
}
```

```jadpo
use app { pay, sheets }
use bakery/entities { SupplierPayment }
use bakery/failures { SheetNotKept }
use bakery/events { InvoiceApproved }
use shared/providers/acme_pay

/// Pays an approved invoice, keeps a copy of it, then records both: two
/// changing calls, each under its own key, then the write.
action pay_invoice(event: InvoiceApproved) fails SheetNotKept -> Unit {
    var payment: AcmePay.Payment = pay.payments.create(
        AcmePay.PaymentRequest {
            reference: event.invoice
        }
    )
    var invoice_copy: File = attempt sheets.put(encoding.to_utf8(event.lines), FileType.csv) {
        storage_full: SheetNotKept
    }
    var paid: SupplierPayment = SupplierPayment.create {
        set: {
            payment: payment.id
            invoice_copy: invoice_copy
        }
    }
    assert paid.payment == payment.id
}
```

### Following steps in `jadpo dev`

In `jadpo dev`, each step a request's action reached is a line of the
request's trace, under the request's line, in place of its call. The
line gives the step's place, its operation, where its answer came from,
and what became of it: `replayed`, `answered`, its failure's code,
`in doubt`, `diverged` or `outcome unknown`. It never shows a request,
an answer or a key.

A delivery whose action reached steps gets a line of its own, as a
schedule's run does. It shows the delivery's id, the subscriber and how
the attempt left it, with its steps under it and, when it needs an
operator, `jadpo deliveries show <id>`. `--json` carries both as
`steps`, and the delivery as an event of its own.

For each store, the tick counts the unfinished records and notes when
the oldest started. Once a record has been unfinished for a day, it
raises an alert with no setup
(`{"event": "step.alert", "alert": "unfinished_steps", ...}`), once
across all instances, until none is unfinished. `jadpo deliveries` shows
the alert beside the items, and its `--json` the count.

```text
$ jadpo dev
9c1f0a2b  POST /partners/subs…               503 unavailable   41 ms
  └─ step 1 · pay.customers.create · generated · answered
  └─ step 2 · sheets.put · live · answered
  └─ step 3 · pay.subscriptions.create · generated · in doubt
41d7e09a  subscriber start_subscription      ◆ diverged
  └─ step 1 · pay.customers.create · replayed
  └─ step 2 · sheets.put · diverged
  → jadpo deliveries show 41d7e09a
```

### Testing it

The bakery's test scripts one payment. `jadpo test` runs the test again
with a crash at each boundary of both steps. Each time, the step sent
again goes under its key and is answered with what the fake kept, so
one scripted answer is enough:

```jadpo
use app { pay }
use shared/providers/acme_pay

test "an approved invoice is paid once" {
    fakes: {
        pay.payments.create: [
            AcmePay.Payment {
                id: "pay-17"
            }
        ]
    }
    approve_invoice("inv-17", "rye,4.20")
}
```