Several changing calls in one action

Some jobs take several calls that each change something at a provider. Starting a subscription at a payment provider, for example, means creating the customer, attaching the card and creating the subscription. Each takes effect at the provider as soon as it is made. So an action that makes them must never send one twice, and must never lose track of one that happened. Jadpo makes this safe by recording each call, a step, as it goes.

domains/billing/subscribers/start_subscription.jadpo and domains/billing/actions/subscribe.jadpo:

// domains/billing/subscribers/start_subscription.jadposubscriber start_subscription {    on: PlanChosen    run: subscribe(event)}// domains/billing/actions/subscribe.jadpoaction subscribe(event: PlanChosen) fails CardDeclined -> Unit {    var customer: Stripe.Customer = attempt stripe.customers.create(...)    attempt stripe.payment_methods.attach(...)    var subscription: Stripe.Subscription = attempt stripe.subscriptions.create(...)    attempt Account.update(...)}

The three Stripe calls are steps 1, 2 and 3. The Account.update comes after the last call, because the action's writes always do.

A changing call is a provider operation with changes: true, or a store's put. Reads aren't counted.

What is allowed

• Up to 8 changing calls, in order, counted on the action's longest path through its code. A package action's calls count too: its steps are steps of the calling action, under one record, and a second package action carries on the numbering.
• Each must take an idempotency key, or be marked by its package as safe to repeat as it is. A changing call with neither must be the only changing call in its action, even as the last step, since a step in doubt with no key always needs an operator.
• Only where the action's key survives a retry. That is in actions run by subscribers, and by routes whose API requires an Idempotency-Key (Idempotent writes). A route without a required key gets a new request id when the client retries, and a schedule's retry is its next run, so neither could resume where it stopped. Making several changing calls from either is a compile error, which names the fixes: require the key; or answer 202 and do the steps in a subscriber, the usual shape for a browser form; or, for a schedule, emit an event for each item.
• No outside call inside for, in any action (Loops), so steps are numbered along a path whose length the compiler knows. For a call per item, use batches.each.
• store.put is a step like a provider call. In an action with several changing calls, its key is made as any step's is, so a rerun writes the same object rather than a second one (Files).
• Large answers stay out. A package may mark an operation large_output: true, and the compiler refuses such an operation in an action with several changing calls.
• Writes come after the last call, with no transaction open during the calls (All-or-nothing actions). The record's own writes are the runtime's, each in a short transaction of its own.

The compiler checks all of these.

Keys

Each step's idempotency key is made from the app, the environment, the connection, the caller's scope, the action's key (the delivery's key, or the client's Idempotency-Key), the step's number and a key generation, hashed to 36 characters or the package's max_length:.

A route's action that makes a changing call keeps its client's key, in the store it writes or in events: { store: }'s, even when it neither writes nor emits. Its calls' keys come from the client's key whether or not any store keeps it.

A delivery's attempt counts its steps from 1 wherever it runs, a route waiting for it with settle: included. A sign-in email (Built-in emails) is made with a new link and code on each attempt, so each attempt's calls go under a key of their own, and it keeps no record.

Hashing also fixes a problem single calls have. Jadpo scopes a client's key to the caller, but a provider such as Stripe scopes keys to the account, so two users sending the same key would otherwise be given one answer.

Each package states how its provider treats keys (Calling): how long they are kept (window:), their maximum length, the codes for "in use" and "different request", whether failed answers are replayed, and any "use a new key" answer.

The record

Before a step is sent, the runtime records its position, its operation, a fingerprint of its request (over the typed value) and the time it was sent. When the answer comes, success or failure, it records the answer, encrypted. An answer over 1 MB is a fault naming the step. A read from a provider after the first changing step is recorded the same way; database reads aren't.

The record lives in the store the action writes, beside the outbox. With no such store, events: { store: } is required. The record is deleted in the action's commit transaction. Records are listed in the personal-data report (Personal data).

A rerun replays

When the action runs again, after a retry, a redelivery or a rerun after a concurrent change, the answered steps are replayed in order without being sent. clock.now is the time stored in the record, fixed for the record's whole life. So timestamps written after a resumed action are from when it started, which is also when the provider acted.

Only your code's clock.now is fixed. The runtime's own timestamps use the host's time: an event's emitted_at, from which within: counts, a request's AWS signature, the day an ai budget counts, and a received event's time.

At each step, the operation and the fingerprint must match the record, and at the end every recorded step must have been replayed. Any difference stops the action as diverged before anything is sent, naming the step and the fields that differ. A deploy that changed the code, a fresh database read that changed a request, or changed configuration can each cause one. A diverged action is held for an operator rather than failed, so deploying a fix and retrying finishes it.

A step in doubt

A step in doubt was sent but has no answer recorded. It ends the run, to be retried later, and the next attempt sends it again under the same key while the provider's window lasts. It parks as outcome_unknown instead when it has no key, when its window would pass, or when the provider replays a 5xx under that key.

A step answered with one of its operation's own failures whose kind is unavailable or rate_limited did nothing. It stays in doubt, and its failure is never kept or replayed.

One runner per record

A route's key is claimed before its action runs. A delivery's lease carries a generation number that every write to the record checks. A provider's "key in use" answer means wait and retry, never a failure.

Ending for good after a step took effect

After the first changing step, every point that can end the action for good is a warning unless handled with match: a step's failure, a write that can be refused, a reject, or a called action's failure. The warning names three fixes:

• check first, before the first step;
• record what happened and finish in a subscriber;
• or undo it as another step.

Package surfaces list which failures follow an effect, so the warning reaches through package actions. It holds for single calls too.

An unfinished record never disappears quietly

A record that ends without committing (failed, diverged, parked, or a route whose key lapsed) becomes an operator item. It lists each step and its answer, and offers retry (--not-done for a new key generation) and abandon (What an operator sees). It is kept at most 30 days; after that only the step list remains.

Deploys

Each record keeps a digest of its action's step shape. Before a deploy, an operator runs jadpo runs unfinished --build <build id> against production. It lists the unfinished multi-step runs and batch runs whose recorded path the new build changes, so they can be let finish first. A run resumed under a changed build still stops as diverged.

Tests and tools

Fakes honour keys and can answer lost and in_use. jadpo test runs every multi-step test again with a crash at each step boundary, on by default (Fakes for integrations). jadpo dev, jadpo watch and jadpo deliveries show list each step's state (Debugging and review).

Providers that take no key

Calls to providers that take no key, such as GitHub, Twilio's messages and Postmark, stay one per action. A Stripe 500 at a later step is stored under its key for a day, so it parks the action for an operator rather than being retried.

What jadpo check checks

• up to 8 changing calls on one way through an action (step.too_many);
• a changing call with no key is the only one in its action (step.unkeyed);
• several only from a subscriber, or a route whose API requires Idempotency-Key (step.context);
• no large_output: true operation among them (step.large_output);
• a store for their record (step.store_required);
• no outside call inside for (effect.call_in_loop).

Each way through an if or a match counts on its own, a return ends its way, and an action called counts the changing calls of the longest way through it.

The late-failure warning

After the first changing call on a way through an action, each point that can end it for good is step.late_failure. There is one finding for the action, at the first such point, with the others beside it. You may decline it, since the condition may be one that can't change.

A point is:

• an attempt whose call or write passes on a failure that isn't unavailable or rate_limited: an operation's or a called action's failure, a write's conflict: or not-found, or a construction's;
• a reject;
• a propagate.

A call whose failures a match handles isn't a point, though a reject in one of its arms is. Where two ways meet, a changing call on either counts. A package's action is never warned itself. Instead its surface lists after_effect for each failure that can follow its own changing call, and the app's attempt of it is the point, even as the app's first call. The rule holds for single calls too.

The same walk adds up how long a route's outside calls can take: route.calls_past_deadline (Time limits).

How the record is kept

An action whose longest way makes several changing calls keeps its steps when it is run by a subscriber's delivery, or by a route whose API requires Idempotency-Key, under the key it claimed. The steps go in jadpo_step_records and jadpo_steps (System tables), in the store it writes or in events: { store: }'s.

• Each step (each changing call, and a provider read after the first) is written before it goes out, with its operation, an HMAC of each top-level field of its request, and when it was sent. Its answer is written after, sealed.
• A rerun (a delivery's retry, a client's retry under its key, or a rerun after a concurrent change) replays an answered step without sending it once its operation and fields match, with clock.now fixed at the first run's start.
• A step in doubt goes again under its key within the provider's window:, less a margin. It parks as outcome_unknown after the window, with no key, or on a 5xx from a provider that replays_failures: true.
• The provider's in_use: answer is retried later; its reused: answer is diverged.
• A request that differs, a run that makes another operation at a step, and a run that ends before replaying every recorded step all stop as diverged before sending anything. The delivery is held in that state with the step, the operation and the fields, never a value.
• A record written by a newer runner (a delivery's later attempt, or a route's later claim) takes no write from an older one.
• A put in such an action keeps its object under a key made from its step.
• The record is deleted in the action's commit.

Answers are sealed with application { steps: { encryption_key: } } (step.encryption_key_required, step.settings_invalid).

An example

A bakery pays an approved invoice and keeps a copy of it, two changing calls each under its own key, and then records the payment, all in a subscriber. Its settings, the Acme Pay package and the types it uses are hidden here; the subscriber and the action follow them.

use bakery/actions { pay_invoice }use bakery/events { InvoiceApproved }/// Pays each invoice once it is approved.subscriber pay_approved_invoice {    on: InvoiceApproved    run: pay_invoice(event)}

use app { pay, sheets }use bakery/entities { SupplierPayment }use bakery/failures { SheetNotKept }use bakery/events { InvoiceApproved }use shared/providers/acme_pay/// Pays an approved invoice, keeps a copy of it, then records both: two/// changing calls, each under its own key, then the write.action pay_invoice(event: InvoiceApproved) fails SheetNotKept -> Unit {    var payment: AcmePay.Payment = pay.payments.create(        AcmePay.PaymentRequest {            reference: event.invoice        }    )    var invoice_copy: File = attempt sheets.put(encoding.to_utf8(event.lines), FileType.csv) {        storage_full: SheetNotKept    }    var paid: SupplierPayment = SupplierPayment.create {        set: {            payment: payment.id            invoice_copy: invoice_copy        }    }    assert paid.payment == payment.id}

Following steps in jadpo dev

In jadpo dev, each step a request's action reached is a line of the request's trace, under the request's line, in place of its call. The line gives the step's place, its operation, where its answer came from, and what became of it: replayed, answered, its failure's code, in doubt, diverged or outcome unknown. It never shows a request, an answer or a key.

A delivery whose action reached steps gets a line of its own, as a schedule's run does. It shows the delivery's id, the subscriber and how the attempt left it, with its steps under it and, when it needs an operator, jadpo deliveries show <id>. --json carries both as steps, and the delivery as an event of its own.

For each store, the tick counts the unfinished records and notes when the oldest started. Once a record has been unfinished for a day, it raises an alert with no setup ({"event": "step.alert", "alert": "unfinished_steps", ...}), once across all instances, until none is unfinished. jadpo deliveries shows the alert beside the items, and its --json the count.

$ jadpo dev9c1f0a2b  POST /partners/subs…               503 unavailable   41 ms  └─ step 1 · pay.customers.create · generated · answered  └─ step 2 · sheets.put · live · answered  └─ step 3 · pay.subscriptions.create · generated · in doubt41d7e09a  subscriber start_subscription      ◆ diverged  └─ step 1 · pay.customers.create · replayed  └─ step 2 · sheets.put · diverged  → jadpo deliveries show 41d7e09a

Testing it

The bakery's test scripts one payment. jadpo test runs the test again with a crash at each boundary of both steps. Each time, the step sent again goes under its key and is answered with what the fake kept, so one scripted answer is enough:

use app { pay }use shared/providers/acme_paytest "an approved invoice is paid once" {    fakes: {        pay.payments.create: [            AcmePay.Payment {                id: "pay-17"            }        ]    }    approve_invoice("inv-17", "rye,4.20")}