Several changing calls in one action

Some jobs need several calls that each change something at a provider, in order: create a customer, attach a card, start a subscription. If the action stops halfway, a rerun must not do the first steps again. Jadpo records each step before it is sent, replays the recorded answers on a rerun, and stops the action as diverged if anything differs.

These rules apply to an app's action and to the package actions it calls, which count as one sequence. "Several changing calls in one action" in Jobs and events describes the same rules from the side of the event delivery that runs the action.

• Words. A step is one outside call made by an action from its first changing call on, numbered along the path the action runs, package actions' calls included. The record is the rows holding an action's steps, keyed by the action's key. A step is in doubt when it was recorded as sent with no answer recorded.
• How many, and which. An action may make up to 8 changing calls in order, counted on its longest path with package actions' calls included. Each must be idempotent: true in its package: it takes the key, or it is safe to repeat as it is (a PUT that replaces, a DELETE). A changing call with neither stays the only changing call in its action. A package action over the limit cannot be published.
• Where. Only in actions run from subscribers, or from routes whose API requires Idempotency-Key. Reaching such an action from a schedule, or from a route whose API does not require the key, is a compile error naming the two fixes: require the key, or answer 202 and do the steps in a subscriber.
• No outside call inside for in such an action, so the compiler knows every path's length; batch work stays one event per item.
• The record. Before a step is sent the runtime records its position, operation, a fingerprint of its request (over the typed value, not the bytes) and the time sent; when the answer comes, success or failure, it records the answer as the provider sent it, encrypted, at most 1 MB (a larger one is a fault naming the step). Provider reads after the first changing step are recorded the same way. Database reads are not, and reads before the first changing step are not.
• Replay. A rerun replays answered steps in order without sending them, decoding each stored answer with the current package. At each step the operation and the fingerprint must match, and at the end every recorded step must have been replayed; otherwise the action stops as diverged before anything is sent, naming the step, its operation and the top-level request fields that differ. A diverged action is held for an operator, not failed: deploying matching code and retrying finishes it.
• Time. Once a record exists, clock.now is the time stored in it (the first run's start) for every rerun, including the rerun after a failed commit (Reads and writes). A resumed action writes timestamps from when it began, which is also when the provider acted. Before the first changing step a rerun takes a fresh time. Configuration and time holds the same.
• Keys and windows. Each step's key is derived as "Calling" says, the step number and key generation included. A keyed step in doubt inside its provider's window: (less a margin, the smaller of a tenth of the window and one hour) ends the run retry-later, and the next attempt sends it again under the same key. It parks as outcome_unknown when it has no key, when its window would pass, or when the package says replays_failures: true and the provider answered 5xx. Only the package's new_key_on: answer makes the runtime send again under the next key generation by itself. The window rule holds for single calls too. A resumable single call with a store already available keeps only its operation, first dispatch time, key generation and runner generation in the step tables, with no request fingerprint or answer and no steps encryption setting. It keeps a fresh code clock and its existing provider key; host time bounds retries. A saved record cannot switch between encrypted replay and dispatch metadata: its saved mode disagrees with the new build's, so it stops as diverged before sending anything, with the problem kept for the operator. A route with no store keeps no durable history across independent requests.
• One runner per record. Routes claim the client's key before the action runs (HTTP). A delivery's lease carries a generation that every record write checks, so a runner whose lease lapsed stops before sending another step. The provider's "key in use" answer means wait: retried within the call's budget, then retry-later, never a failure. The provider's "different request" answer is diverged, never misconfigured.
• Transactions. Database writes still come after the last call, with no transaction open during calls (Outside calls hold no transaction). Record writes are the runtime's own, each in a short transaction.
• The warning. After the first changing step, every point that can end the action for good (a step's failure, a write that can be refused, reject, a called action's failure) is warned unless handled with match, naming three fixes: check the condition before the first step; record the answers and finish in a subscriber of an event the write emits; undo as another step. A package's surface lists which of its failures follow an effect, so the warning reaches through package actions. It holds for single calls too.
• Undo is code. There is no built-in compensation. An undo is a step like any other: it needs a key, gets the same warning, and a failure mapped as "already undone" (Stripe's resource_missing on a delete) counts as handled.
• Ending. The record is removed in the action's commit transaction, with the delivery's handled marker or the route's stored response. A record that ends any other way (failed, diverged, parked, or a route's key lapsing after 24 hours) becomes an operator item listing each step and its answer, with retry (--not-done to send the step in doubt under a new key once the operator has checked it did not happen) and abandon (end the action as failed, keeping the list of effects for tidying up by hand). It is kept at most 30 days, after which the answers are dropped and the step list stays.
• Where records live. In the store the action writes, beside the outbox; an action that writes no store needs events: { store: }, or it is a compile error.
• Personal data. Requests are kept only as fingerprints. Answers are encrypted with application { steps: { encryption_key: } } (Jobs and events). Nothing links them to a person, so the personal-data report lists "provider answers for unfinished actions, at most 30 days", as it does for logs. A metric and a no-setup alert watch the number of unfinished records and the age of the oldest.
• Deploys. Each record keeps the build id it started under and a digest of its action's step shape (the operations on each path, package steps included). No deploy command is involved: a rerun under another build sees the different build id in the manifest, compares the step shape, and stops as diverged before sending anything if it changed. Code after the last recorded step, new steps after it, writes and failure mappings are not changes to the shape.
• Run limits. A run cut off with a record (a Workers limit, a deploy, a crash) resumes; only its step in doubt follows the window rule. The deadline warning sums the timeout times the attempts of every step on the longest path, against the route's deadline.
• Tests and tools. Fakes keep keys as the provider does (the same key and request return the first answer; a different request returns the package's mismatch answer) and gain two scripted answers, lost (the provider acted and the answer never arrived) and in_use. jadpo test reruns every test that reaches a multi-step action with a crash at each step boundary (before sending, after sending but before recording, after recording) and checks that the end state and the fake's effects match the run without one; this is on by default. jadpo dev, jadpo watch and jadpo deliveries show list each step as sent (with its key), answered, replayed or in doubt.

$ jadpo deliveries show d_7Q2billing.subscribe, from subscriber start_subscriptionOutcome unknown at step 3 of 4  1  stripe.customers.create         answered     10:02:11  cus_P4x…  2  stripe.payment_methods.attach   answered     10:02:12  pm_1N…  3  stripe.subscriptions.create     no answer    10:02:12  Stripe answered 500 and repeats it for this key until 10:02 tomorrow  4  stripe.invoices.finalize        not reachedCheck in Stripe whether customer cus_P4x… has a subscription.  It doesn't:  jadpo deliveries retry d_7Q2 --not-done  It does:     jadpo deliveries abandon d_7Q2, then tidy up in Stripe

Calls without idempotency keys (GitHub, Twilio Messages, Postmark) stay one per action. A receipt sent after a charge belongs in a subscriber of the action's event, not as an unkeyed last step.

How the compiler keeps these rules

jadpo check keeps these rules with these errors: step.context, step.large_output, step.store_required, step.too_many, step.unkeyed, step.late_failure, effect.call_in_loop and route.calls_past_deadline. It reads an operation's large_output:. A provider's idempotency: window:, replays_failures: and new_key_on: are read into the plan, and an operation's changes: marks a step. The provider call records, replays and keys each step through the capability seam, whose exchange writes a step before it is sent.

The warning counts each point that can end the action for good after its first changing call as one finding for the action, which you can decline, unless it is handled with match. Those points are an attempt passing on a failure that isn't unavailable or rate_limited, reject, and propagate. A package's action lists on its surface which failures can follow its own changing call (after_effect), and the app's call of it is the point.

Each route's action sums the timeout: times the attempts of every outside call on its longest path, a provider's operation and an ai call alike, against the route's deadline:. A single call is checked the same way.

Where else this is covered

• What an operator sees, and the metric and the alert on unfinished records: "What an operator sees" and "Several changing calls in one action" in Jobs and events. Each step's state shows under its request's or delivery's line in jadpo dev.
• jadpo runs unfinished --build: Tooling.
• The fakes and crash points: "Provider calls and capabilities in tests" in Testing.
• The choices behind these rules: the design notes for Jobs and events.