Delivery: at least once, safe to repeat

Every committed event reaches each of its subscribers at least once. A crash can deliver it again, so Jadpo makes a repeat harmless: database work happens once, and outside calls carry keys that let the provider ignore a repeat.

At least once

An event committed is delivered to each subscriber at least once. A crash can deliver it again.

Database work happens once

The subscriber's writes commit together with a handled record, a note that this delivery was dealt with. A repeated delivery finds the note and does nothing.

The handled record (jadpo_handled_deliveries, keyed by the delivery's id) is written after the action's outbox rows, in the store its transaction is in by then: the store it writes or, if it writes none, the one events: { store: } names, where its events are. So an action that only emits keeps one, and a repeat emits nothing twice. An action that neither writes nor emits keeps none, since running it again changes nothing in a database. The handled record references nothing in another store.

Outside calls can repeat

A call to a provider can't join the database transaction. So each delivery has a stable key, made from the event and the subscriber, and the runtime derives from it the idempotency key each provider operation sends (Calling, Several changing calls in one action). A provider that supports idempotency keys ignores the repeat.

Every action repeats as a unit

A changing call is a call that changes something at the provider, such as taking a payment. A changing call that takes no idempotency key, and that its package doesn't mark as safe to repeat, must be the only changing call in its action. Once it has gone out, it is never sent again blindly. A retry the action would otherwise get, or a rerun after a concurrent change (All-or-nothing actions), stops it as outcome_unknown for an operator to check instead.

Changing calls that do take keys may be several, in order. Each is recorded as it goes, so a rerun replays what already happened rather than sending it again. Writes aren't limited this way: every write to the action's store, across any number of domains, commits together after the last call.

No order between events

No order is promised between events. A subscriber that cares about order reads the current state rather than trusting the event's copy, which is also what makes repeats safe.

A payload is the contract's JSON

An event saved in the outbox, a job's arguments and the values a batch captures are all written as the JSON of their types, by the same codec a request's body goes through. So a standard type keeps its currency, its unit and every decimal place (Every boundary through the contract). Every event type is numbered in the plan's JSON table by its identity key, and the outbox writes and reads its payload through that table (Structured values).

A payload is read with its type as the type is when the payload is delivered. An event type's shape is recorded as an API record's is (The event record), and a change that would break an API record, such as a field removed, renamed or narrowed, needs an acceptance. A payload saved before such a change that no longer reads is parked with its problem, never dropped, as a message that keeps failing is (Events from outside).

A delivery reads its payload by the encoding its outbox row records (Changing a standard type's encoding): this release's encoding, or none for a payload kept before encodings were recorded, which is read as encoding 1's. A payload in an encoding this release doesn't read, such as a later release's after a rollback, is never guessed at. Its delivery is parked, failed with payload_unreadable and why, as a payload that no longer reads as its type is, with its problem: the rule and the field, never the value.

How a delivery runs

Once the action commits, each delivery is left to the runner, which runs it after the response (Running background work). The runner:

1. claims the delivery's record in one statement, at the host's time, marking it running with a lease of what is left of the invocation's budget plus a minute. A delivery another runner holds, or one not yet due, is left alone;
2. looks for its handled record. A delivery already handled is marked succeeded without running again;
3. otherwise runs the subscriber's action as the system, limited to its event's tenant (Background work and tenants), in a boundary of its own, the unit that commits or rolls back as one, and in a new span of the event's trace;
4. records how it ended.

Tracing follows the work through events. A request continues a valid W3C traceparent it brings, in a span of its own, or starts a trace. An event keeps the trace context of the action that emitted it, so what a delivery emits keeps it too.

A delivery to a subscriber the build no longer has fails as a defect. An event 16 deep in a chain of events fails with event_chain_limit.