Retries and giving up

When a subscriber's action fails, Jadpo decides whether to try again from the way it failed. You don't write anything for that. A failure that may pass, such as a provider being down, is retried later; one that would happen again with the same input isn't. The failure kinds are those in Kinds:

A fault reaching a store, and a run out of time, are retried as unavailable is. A delivery's record keeps the failure's code and kind, never its fields.

How long between tries

Each retry waits longer than the last, with random jitter, so many failures don't all retry at once. Each wait is drawn evenly from zero to a ceiling that starts at 30 seconds and doubles with each attempt, up to 6 hours, and is never under a second. A Retry-After from a provider is honoured.

How many tries: the default

By default a subscriber gets 10 attempts within 24 hours, then fails. A delivery fails when it is past its attempts, or when its next attempt would fall more than within: after the event was emitted.

Changing the budget: retry:

Use it for a subscriber that must keep trying for longer, such as one recording payments from a provider that may be down for a while.

subscriber record_payment_webhook {    on: PaymentReceived    run: record_payment(event.payment_id)    retry: {        attempts: 20        within: 3d    }}

This subscriber gets 20 attempts within 3 days. attempts: takes 1 to 100, and within: takes up to 30 days; anything else is subscriber.retry_invalid. A schedule has no retry:, since its next due time is its retry.

A run that takes too long

A delivery whose action is stopped by the run's time limit (Running background work) is an attempt that ran out of time (deadline_exceeded). It counts against its budget at once: it is tried again after a wait while the budget lasts, and failed after that, with nothing it did kept. So a subscriber that never ends fails after its attempts, rather than using up every invocation's budget.

Given up at the claim

A delivery claimed again after its retry: attempts are used, or later than within: after its event was emitted, fails without running. Claiming one a worker held past its lease marks its last failure lease_expired, as a schedule's run lost with its worker ends. The claim that gives up isn't counted as an attempt. The record keeps the last attempt's failure, or deadline_exceeded for a delivery never tried in time. A handled record found first still makes it succeeded, and a store that can't be reached while looking for one is retried within the same budget.

When a provider keeps failing

Jitter keeps retries from arriving together, but 5,000 deliveries calling a provider that is down would still each be tried ten times. So the outbox keeps a circuit breaker for each connection. After a run of unavailable failures from one provider, the deliveries that call it are held rather than tried. After a pause, one of them, the probe, is tried, and if it succeeds the rest are released. Held deliveries keep their retry budget, are shown in jadpo deliveries, and raise an alert. You don't write anything for this.

• What opens the breaker. Five delivery attempts in a row whose call through the connection found its provider unavailable open its breaker. An attempt counts once for each connection it called through, by how the last such call ended. It counts as unavailable when the call surely did nothing or no answer came back: a 5xx, a 408, a 429, a timeout, or a call that never left. It counts as answered, ending the run of failures, when the provider answered at all, a refusal or a rejected credential included. A delivery attempt counts once however many times the connection's own retry: sent its call (Calling), by how the last of them ended. So one a retry got through ends the run, and five attempts of three refused sends each open the breaker, not fifteen sends. Only calls to the connection's own driver count, never a fake's or a recording's answer, and only calls a delivery makes: a route's calls are its caller's to retry.
• Which deliveries are held. A subscriber's delivery, when its action calls out through the connection, itself or through an action it calls, or when it hands its event to a capability the connection fills. An action a batch's run calls runs after the commit, not in the delivery, so it doesn't count. Each delivery is held at its claim, before it runs: it becomes held and not due, the attempt its claim counted is given back, and its connection and the time it was held from are kept on its record.
• The pause and the probe. The first pause is a minute. Once it has passed, the first delivery claimed for the connection is the probe, and the rest stay held. The tick's sweep wakes the delivery held longest to be the probe, so a probe goes even when nothing new arrives. A probe the provider answers closes the breaker and releases every delivery it held, due at once. A probe that finds the provider unavailable opens the breaker again for twice the last pause, at most ten minutes. A probe whose run no longer calls the provider gives the probe back, and one whose worker is lost loses it when its claim's lease ends. A delivery already running when the breaker opened finishes, and its answer closes the breaker too.
• The budget. Being held spends no attempt, and the time held doesn't count towards within:. A delivery's within: counts from its event, or from an operator's retry, moved on by the time it was held. An operator's retry gives a fresh budget, the time held included.
• Where it is kept. In jadpo_circuit_breakers (System tables), in every store that keeps deliveries, each store's breakers holding its own deliveries. Every change is one conditional statement, so the breaker holds across processes and on Workers. The sweep also releases any delivery whose breaker has closed, which a release racing a hold can leave.
• The alert. The tick raises one alert for each outage, connection_unavailable, recorded on the breaker's row as a schedule's alerts are on its state, so of several instances only one raises it. It is raised again only after the breaker has closed and opened again. Its log event is {"event": "delivery.alert", "alert": "connection_unavailable", "connection": …, "since": …, "held": …}, and jadpo dev prints it with the schedules' alerts.

Not built yet

The jitter and Retry-After are built for a call's own retries within one attempt (Calling). Between attempts of a delivery, the next attempt waits only its backoff.