# Retries and giving up

When a subscriber's action fails, Jadpo decides whether to try again
from the way it failed. You don't write anything for that. A failure
that may pass, such as a provider being down, is retried later; one that
would happen again with the same input isn't. The failure kinds are
those in [Kinds](kinds.md):

| How the action ended | What happens |
|---|---|
| Succeeded | Done |
| A failure of kind `unavailable` or `rate_limited` | Retried later |
| Any other declared failure (`not_found`, `conflict`, `rejected`, ...) | Failed; not retried, because the same input fails the same way |
| A fault before anything committed (database unavailable, timed out) | Retried later |
| `outcome_unknown`: an outside call may have happened | Parked; never retried automatically |
| A defect (`internal_fault`, overflow) | Failed; not retried until someone fixes the code |

A fault reaching a store, and a run out of time, are retried as
`unavailable` is. A delivery's record keeps the failure's code and kind,
never its fields.

## How long between tries

Each retry waits longer than the last, with random jitter, so many
failures don't all retry at once. Each wait is drawn evenly from zero to
a ceiling that starts at 30 seconds and doubles with each attempt, up to
6 hours, and is never under a second. A `Retry-After` from a provider is
honoured.

## How many tries: the default

By default a subscriber gets 10 attempts within 24 hours, then fails. A
delivery fails when it is past its attempts, or when its next attempt
would fall more than `within:` after the event was emitted.

## Changing the budget: `retry:`

Use it for a subscriber that must keep trying for longer, such as one
recording payments from a provider that may be down for a while.

```jadpo
subscriber record_payment_webhook {
    on: PaymentReceived
    run: record_payment(event.payment_id)
    retry: {
        attempts: 20
        within: 3d
    }
}
```

This subscriber gets 20 attempts within 3 days. `attempts:` takes 1 to
100, and `within:` takes up to 30 days; anything else is
`subscriber.retry_invalid`. A schedule has no `retry:`, since its next
due time is its retry.

## A run that takes too long

A delivery whose action is stopped by the run's time limit
([Running background work](where-events-travel.md#running-background-work)) is an attempt
that ran out of time (`deadline_exceeded`). It counts against its budget
at once: it is tried again after a wait while the budget lasts, and
failed after that, with nothing it did kept. So a subscriber that never
ends fails after its attempts, rather than using up every invocation's
budget.

## Given up at the claim

A delivery claimed again after its `retry:` attempts are used, or later
than `within:` after its event was emitted, fails without running.
Claiming one a worker held past its lease marks its last failure
`lease_expired`, as a schedule's run lost with its worker ends. The
claim that gives up isn't counted as an attempt. The record keeps the
last attempt's failure, or `deadline_exceeded` for a delivery never
tried in time. A handled record found first still makes it `succeeded`,
and a store that can't be reached while looking for one is retried
within the same budget.

## When a provider keeps failing

Jitter keeps retries from arriving together, but 5,000 deliveries
calling a provider that is down would still each be tried ten times. So
the outbox keeps a **circuit breaker** for each connection. After a run
of `unavailable` failures from one provider, the deliveries that call it
are held rather than tried. After a pause, one of them, the probe, is
tried, and if it succeeds the rest are released. Held deliveries keep
their retry budget, are shown in `jadpo deliveries`, and raise an alert.
You don't write anything for this.

- **What opens the breaker.** Five delivery attempts in a row whose call
  through the connection found its provider unavailable open its
  breaker. An attempt counts once for each connection it called
  through, by how the last such call ended. It counts as unavailable
  when the call surely did nothing or no answer came back: a `5xx`, a
  `408`, a `429`, a timeout, or a call that never left. It counts as
  answered, ending the run of failures, when the provider answered at
  all, a refusal or a rejected credential included. A delivery attempt
  counts once however many times the connection's own `retry:` sent its
  call ([Calling](calling.md)), by how the last of them
  ended. So one a retry got through ends the run, and five attempts of
  three refused sends each open the breaker, not fifteen sends. Only
  calls to the connection's own driver count, never a fake's or a
  recording's answer, and only calls a delivery makes: a route's calls
  are its caller's to retry.
- **Which deliveries are held.** A subscriber's delivery, when its action
  calls out through the connection, itself or through an action it
  calls, or when it hands its event to a capability the connection
  fills. An action a batch's run calls runs after the commit, not in the
  delivery, so it doesn't count. Each delivery is held at its claim,
  before it runs: it becomes `held` and not due, the attempt its claim
  counted is given back, and its connection and the time it was held
  from are kept on its record.
- **The pause and the probe.** The first pause is a minute. Once it has
  passed, the first delivery claimed for the connection is the probe,
  and the rest stay held. The tick's sweep wakes the delivery held
  longest to be the probe, so a probe goes even when nothing new
  arrives. A probe the provider answers closes the breaker and releases
  every delivery it held, due at once. A probe that finds the provider
  unavailable opens the breaker again for twice the last pause, at most
  ten minutes. A probe whose run no longer calls the provider gives the
  probe back, and one whose worker is lost loses it when its claim's
  lease ends. A delivery already running when the breaker opened
  finishes, and its answer closes the breaker too.
- **The budget.** Being held spends no attempt, and the time held doesn't
  count towards `within:`. A delivery's `within:` counts from its event,
  or from an operator's retry, moved on by the time it was held. An
  operator's retry gives a fresh budget, the time held included.
- **Where it is kept.** In `jadpo_circuit_breakers`
  ([System tables](system-tables.md)), in every store that keeps
  deliveries, each store's breakers holding its own deliveries. Every
  change is one conditional statement, so the breaker holds across
  processes and on Workers. The sweep also releases any delivery whose
  breaker has closed, which a release racing a hold can leave.
- **The alert.** The tick raises one alert for each outage,
  `connection_unavailable`, recorded on the breaker's row as a
  schedule's alerts are on its state, so of several instances only one
  raises it. It is raised again only after the breaker has closed and
  opened again. Its log event is
  `{"event": "delivery.alert", "alert": "connection_unavailable", "connection": …, "since": …, "held": …}`,
  and `jadpo dev` prints it with the schedules' alerts.

## Not built yet

The jitter and `Retry-After` are built for a call's own retries within
one attempt ([Calling](calling.md)). Between attempts of
a delivery, the next attempt waits only its backoff.