# What an operator sees

Background work that goes wrong waits for a person rather than
disappearing. Every delivery and schedule run has a state, and the
operator commands show the ones that need someone:

| State | Meaning | The operator can |
|---|---|---|
| `pending` | Waiting to run, or waiting to retry | Cancel it |
| `running` | A worker has it | Wait; a worker that dies loses it after its time limit |
| `held` | Held while a provider it calls is unavailable ([When a provider keeps failing](retries-and-giving-up.md#when-a-provider-keeps-failing)) | Wait; a probe that gets through releases it |
| `succeeded` | Done | Nothing |
| `failed` | Gave up, or a failure that is not retried (the dead-letter list) | Retry it once the cause is fixed, or dismiss it |
| `outcome_unknown` | An outside call may have happened | Check with the provider, then mark it done or not done; "not done" retries it |
| `diverged` | A rerun of an action with several changing calls no longer matches its record | Deploy a fix and retry, or abandon it |

Each failed or unknown item keeps the event, the subscriber, every
attempt with its failure code and internal fields (logged by the rules
in [Logging](logging.md)), and the request that emitted it.
A `failed` or `outcome_unknown` item writes a log event and counts
towards an alert. The commands are described in
[Tooling](the-jadpo-command.md).

## `jadpo deliveries`

`jadpo deliveries` lists the items that need an operator, `failed` and
`outcome_unknown`, newest first, 20 to a page. `--state` lists those
`pending`, `running`, `held`, `dismissed` or `cancelled` instead. A
connection whose circuit breaker holds deliveries is an alert line on
the list and on the held ones, with when it became unavailable, how many
it holds and when one is tried again.

You act on an item by its id: the last 8 hex digits of its UUID, or more.
The last digits are the random part of a version 7 UUID, whose first
digits are its time.

- **Two more states.** `dismissed` and `cancelled` mean an operator set
  the delivery aside. Neither runs again. Each is kept 30 days after the
  operator's last action on it, with its event, and then the tick
  deletes it ([The tick](schedules.md#the-tick)).
- **What each command does to a delivery.**
  - `retry` (from `failed`) makes it `pending` and due now, with a fresh
    budget: its attempts counted from none, and its `within:` from the
    retry (the record's `retried_at`), not from its event. So one retried
    after its window still runs. The tick's sweep then delivers it.
  - `dismiss` (from `failed`) and `cancel` (from `pending`) set it
    aside.
  - `resolve --done` (from `outcome_unknown`) marks it `succeeded`, and
    `resolve --not-done` retries it as `retry` does.

  A delivery whose outcome is unknown is never retried blindly: `retry`
  is refused and points at `resolve`. Each command is one statement that
  changes the record only from the state it was read in, so of two
  operators acting at once, one acts and the other is told it changed.
- **Schedule runs.** A schedule's item is its latest run, a skipped tick
  aside, when that run failed or its outcome is unknown. It stays until
  a later run ends it or an operator deals with it, so a schedule failing
  every 15 minutes is one item, not a backlog. Its earlier runs stay in
  `jadpo schedules history`. `retry`, and `resolve --not-done`, ask the
  app's tick to run the schedule by hand and wait for its record, as
  `jadpo schedules run` does against a deployment
  ([Running a schedule by hand](what-an-operator-sees.md#running-a-schedule-by-hand)). A run
  asked for that way is `pending` until a tick takes it, and `cancel`
  withdraws it. `dismiss` and `resolve --done` leave the run record as it
  is, and settle the item in the log.
- **Who did it.** Every command writes a row of `jadpo_operator_actions`
  ([System tables](system-tables.md)) in the store the item is
  in, in the same statement as its change. The row holds when, which
  command, the item and its state then, the name the operator's machine
  gives (`USER`, else `LOGNAME` or `USERNAME`), and the database role
  PostgreSQL authenticated, which the statement reads itself
  (`session_user`), so it can't be faked. The CLI shows both under the
  result: `logged  sam · role app_owner`. A row is kept 30 days, as a
  run record is, and after that until its delivery or run is gone. So an
  item still on record keeps the last thing done to it, and a dismissed
  run stays dismissed; then the tick deletes the row. Jadpo has no
  operator accounts of its own, as it has no reviewer accounts
  ([Review and sign-off](review-and-sign-off.md)): the
  database role is the identity.
- **Which databases.** The commands read every store that holds an
  entity, which is where deliveries are, and `schedules: { store: }`.
  Each is found as `jadpo schedules` finds its database: `--url` (with
  `--store` when there are several), else the connection's setting,
  else a running `jadpo dev`'s database. The commands run none of the
  app's code, and read none of its settings beyond those addresses.

## Unfinished multi-step runs

An action with several changing calls that ends without committing
(failed, diverged, parked, or a route whose key lapsed) is an operator
item too. The operator sees what took effect at each provider, step by
step with its answer, and can retry it (with `--not-done`, under a new
key generation) or abandon it. Nothing about it is quietly dropped. An
item is kept 30 days, then only its step list.

- **Listing them.** A delivery that `diverged` is listed by default
  beside those that failed or are in doubt, and `--state` also takes
  `diverged`, `unfinished` and `abandoned`. A route's record whose
  client didn't retry within a day of its last run is an item of its
  own, `unfinished`, with the action as its target.
- **`jadpo deliveries show <id>`** lists an item and, when its action
  keeps a record, each step: its operation; `answered`, `failed`,
  `no answer` or `answer dropped`; when it was sent; and its key's
  generation once replaced. It never shows a request, an answer or a
  key. Then it says the one thing to do, a decision for a step in doubt.
- **`retry`** takes a delivery that failed or diverged, its record's
  generation counted again from none, as its attempts are.
- **`retry --not-done`** (and `resolve --not-done`) sends the last step
  again under its next key generation when it has no answer or failed,
  its answer cleared.
- **`abandon`** ends a delivery that failed, diverged or is in doubt, or
  a route's record, keeping its step list. A delivery with a record is
  abandoned, not dismissed or resolved as done, since its action's
  writes never committed.

Each change is one statement with its log row, when the record is in
the delivery's store. The tick drops a record's answers 30 days after
its last run, and deletes an abandoned record, and its delivery, 30
days after it was abandoned. A route's client retrying under an
abandoned record's key is answered `500`, and nothing is sent. The
provider's `new_key_on:` answer sends a step again at once, once, under
its next key generation, which the record keeps first.
`jadpo runs unfinished --build <build id>` lists the records and batch
runs a new build changes.

## Batch runs and queue messages

Batch runs and queue messages show here as well. Each batch run shows
its state, its progress against the file's size, its problems and an
estimated finish, which `batches.status` gives the app. Each queue
message shows when it was parked because its type isn't listed or
because it kept failing.

## Watching schedules

A schedule that stops running usually produces no error at all. Jadpo
records and checks every part of a schedule's life, so you find out.

- **Every run is recorded** in a system table, `jadpo_schedule_runs`:
  its due time, start and end; its outcome (`running`, `succeeded`,
  `failed`, or `skipped` with `paused` or `lease_held`) with its
  failure's code and kind, never a value; the records a purge deleted;
  the events the run kept once it committed; and its `traceparent`, so a
  run opens as one trace. Records are kept 30 days.
- **Every tick is recorded too,** skipped ones included, with why (lease
  held, or paused). Each state row keeps the last tick that reached it.
- **`jadpo schedules`** lists each schedule with its last run and next
  due time (or `‖ paused`), the alerts that hold, and a footer. A run
  holding the lease shows as the last run, `● running`, with its start.
  Recent failures are the failed runs since the last success, shown
  beside a failed last run (`· 3 in a row`). The command succeeds
  whatever it lists, a failed last run or an alert included, as other
  listing commands do, and `--json` carries the alerts for a health
  check.
- **`pause` and `resume`** set the state row's `paused`, making the row
  as the first tick would if none has yet. A run under way carries on.
  Pausing a paused schedule, or resuming one that isn't paused, changes
  nothing and says so.
- **`history <name>`** gives every record, runs and skipped ticks,
  newest first, 20 to a page (`--page <n>`), grouped by UTC day. Then it
  gives figures over the records kept: runs, failed runs and skipped
  ticks, the median and longest duration, and the median and greatest
  delay (how late each run started).
- **Which database the commands read.** The one the schedules' state is
  in: the one `--url` names, else the address its connection's setting
  gives ([Database and other connection URLs](database-and-other-connection-urls.md)),
  else the one a running `jadpo dev` started for the session. `jadpo dev`
  notes that one in `build/dev/stores.json`, and removes it when it
  stops. There is no `--env` flag: an environment is its configuration.
  Times are UTC, as schedules count them.
- **`jadpo why <schedule>`** answers "why didn't it run?", in the shape
  of every `jadpo why` answer ([Debugging and review](debugging-and-review.md)).
  It gives the first of these that holds: paused, with the due times
  skipped since; no tick since a time; a lease held past its end; a run
  still holding it; a tick skipped since the last run; the last run
  failed, with its code, kind, failures before it, next due time and
  trace; it succeeded; not due yet; or no tick has seen it. Its `.jadpo`
  place is the schedule's declaration, or its doc comment's line when it
  has one. Other alerts follow. The answer is the runtime's, as data (the
  answer, the alerts that hold, a standing request), which the CLI says
  in words, and the console and the MCP server read too.
- **Metrics.** Each run's lateness (its start minus its due time) and
  duration, measured on the monotonic clock from the run's own time, are
  carried by the tick's result and `jadpo dev`'s `--json` lines, and by
  `history` for each schedule. So are skipped ticks and failures for each
  schedule ([Telemetry](built-in-services.md)). Items per run wait
  for a schedule run that counts them, as a purge does.
- **Alerts with no setup.** Each alert is a condition worked out from the
  rows at a given time:
  - ticks stopped, when the last tick any schedule received is older
    than twice the host's tick period;
  - a lease past its end whose run never ended;
  - a run that has taken, or took, 80% of the 15-minute limit;
  - three or more failed runs since the last success;
  - and, unless paused, no success for over twice `every:`, counted from
    the last success's start (kept on the state row as
    `last_succeeded_at`, so a monthly schedule isn't lost with its
    30-day records) or else from its first due time.

  Each is a log event, `{"event": "schedule.alert", "alert": …}`.
- **The tick raises the alerts.** Once a tick's schedules have run, it
  works out the alerts that hold for each schedule, as any reader does.
  It records them on the schedule's state row (`alerts`: each alert's id
  and when it was raised), in one statement that changes the row only
  while it is what the tick read. Of several instances ticking, or
  several Workers cron invocations, the one whose statement changed it
  raises what newly holds, and the others raise nothing. An alert is
  raised once until it stops holding; changes to its details, such as a
  growing failure count, change nothing. One that stops and starts again
  is raised again. A tick never raises "ticks stopped", since a tick
  arriving proves they haven't. `jadpo dev` prints the alerts its tick
  raised, so a restarted session doesn't print one already raised.
  Readers still work the alerts out when they read: `jadpo schedules`
  and `jadpo why` show those that hold at their own time, since a
  stopped app raises nothing, and its operator must still see its
  alerts.

## Running a schedule by hand

`jadpo schedules run <name>` runs a schedule now, paused or not, under
its lease. While a run holds the lease, it is refused with
`schedules.lease_held` and nothing is done. Its record's reason is
`by_hand`, and the grid isn't moved. Where it runs depends on the
database:

- **Against `jadpo dev`**, meaning the database a running `jadpo dev`
  noted for that store in `build/dev/stores.json`, it runs the schedule
  straight away in the command's own process. It uses the app's settings
  from the environment over `.env.local`, runs as the system reaching
  every tenant at the command's time, with its due time when it started,
  and shows the outcome.
- **Against a deployment**, meaning any other database, including one
  `jadpo dev --url` was given, it reads none of the app's settings and
  builds none of its code, so a deployment's settings and secrets never
  leave its servers. It writes a new run's id to the state row's
  `run_requested` column (making the row as the first tick would if it
  doesn't exist yet), and waits for the record with that id. A request
  made while another stands joins it, so several requests before a tick
  make one run. The tick reads `run_requested` with each state row. A
  schedule's due run goes first, and a request runs on the same tick
  only when that left the lease free (the due tick was skipped as
  paused, or none was due) and no live lease holds it. So a tick runs a
  schedule once at most. It takes the lease in one conditional statement
  that also clears the request, so of two instances only one runs it.
  Its record's id is the one requested, its due time is the tick's, and
  what it emits is delivered after it.
- **The wait.** The command reads the record every 200 ms, timing the
  wait on its own clock, so the two machines' clocks never have to
  agree. A tick has to take the request within twice the period the app
  ticks at, which is the ticks-stopped alert's limit: 2 minutes for a
  minute's tick. While another run holds the lease it waits longer, up
  to the longest lease, 16 minutes, and that again, since it can't know
  the deployment's budget. Otherwise it withdraws the request, unless a
  tick took it meanwhile, and fails with `schedules.no_tick`: "No tick
  picked up `<name>`'s run in 2 min", its next step to check the app is
  running and run it again. Withdrawing keeps a request from running
  whenever the app next starts, perhaps much later. Once a tick takes
  the request, the command waits for the run to end. A run still going
  past the lease's length was lost with its worker
  (`schedules.run_failed`). A command interrupted while it waits leaves
  the request, which the next tick runs.
- **What it shows.** The run's outcome from its record, on the screens
  above: ``✓ The app's tick ran `<name>` in 12 ms, emitting 1 event``, or
  the failed run's block. A run that fails exits 1 with
  `schedules.run_failed`. `--json` carries `"requested": true`, and
  while the command waits, the live line says so. `history` lists a
  requested run once a tick has started it, as a run by hand (due `–`,
  `· by hand`), and `jadpo dev` prints it with `· by hand`. While a
  request waits, `jadpo why` adds `request  a run by hand, waiting for
  the app's next tick`, and `jadpo schedules --json` gives each
  schedule's `runRequested`.

## Not built yet

Three things aren't built yet: names for instances, so the lease holder
shows as a run rather than an instance; sending the metrics and alerts
to a monitoring tool; and the console's timeline of each schedule's runs
and ticks ([Debugging and review](debugging-and-review.md)).