What an operator sees

Background work that goes wrong waits for a person rather than disappearing. Every delivery and schedule run has a state, and the operator commands show the ones that need someone:

Each failed or unknown item keeps the event, the subscriber, every attempt with its failure code and internal fields (logged by the rules in Logging), and the request that emitted it. A failed or outcome_unknown item writes a log event and counts towards an alert. The commands are described in Tooling.

jadpo deliveries

jadpo deliveries lists the items that need an operator, failed and outcome_unknown, newest first, 20 to a page. --state lists those pending, running, held, dismissed or cancelled instead. A connection whose circuit breaker holds deliveries is an alert line on the list and on the held ones, with when it became unavailable, how many it holds and when one is tried again.

You act on an item by its id: the last 8 hex digits of its UUID, or more. The last digits are the random part of a version 7 UUID, whose first digits are its time.

• Two more states. dismissed and cancelled mean an operator set the delivery aside. Neither runs again. Each is kept 30 days after the operator's last action on it, with its event, and then the tick deletes it (The tick).

• What each command does to a delivery.

  • retry (from failed) makes it pending and due now, with a fresh budget: its attempts counted from none, and its within: from the retry (the record's retried_at), not from its event. So one retried after its window still runs. The tick's sweep then delivers it.
  • dismiss (from failed) and cancel (from pending) set it aside.
  • resolve --done (from outcome_unknown) marks it succeeded, and resolve --not-done retries it as retry does.

  A delivery whose outcome is unknown is never retried blindly: retry is refused and points at resolve. Each command is one statement that changes the record only from the state it was read in, so of two operators acting at once, one acts and the other is told it changed.

• Schedule runs. A schedule's item is its latest run, a skipped tick aside, when that run failed or its outcome is unknown. It stays until a later run ends it or an operator deals with it, so a schedule failing every 15 minutes is one item, not a backlog. Its earlier runs stay in jadpo schedules history. retry, and resolve --not-done, ask the app's tick to run the schedule by hand and wait for its record, as jadpo schedules run does against a deployment (Running a schedule by hand). A run asked for that way is pending until a tick takes it, and cancel withdraws it. dismiss and resolve --done leave the run record as it is, and settle the item in the log.

• Who did it. Every command writes a row of jadpo_operator_actions (System tables) in the store the item is in, in the same statement as its change. The row holds when, which command, the item and its state then, the name the operator's machine gives (USER, else LOGNAME or USERNAME), and the database role PostgreSQL authenticated, which the statement reads itself (session_user), so it can't be faked. The CLI shows both under the result: logged sam · role app_owner. A row is kept 30 days, as a run record is, and after that until its delivery or run is gone. So an item still on record keeps the last thing done to it, and a dismissed run stays dismissed; then the tick deletes the row. Jadpo has no operator accounts of its own, as it has no reviewer accounts (Review and sign-off): the database role is the identity.

• Which databases. The commands read every store that holds an entity, which is where deliveries are, and schedules: { store: }. Each is found as jadpo schedules finds its database: --url (with --store when there are several), else the connection's setting, else a running jadpo dev's database. The commands run none of the app's code, and read none of its settings beyond those addresses.

Unfinished multi-step runs

An action with several changing calls that ends without committing (failed, diverged, parked, or a route whose key lapsed) is an operator item too. The operator sees what took effect at each provider, step by step with its answer, and can retry it (with --not-done, under a new key generation) or abandon it. Nothing about it is quietly dropped. An item is kept 30 days, then only its step list.

• Listing them. A delivery that diverged is listed by default beside those that failed or are in doubt, and --state also takes diverged, unfinished and abandoned. A route's record whose client didn't retry within a day of its last run is an item of its own, unfinished, with the action as its target.
• jadpo deliveries show <id> lists an item and, when its action keeps a record, each step: its operation; answered, failed, no answer or answer dropped; when it was sent; and its key's generation once replaced. It never shows a request, an answer or a key. Then it says the one thing to do, a decision for a step in doubt.
• retry takes a delivery that failed or diverged, its record's generation counted again from none, as its attempts are.
• retry --not-done (and resolve --not-done) sends the last step again under its next key generation when it has no answer or failed, its answer cleared.
• abandon ends a delivery that failed, diverged or is in doubt, or a route's record, keeping its step list. A delivery with a record is abandoned, not dismissed or resolved as done, since its action's writes never committed.

Each change is one statement with its log row, when the record is in the delivery's store. The tick drops a record's answers 30 days after its last run, and deletes an abandoned record, and its delivery, 30 days after it was abandoned. A route's client retrying under an abandoned record's key is answered 500, and nothing is sent. The provider's new_key_on: answer sends a step again at once, once, under its next key generation, which the record keeps first. jadpo runs unfinished --build <build id> lists the records and batch runs a new build changes.

Batch runs and queue messages

Batch runs and queue messages show here as well. Each batch run shows its state, its progress against the file's size, its problems and an estimated finish, which batches.status gives the app. Each queue message shows when it was parked because its type isn't listed or because it kept failing.

Watching schedules

A schedule that stops running usually produces no error at all. Jadpo records and checks every part of a schedule's life, so you find out.

• Every run is recorded in a system table, jadpo_schedule_runs: its due time, start and end; its outcome (running, succeeded, failed, or skipped with paused or lease_held) with its failure's code and kind, never a value; the records a purge deleted; the events the run kept once it committed; and its traceparent, so a run opens as one trace. Records are kept 30 days.

• Every tick is recorded too, skipped ones included, with why (lease held, or paused). Each state row keeps the last tick that reached it.

• jadpo schedules lists each schedule with its last run and next due time (or ‖ paused), the alerts that hold, and a footer. A run holding the lease shows as the last run, ● running, with its start. Recent failures are the failed runs since the last success, shown beside a failed last run (· 3 in a row). The command succeeds whatever it lists, a failed last run or an alert included, as other listing commands do, and --json carries the alerts for a health check.

• pause and resume set the state row's paused, making the row as the first tick would if none has yet. A run under way carries on. Pausing a paused schedule, or resuming one that isn't paused, changes nothing and says so.

• history <name> gives every record, runs and skipped ticks, newest first, 20 to a page (--page <n>), grouped by UTC day. Then it gives figures over the records kept: runs, failed runs and skipped ticks, the median and longest duration, and the median and greatest delay (how late each run started).

• Which database the commands read. The one the schedules' state is in: the one --url names, else the address its connection's setting gives (Database and other connection URLs), else the one a running jadpo dev started for the session. jadpo dev notes that one in build/dev/stores.json, and removes it when it stops. There is no --env flag: an environment is its configuration. Times are UTC, as schedules count them.

• jadpo why <schedule> answers "why didn't it run?", in the shape of every jadpo why answer (Debugging and review). It gives the first of these that holds: paused, with the due times skipped since; no tick since a time; a lease held past its end; a run still holding it; a tick skipped since the last run; the last run failed, with its code, kind, failures before it, next due time and trace; it succeeded; not due yet; or no tick has seen it. Its .jadpo place is the schedule's declaration, or its doc comment's line when it has one. Other alerts follow. The answer is the runtime's, as data (the answer, the alerts that hold, a standing request), which the CLI says in words, and the console and the MCP server read too.

• Metrics. Each run's lateness (its start minus its due time) and duration, measured on the monotonic clock from the run's own time, are carried by the tick's result and jadpo dev's --json lines, and by history for each schedule. So are skipped ticks and failures for each schedule (Telemetry). Items per run wait for a schedule run that counts them, as a purge does.

• Alerts with no setup. Each alert is a condition worked out from the rows at a given time:

  • ticks stopped, when the last tick any schedule received is older than twice the host's tick period;
  • a lease past its end whose run never ended;
  • a run that has taken, or took, 80% of the 15-minute limit;
  • three or more failed runs since the last success;
  • and, unless paused, no success for over twice every:, counted from the last success's start (kept on the state row as last_succeeded_at, so a monthly schedule isn't lost with its 30-day records) or else from its first due time.

  Each is a log event, {"event": "schedule.alert", "alert": …}.

• The tick raises the alerts. Once a tick's schedules have run, it works out the alerts that hold for each schedule, as any reader does. It records them on the schedule's state row (alerts: each alert's id and when it was raised), in one statement that changes the row only while it is what the tick read. Of several instances ticking, or several Workers cron invocations, the one whose statement changed it raises what newly holds, and the others raise nothing. An alert is raised once until it stops holding; changes to its details, such as a growing failure count, change nothing. One that stops and starts again is raised again. A tick never raises "ticks stopped", since a tick arriving proves they haven't. jadpo dev prints the alerts its tick raised, so a restarted session doesn't print one already raised. Readers still work the alerts out when they read: jadpo schedules and jadpo why show those that hold at their own time, since a stopped app raises nothing, and its operator must still see its alerts.

Running a schedule by hand

jadpo schedules run <name> runs a schedule now, paused or not, under its lease. While a run holds the lease, it is refused with schedules.lease_held and nothing is done. Its record's reason is by_hand, and the grid isn't moved. Where it runs depends on the database:

• Against jadpo dev, meaning the database a running jadpo dev noted for that store in build/dev/stores.json, it runs the schedule straight away in the command's own process. It uses the app's settings from the environment over .env.local, runs as the system reaching every tenant at the command's time, with its due time when it started, and shows the outcome.
• Against a deployment, meaning any other database, including one jadpo dev --url was given, it reads none of the app's settings and builds none of its code, so a deployment's settings and secrets never leave its servers. It writes a new run's id to the state row's run_requested column (making the row as the first tick would if it doesn't exist yet), and waits for the record with that id. A request made while another stands joins it, so several requests before a tick make one run. The tick reads run_requested with each state row. A schedule's due run goes first, and a request runs on the same tick only when that left the lease free (the due tick was skipped as paused, or none was due) and no live lease holds it. So a tick runs a schedule once at most. It takes the lease in one conditional statement that also clears the request, so of two instances only one runs it. Its record's id is the one requested, its due time is the tick's, and what it emits is delivered after it.
• The wait. The command reads the record every 200 ms, timing the wait on its own clock, so the two machines' clocks never have to agree. A tick has to take the request within twice the period the app ticks at, which is the ticks-stopped alert's limit: 2 minutes for a minute's tick. While another run holds the lease it waits longer, up to the longest lease, 16 minutes, and that again, since it can't know the deployment's budget. Otherwise it withdraws the request, unless a tick took it meanwhile, and fails with schedules.no_tick: "No tick picked up <name>'s run in 2 min", its next step to check the app is running and run it again. Withdrawing keeps a request from running whenever the app next starts, perhaps much later. Once a tick takes the request, the command waits for the run to end. A run still going past the lease's length was lost with its worker (schedules.run_failed). A command interrupted while it waits leaves the request, which the next tick runs.
• What it shows. The run's outcome from its record, on the screens above: ✓ The app's tick ran `<name>` in 12 ms, emitting 1 event, or the failed run's block. A run that fails exits 1 with schedules.run_failed. --json carries "requested": true, and while the command waits, the live line says so. history lists a requested run once a tick has started it, as a run by hand (due –, · by hand), and jadpo dev prints it with · by hand. While a request waits, jadpo why adds request a run by hand, waiting for the app's next tick, and jadpo schedules --json gives each schedule's runRequested.

Not built yet

Three things aren't built yet: names for instances, so the lease holder shows as a run rather than an instance; sending the metrics and alerts to a monitoring tool; and the console's timeline of each schedule's runs and ticks (Debugging and review).