# How a run is kept

This topic is the detail of how a `batches.each` run is stored, resumed
and checked. Most apps never need it: [Batches](batches-work-over-many-items.md)
says what a batch does. The rules for a file's run and an export's run
build on this one ([How a file's run is kept](file-batch-runs.md),
[How an export's run is kept](export-batch-runs.md)).

## A batch is a page

Each transaction is one page of `from:`, at the size its query's `page:`
reads. So 60 items in pages of 3 commit 20 times.

## Where a run is kept

A run is kept in the store `from:`'s query reads, in three
[system tables](system-tables.md):

- `jadpo_batch_runs` has one row per run: which call started it, the
  shape it was started under, its `key:`, what it saved, the tenant it is
  limited to, how deep in a chain of events it started, the cursor of the
  next page, its counts, its state and its runner's lease;
- `jadpo_batch_problems` holds each problem;
- `jadpo_batch_cancels` holds each cancel asked for.

Only a store some batch walks has these tables. Starting a run writes
that store, and `each:`'s and `finished:`'s actions may write no other
(`store.action_spans_stores`, [One store per action](one-store-per-action.md)),
so an item's writes and the run's position always commit together. A run
always has its query's store, so it never needs `events: { store: }`.

## Starting

The run's row is written in the starting action's transaction, and the
run is left to start after the response. An action that rolls back
leaves no row, so nothing runs. A start needs no outbox row: the run's
row is as durable, and the tick picks up a start a host dropped, as the
outbox's sweep picks up a delivery.

A start with a `key:` that finds a run of the same call with the same
key, compared by its contract JSON, in the same tenant, gives back that
run, whatever its state and whatever else the call was given. A start
without a `key:` is always a new run.

In an app with a tenant, `key:` is unique for each call, run's tenant and
key, so two tenants' runs with the same key never meet. The row keeps the
tenant: all zeros when the run reaches no tenant's records, or the app
has no tenant. A later start never finds a row with no tenant kept,
since nothing says which tenant its key was given in. A retry then
starts a new run, rather than one tenant being handed another's.

## Saved values

`each:`'s and `finished:`'s arguments may be the item (or the summary), a
field inside it, or a value that reads nothing of it. That last kind is
worked out as the run starts and saved with it. An argument that works
something out of the item is `type.batch_each_invalid`: that work belongs
in the action.

`from:`'s query arguments are saved as a request carries them, so an
`Input` may be among them. What `each:` and `finished:` capture, and
`key:`, are saved as an entity field holds a value, so an `Input` there
is `type.batch_capture_not_storable`. A secret is never saved.

The saved values are a JSON array of each value's own JSON (its type's
contract JSON), with `null` for none. They are kept with the encoding
they were written in (`jadpo_batch_runs.encoding`) and read with the
standard JSON reader, a number keeping the digits it was written with.

## Which call started a run

A run names the `batches.each` call that started it by what the call is,
in identity keys that survive a rename
([Stable identity](stable-identity.md)): the
action starting it, the query `from:` walks, and the action `each:` runs,
as in `action check_stock from query products_to_check each action
check_product`. Adding or moving another call leaves the name as it was.
A second call of the same three in one action adds its place among them
(` #2`).

The run's shape, what it saves and calls, is kept as canonical type
text, naming declared types by their identity keys, with a digest. So a
release that changes the compiler's own internals leaves every running
batch's shape as it was.

## Problems

An item whose action fails is undone on its own and becomes a problem.
Its row is its place in the walk, counting from 1. Its column is empty,
since a query's item has no columns (the file helpers fill it). Its rule
is the failure's code.

An item whose action faults with a defect or an overflow is a problem
too, undone on its own as a failure is. Its rule is the fault's
classification, `internal_fault` or `int_overflow`.

Some faults belong to no single item: a store that can't be reached, a
deadline, or a change made by another transaction. These undo the whole
page. A page undone by another transaction's change runs again, up to
three times; otherwise the run waits for the tick.

## Items that make outside calls

When `each:`'s action makes outside calls, itself or through an action it
calls, each item is a batch of its own, so no call goes out while a
transaction is open ([All-or-nothing actions](all-or-nothing-actions-and-concurrent-changes.md)).
Each item goes like this:

1. The run's place is read in a statement of its own, locking nothing.
2. The item is read in a snapshot: a query's page of one item, or a
   file's next row. Its first call closes the snapshot.
3. Its action runs as the system, as a subscriber's delivery does.
4. In the transaction its writes are in, the run's row is locked and
   held to the position the item was read at. The item's problem, if it
   failed, and the run's next position commit with its writes.

A query's run is then at the item's own cursor, and a file's run just
after its row, so a run resumed after a crash meets the same item at the
same place. Each call's key is made from the run and that place (the
item's cursor, or where its row begins in the file), so the item's calls
go out again under the keys they were first sent with. A change by
another transaction runs the item again, up to three times, as it would
a page.

A `finished:` whose action makes outside calls runs the same way: its
calls are keyed from the run, and the run's end commits with its writes.

An unkeyed call that may have acted (`outcome_unknown`) is the item's
problem, and is never sent again. One lost with its runner waits for the
step record that several changing calls keep, as a subscriber's does
([Several changing calls in one action](several-changing-calls-in-one-action.md)).

## Who runs it

Each page's read, its items and `finished:` act as the system
([Background work and tenants](background-work-and-tenants.md)), and read
the clock as their page began. Each page begins at repeatable read when
`each:`'s or `finished:`'s action loads a record, and at read committed
otherwise.

A run keeps how deep in a chain of events the action that started it
was, and its items run at that depth. So what they emit is one deeper
than the event that started the run, and a loop through a batch stops at
16 deep, as any loop of events does. Each item's action, and
`finished:`'s, may emit as many events as one action may, 1,000, however
many the items before it in the page emitted.

## Finishing

`finished:` runs once after the last page, in a transaction of its own.
It is given the `ImportSummary` with every problem in row order, and the
run becomes `finished` in the same transaction. If its action fails,
what it wrote is undone and the run is `failed` with the failure's code.
A cancelled run doesn't run `finished:`.

## Cancelling

`batches.cancel` writes a row of its own in the calling action's
transaction and never touches the run's row, so it never waits for the
page in hand. The next page finds it and ends the run `cancelled`. Until
then `batches.status` says `running`.

## Status

`batches.status` reads the counts and the state in the calling action's
transaction. A `BatchRun` that no store keeps is a fault. Both
`batches.status` and `batches.cancel` find a run by its `id` alone, so
they need the app's runs in one store.

## The estimated total

A query's run counts its query as its first page is read, in that page's
transaction, as the system limited to the run's tenant. It runs the
query's statement with no page limit or cursor, so the count is what the
walk would meet if nothing changed. Later pages keep that count.

A file's run works its estimate out with each page: the items done so
far, scaled by the file's size over how far into the file they end, and
never fewer than are done. For a gzip file, the size is what it is
unpacking to, at the rate so far.

`estimated_total` is `none` until then. Once the run has finished it is
the items done, and once it failed or was cancelled, the last estimate.
It is kept in `jadpo_batch_runs.estimated_total`. A count the store
refuses leaves the run without one rather than stopping it. A query whose
records Jadpo checks again after its statement, such as one with `geo.`
conditions ([Geospatial queries](geospatial-queries.md)), has none.

## An order an update can change

A `from:` whose query orders by a field that can be updated, before any
`id`, is `type.batch_order_updatable`. It is reported at the `from:` and
declined on the action that starts the run. A field can't be updated
when it is `id`, is marked `immutable: true`, or is generated with
`Generated.on_create`. What comes after `id`, or after a unique field
that can't be updated, is passed over, since it never moves a record.

The check doesn't look at which updates the program makes, so adding one
elsewhere never makes a run's order wrong after the fact. A policy with
no `update:` doesn't count either, since adding one is a one-line change.

## Carried on over time

A runner holds its run by a lease lasting what is left of its budget
plus a minute ([Running background work](where-events-travel.md#running-background-work)). The
lease is renewed with each page it commits. The runner goes page after
page while its budget lasts, then gives its lease back, and a fresh
invocation carries on from the position that committed. If the budget
ends in the middle of a page, that page is dropped and nothing it did is
kept. A run that couldn't commit a page in a whole budget gives its lease
back and waits for the tick, which picks up every running run whose lease
has lapsed.

## What a deploy changes

A deploy may change a run's call: its query, its actions or what it
saves. Or the run's saved values may be in an encoding this release
doesn't read, such as a later release's after a rollback. Either way,
the run ends `failed` and says why ("its saved values were written in
encoding 2, which this release doesn't read"), rather than reading what
it saved as something else.

A run with no encoding recorded, whose call and shape match, is read
with its values as encoding 1's. A run kept before calls were named by
identity has a name no call now gives, and ends `failed` because "it was
started by a release that named its `batches.each` call another way".
To list such runs before a deploy, run `jadpo runs unfinished`
([The `jadpo` command](the-jadpo-command.md)).

## Kept 30 days

In each hourly cleanup pass, in every store that keeps batch runs, a run
that ended (finished, failed or cancelled) 30 days ago or more is
deleted, with its problems and its cancel. So are cancels as old for
runs no store keeps, which happens when a cancel is asked of a
`BatchRun` a field held after its run was deleted. A run still going is
never deleted, however old.

## Names

`batches` is a standard namespace, so, as with every standard namespace,
no variable may be named `batches`.

## Not built yet

A page type allowing more than the 1,000 items of one batch
([Limits and runtime faults](limits-and-runtime-faults.md)) isn't split
into smaller batches yet; `PageSize` stops at 100. `batches.status` and
`batches.cancel` in an app whose runs are kept in no store, or in more
than one, are `toolchain.not_built_yet`.