Structured values

A field whose type is one of your app's enums with data, objects, lists, sets or maps is kept in one jsonb column. It holds exactly what json.write writes for its type, its contract JSON, as the standard collections do. It is read back as a request's body is, with every rule of its type and of the field: one codec, the route's.

type Tag = Text {    max_length: 20}type ParcelWeight = Mass {    min: 0g    max: 100kg}/// What is printed on a parcel.type Label = Object {    line: Tag    note: Tag?}/// How a parcel leaves.type Delivery = Enum {    pickup    courier: {        company: Tag        weight: ParcelWeight    }}entity Parcel {    id: Uuid    label: Label    tags: List<Tag>    weights: Map<Tag, ParcelWeight>    delivery: Delivery    second_label: Label?  // `NULL` when it is `none`}

second_label is NULL in the database when it is none. A parcel going by courier keeps {"tag": "courier", "company": "Swift", "weight": {"value": "0.92", "unit": "kg"}} in delivery, and one to be picked up keeps "pickup". Its weights are {"box": {"value": "0.12", "unit": "kg"}, "loaf": {"value": "0.8", "unit": "kg"}}, the keys in their canonical order.

A set, and a map's keys, are written in their canonical order, and a variant without data as its name, as on the wire. PostgreSQL keeps jsonb in its own key order and spacing, so the text isn't byte for byte what was written, but it reads back ==.

• What a stored value may hold: what that JSON holds. That is values built on the built-in and standard types, enums, and Objects made of them, in lists, sets and maps, a type that holds itself included.
• What it may not hold. An Input, an Output, an event, a failure, an email, a message or a prompt inside one, a map keyed by a type with no single text form, a format and a plural count are each data.field_without_json (under "Errors"). A secret value, an API token or a service key, an entity and text for people are refused inside a stored value as they are in a column of their own (type.secret_exposed, type.auth_credential_stored, data.entity_in_field, type.presentation_text_persistence). A PasswordHash is kept, as a field of its own is; every Input and Output that would carry one off the server is already refused (Standard library).
• A value that doesn't read is a fault. A row edited outside Jadpo whose JSON breaks a rule, isn't its type, or is past the limits below, is a fault when it is read. The fault names the rule and where in the value it broke (max on weight), never the value.
• none is the column's NULL, never JSON's null. A JSON null in the column is a row written outside Jadpo, and a fault. Inside the value, a part that is none is JSON's null, as the wire writes it.
• No query compares or sorts one, as for the standard collections; the field is read whole wherever its record is. An order by one is query.order_field. A lifecycle tests one only with == none or != none, and a transition or initial: sets one only to none (data.lifecycle_condition, data.lifecycle_value), since PostgreSQL compares jsonb by its own rules rather than Jadpo's ==. Any other value is written by an action, through the codec.
• An id inside is a value, not a reference. A User.id inside a stored value has no foreign key and no on_delete:, as an event's payload has none (Jobs and events). A link the database must keep is a field of its own, or a link entity.
• Policy is the field's. A field's policy: covers its whole stored value, as any field's does. A field of an Object takes no policy: (type.unknown_setting), and a part is read only by reading its field, so Policy's proofs hold it as they hold the field.
• Limits. A stored value is at most 1 MiB of JSON, the most a request's body may be (HTTP), nested at most 128 arrays and objects deep, the most a request's body is read to. So whatever is kept could have arrived in a request and always reads back. Writing a larger or deeper one is a fault naming the limit, as is reading one written outside Jadpo. PostgreSQL's own limit on jsonb is far above both.
• Encodings. The column holds the contract's JSON, so a standard type inside it is pinned by that type's golden file and the release's encoding (Changing a standard type's encoding). Like an entity's columns, a JSON column records no encoding of its own: a release that changed what a standard type's JSON holds would come with the migration that rewrites stored values.
• Event payloads use the same codec. Every event type is numbered in the plan's JSON table, by the identity key the outbox's event column records, so the outbox writes and reads a payload through the one codec, with the type's rules applied on read (Jobs and events). No job takes arguments yet (a schedule's run: takes none). A batch's saved values are numbered in the same table by their types.

Migrating structured values

Each migration's snapshot records what a JSON column's type reads (its allows holds json):

• the value's shape: a value built on a built-in or standard type, with the bounds, lengths, patterns and places along its chain and any other rule as text; a part that may be none; a list, a set or a map with its lengths;
• each of your app's objects and enums it names, by name, with an object's fields and rules across its fields, and an enum's variants and their fields.

The field's own settings count, as they do for any column. So changing the stored type, or the field's own rules, is drift like any other.

When a change runs nothing. A change runs nothing only when the new type can be shown to read every value the old one wrote: the same built-in or standard type and places, every other rule the same, no bound or length tighter or new, no pattern new, a part made optional but never required, a list never made a set, rules across fields only removed, and variants only added, each with the fields it had. An object's JSON is read member for member, every field required and no other allowed, so adding a field (even one that may be none) or removing one may read less. A type renamed reads as its shape.

Any other change may read less. The migration checks every stored value in batches, before anything else in the migration, and jadpo migrate apply refuses it while any value doesn't read. A backfill: rewrites those values instead; you need to write one only then. Here Tag gets shorter, and parcels whose label no longer fits are relabelled:

type Tag = Text {    max_length: 12}

database/migrations/20261020T1401_shorten_tags/changes.jadpo:

// changes.jadpo/// Shortens tags, relabelling parcels whose line no longer fits.migration {    Parcel.label: {        backfill: Label {            line: "Unlabelled"            note: none        }    }}

Without the decision, the migration's SQL reads each batch of labels (-- check Parcel.label: in Jadpo, 1000 rows a batch, …). With it, it rewrites those the new Tag doesn't read (-- rewrite Parcel.label: …) and leaves the rest as they are.

• The check comes before anything else in the store's section, so a refusal applies nothing of the migration; the migrations before it stay. It reads the rows with a value (NULL is none), 1000 a batch in key order. Each batch is its own read-committed transaction under jadpo_migrations' lock, locking no row and changing nothing, and reads each value with the codec of the field's type now, as a query reads it. It records nothing as it goes: a run stopped during it starts it again, and a refusal clears the migration's progress, so the next run reads every value again. The refusal is migration.apply_failed, naming how many values don't read and the first rule broken and where (`max_length` on `line`), never a value.
• The decision may be left out. No decision is required: jadpo migrate new finishes the migration and writes a changes.jadpo holding only a comment saying how to give one. Writing backfill: there and running jadpo migrate new again replaces that field's check with a rewrite.
• The rewrite runs in batches, as a backfill worked out in Jadpo does (Computed backfills), recorded in the resume marker. Each batch reads the rows with a value, locked FOR NO KEY UPDATE, reads each with the new type, works the backfill out for those that don't read, and writes them back through the codec, leaving the rest as they are. The backfill may be written out or worked out from the row, as any backfill in Jadpo may; a narrowed column's refit outside JSON still takes only a value written out. A backfill still reads no field held as JSON, its own included, whose old type is gone.
• Values written meanwhile. A server still on the old build may write a value the new type doesn't read after the check or the rewrite has passed its row. As with a narrowed column's refit, no lock holds writes to the whole table out for that long, and such a value is a fault when read, as any stored value that doesn't read is.
• A required structured field added to a table with rows takes a backfill:, as any other field does. Its value, written out (a construction such as Label { … }, or a variant) or worked out from the row, has no SQL form. So it is always worked out in Jadpo and written by the codec of the field's type, as a write keeps it, then made required in the same last steps as any backfill.
• Only the latest migration is read with the project's code, as for backfills. An earlier migration's check or rewrite with rows to read stops jadpo migrate apply (Checking Parcel.label for <migration> needs the build it was written in).
• An older snapshot that recorded nothing of what a JSON column read has the next migration check its stored values, rather than recording the type with nothing to do, since nothing shows that the values stored then read now. On an empty table the check reads nothing.
• Output. jadpo migrate new lists "check stored values in batches in Jadpo" or "rewrite values no longer read in batches in Jadpo". The live line says Checking Parcel.label in Jadpo 2000 of 5000 rows. The result says 1200 rows checked in 2 batches in Jadpo or 3 rows rewritten in 2 batches in Jadpo, and --json gives each its work.

Not built yet: a condition on a structured field (toolchain.not_built_yet); a unique: or index over one (toolchain.not_built_yet), since PostgreSQL compares jsonb by its own rules; and an update by id that works a structured value out in the database rather than setting one it is given (toolchain.not_built_yet, as for money).