# Structured values

A field whose type is one of your app's enums with data, objects, lists,
sets or maps is kept in one `jsonb` column. It holds exactly what
`json.write` writes for its type, its contract JSON, as the standard
collections do. It is read back as a request's body is, with every rule
of its type and of the field: one codec, the route's.

```jadpo
type Tag = Text {
    max_length: 20
}

type ParcelWeight = Mass {
    min: 0g
    max: 100kg
}

/// What is printed on a parcel.
type Label = Object {
    line: Tag
    note: Tag?
}

/// How a parcel leaves.
type Delivery = Enum {
    pickup
    courier: {
        company: Tag
        weight: ParcelWeight
    }
}

entity Parcel {
    id: Uuid
    label: Label
    tags: List<Tag>
    weights: Map<Tag, ParcelWeight>
    delivery: Delivery
    second_label: Label?  // `NULL` when it is `none`
}
```

`second_label` is `NULL` in the database when it is `none`. A parcel going
by courier keeps `{"tag": "courier", "company": "Swift", "weight":
{"value": "0.92", "unit": "kg"}}` in `delivery`, and one to be picked up
keeps `"pickup"`. Its `weights` are `{"box": {"value": "0.12", "unit":
"kg"}, "loaf": {"value": "0.8", "unit": "kg"}}`, the keys in their
canonical order.

A set, and a map's keys, are written in their canonical order, and a
variant without data as its name, as on the wire. PostgreSQL keeps
`jsonb` in its own key order and spacing, so the text isn't byte for byte
what was written, but it reads back `==`.

- **What a stored value may hold:** what that JSON holds. That is values
  built on the built-in and standard types, enums, and `Object`s made of
  them, in lists, sets and maps, a type that holds itself included.
- **What it may not hold.** An `Input`, an `Output`, an event, a failure,
  an email, a message or a prompt inside one, a map keyed by a type with
  no single text form, a format and a plural count are each
  `data.field_without_json` (under "Errors"). A secret value, an API token
  or a service key, an entity and text for people are refused inside a
  stored value as they are in a column of their own
  (`type.secret_exposed`, `type.auth_credential_stored`,
  `data.entity_in_field`, `type.presentation_text_persistence`). A
  `PasswordHash` is kept, as a field of its own is; every `Input` and
  `Output` that would carry one off the server is already refused
  ([Standard library](standard-library.md)).
- **A value that doesn't read is a fault.** A row edited outside Jadpo
  whose JSON breaks a rule, isn't its type, or is past the limits below,
  is a fault when it is read. The fault names the rule and where in the
  value it broke (`max` on `weight`), never the value.
- **`none` is the column's `NULL`,** never JSON's `null`. A JSON `null` in
  the column is a row written outside Jadpo, and a fault. Inside the
  value, a part that is `none` is JSON's `null`, as the wire writes it.
- **No query compares or sorts one,** as for the standard collections; the
  field is read whole wherever its record is. An order by one is
  `query.order_field`. A lifecycle tests one only with `== none` or `!=
  none`, and a transition or `initial:` sets one only to `none`
  (`data.lifecycle_condition`, `data.lifecycle_value`), since PostgreSQL
  compares `jsonb` by its own rules rather than Jadpo's `==`. Any other
  value is written by an action, through the codec.
- **An id inside is a value, not a reference.** A `User.id` inside a
  stored value has no foreign key and no `on_delete:`, as an event's
  payload has none ([Jobs and events](jobs-and-events.md)). A link the
  database must keep is a field of its own, or a link entity.
- **Policy is the field's.** A field's `policy:` covers its whole stored
  value, as any field's does. A field of an `Object` takes no `policy:`
  (`type.unknown_setting`), and a part is read only by reading its field,
  so [Policy](policy.md)'s proofs hold it as they hold the field.
- **Limits.** A stored value is at most 1 MiB of JSON, the most a
  request's body may be ([HTTP](http.md)), nested at most 128 arrays and
  objects deep, the most a request's body is read to. So whatever is kept
  could have arrived in a request and always reads back. Writing a larger
  or deeper one is a fault naming the limit, as is reading one written
  outside Jadpo. PostgreSQL's own limit on `jsonb` is far above both.
- **Encodings.** The column holds the contract's JSON, so a standard type
  inside it is pinned by that type's golden file and the release's
  encoding
  ([Changing a standard type's encoding](standard-type-encodings.md)).
  Like an entity's columns, a JSON column records no encoding of its own:
  a release that changed what a standard type's JSON holds would come with
  the migration that rewrites stored values.
- **Event payloads use the same codec.** Every event type is numbered in
  the plan's JSON table, by the identity key the outbox's `event` column
  records, so the outbox writes and reads a payload through the one codec,
  with the type's rules applied on read
  ([Jobs and events](jobs-and-events.md)). No job takes arguments yet
  (a schedule's `run:` takes none). A batch's saved values are numbered in
  the same table by their types.

## Migrating structured values

Each migration's snapshot records what a JSON column's type reads (its
`allows` holds `json`):

- the value's shape: a value built on a built-in or standard type, with
  the bounds, lengths, patterns and places along its chain and any other
  rule as text; a part that may be `none`; a list, a set or a map with its
  lengths;
- each of your app's objects and enums it names, by name, with an
  object's fields and rules across its fields, and an enum's variants and
  their fields.

The field's own settings count, as they do for any column. So changing the
stored type, or the field's own rules, is drift like any other.

**When a change runs nothing.** A change runs nothing only when the new
type can be shown to read every value the old one wrote: the same built-in
or standard type and places, every other rule the same, no bound or length
tighter or new, no pattern new, a part made optional but never required, a
list never made a set, rules across fields only removed, and variants only
added, each with the fields it had. An object's JSON is read member for
member, every field required and no other allowed, so adding a field (even
one that may be `none`) or removing one may read less. A type renamed
reads as its shape.

**Any other change may read less.** The migration checks every stored
value in batches, before anything else in the migration, and `jadpo
migrate apply` refuses it while any value doesn't read. A `backfill:`
rewrites those values instead; you need to write one only then. Here `Tag`
gets shorter, and parcels whose label no longer fits are relabelled:

```jadpo
type Tag = Text {
    max_length: 12
}
```

database/migrations/20261020T1401_shorten_tags/changes.jadpo:

```jadpo
// changes.jadpo
/// Shortens tags, relabelling parcels whose line no longer fits.
migration {
    Parcel.label: {
        backfill: Label {
            line: "Unlabelled"
            note: none
        }
    }
}
```

Without the decision, the migration's SQL reads each batch of labels
(`-- check Parcel.label: in Jadpo, 1000 rows a batch, …`). With it, it
rewrites those the new `Tag` doesn't read (`-- rewrite Parcel.label: …`)
and leaves the rest as they are.

- **The check** comes before anything else in the store's section, so a
  refusal applies nothing of the migration; the migrations before it stay.
  It reads the rows with a value (`NULL` is `none`), 1000 a batch in key
  order. Each batch is its own read-committed transaction under
  `jadpo_migrations`' lock, locking no row and changing nothing, and reads
  each value with the codec of the field's type now, as a query reads it.
  It records nothing as it goes: a run stopped during it starts it again,
  and a refusal clears the migration's progress, so the next run reads
  every value again. The refusal is `migration.apply_failed`, naming how
  many values don't read and the first rule broken and where (``
  `max_length` on `line` ``), never a value.
- **The decision may be left out.** No decision is required: `jadpo
  migrate new` finishes the migration and writes a `changes.jadpo` holding
  only a comment saying how to give one. Writing `backfill:` there and
  running `jadpo migrate new` again replaces that field's check with a
  rewrite.
- **The rewrite** runs in batches, as a backfill worked out in Jadpo does
  ([Computed backfills](computed-backfills.md)), recorded in the resume
  marker. Each batch reads the rows with a value, locked `FOR NO KEY
  UPDATE`, reads each with the new type, works the backfill out for those
  that don't read, and writes them back through the codec, leaving the
  rest as they are. The backfill may be written out or worked out from the
  row, as any backfill in Jadpo may; a narrowed column's refit outside
  JSON still takes only a value written out. A backfill still reads no
  field held as JSON, its own included, whose old type is gone.
- **Values written meanwhile.** A server still on the old build may write
  a value the new type doesn't read after the check or the rewrite has
  passed its row. As with a narrowed column's refit, no lock holds writes
  to the whole table out for that long, and such a value is a fault when
  read, as any stored value that doesn't read is.
- **A required structured field added to a table with rows** takes a
  `backfill:`, as any other field does. Its value, written out (a
  construction such as `Label { … }`, or a variant) or worked out from the
  row, has no SQL form. So it is always worked out in Jadpo and written by
  the codec of the field's type, as a write keeps it, then made required
  in the same last steps as any backfill.
- **Only the latest migration** is read with the project's code, as for
  backfills. An earlier migration's check or rewrite with rows to read
  stops `jadpo migrate apply` (`Checking Parcel.label for <migration>
  needs the build it was written in`).
- **An older snapshot** that recorded nothing of what a JSON column read
  has the next migration check its stored values, rather than recording
  the type with nothing to do, since nothing shows that the values stored
  then read now. On an empty table the check reads nothing.
- **Output.** `jadpo migrate new` lists "check stored values in batches in
  Jadpo" or "rewrite values no longer read in batches in Jadpo". The live
  line says `Checking Parcel.label in Jadpo  2000 of 5000 rows`. The
  result says `1200 rows checked in 2 batches in Jadpo` or `3 rows
  rewritten in 2 batches in Jadpo`, and `--json` gives each its `work`.

Not built yet: a condition on a structured field (`toolchain.not_built_yet`);
a `unique:` or index over one (`toolchain.not_built_yet`), since PostgreSQL
compares `jsonb` by its own rules; and an update by `id` that works a
structured value out in the database rather than setting one it is given
(`toolchain.not_built_yet`, as for money).