Files: an export's pieces

A large export is written a page of rows at a time, and each page is a piece of the file. This topic is what that asks of the files capability (How an export's run is kept; Data formats).

Writing the pieces

Each page's rows are written by an ordinary keyed put, under a key made from the file's own key (jadpo_std::files::piece_key). That key has the runtime's shape, so a driver refuses no piece, and finds each piece from the file's key alone.

Two more operations go through the seam, each a keyed call:

• join(key, pieces, type) joins the pieces, in order, into the file under key. Repeating it once the file is whole does nothing.
• discard(key, pieces) drops a cancelled export's pieces, and its file if they were joined.

A store with no room answers storage_full, and a join with a piece missing answers not_found.

The disk driver joins whole

The disk driver copies the pieces in order into a temporary file readable by the application's user only, a window at a time. It syncs the file, renames it into place and syncs its folder, then removes the pieces, so the file appears whole or not at all. The in-memory store a test's files are in does the same in memory.

FileType.json_lines

FileType.json_lines is a written JSON Lines file's type, application/jsonl, as jsonlines.org names it. It comes after json.