How a file's run is kept

This topic is what a file helper's read run does beyond a query's run (How a run is kept). It covers csv.read, tsv.read, json_lines.read and json.read's file form. Where Batches and Data formats leave a choice open, the run takes the most cautious one.

Where it is kept

A file has no query whose store could keep its run, so the run is kept in the store the starting action writes, counting what each:'s and finished:'s actions write. The rule of one store per action makes that a single store. When none of them writes, the run is kept in the app's only database.

A batch is rows

Each transaction holds a batch of a hundred of the file's rows. Under jadpo test it holds two, so every import in a test resumes between batches. The run's position is where the next row begins: its byte, its row and its line, kept with the run. The file is saved with the run as its File, so a resumed run reads the same one.

Read before the transaction

No outside call goes out while a transaction is open (The seam every outside call goes through), and an item's action runs inside its batch's transaction. So each batch's rows are read from the store first, through the files capability, in windows of at most a megabyte until the rows are whole, starting from the position the last batch committed. Then the batch's transaction opens. It holds the run's row to that same position, runs the rows' items, and commits their writes with the problems and the position after the last row read.

Each time a runner picks the run up, it reads the start of the file once, for its encoding and, for CSV and TSV, its header.

Columns

The header names the columns. Each is a field of as:'s Input, by the field's name or its column: for a CSV or TSV column; for JSON Lines, the fields are the Input's JSON.

These stop the run before any row is read, because an Input refuses an unknown field: a column no field reads, a column named twice, or a field that may not be left out with no column for it.

A row with more cells than the header is a problem (unknown_field). An empty cell is left out, so optional: and default: apply, and an empty required cell is a required problem.

Rows and problems

A row is read as a value in a query string is: each cell is its field's text, and a standard type's cell is its canonical text. A row that can't be read is a problem, with its row as a spreadsheet numbers it (the header is row 1, and a blank line is counted but makes no item), its column by the header's name, and the rule. In JSON Lines, a problem's row is its line and its column the field's path. A row whose action fails is a problem with no column and the failure's code. A problem never holds the value.

Quotes and line endings

Quoting follows RFC 4180: a quoted cell may hold the delimiter, line breaks and "". LF, CRLF and a lone CR each end a row. A quote inside an unquoted cell, or text after a cell's closing quote, is kept as written, as spreadsheets read it.

What stops the run

These stop the run at the place they are, with the rows before them done:

• a quote that never closes, named by the line it opened on;
• bytes that aren't text in the file's encoding, a NUL among them, named by their line;
• a gzip file read without max_unpacked_size:, or one that unpacks past it, is damaged or is cut short (gzip);
• a row longer than 16 MB.

The run ends failed and says why. A store that can't be reached pauses the run, as it does a query's.

A JSON file's list

json.read's file form reads the list items_at: names, or the whole document when it names none. Each item is a row. Its row number is its place in the list, counting from 1, and a problem's column is the field's path, as in JSON Lines.

The document is walked a window at a time, holding only where the walk is and the item in hand, so a document of any size is read in the memory of its largest item. Objects on the way to the list are read only for their members' names. Every other value is passed over, its brackets matched and its strings followed.

An item is the text of one value in the list, which the row type reads as a request's body is read. So an item that isn't JSON (tru), or isn't of the row type, is that item's problem, and the rest carry on.

These stop the run at the place they are, with the items before them done:

• no list where items_at: says, or, with no items_at:, a document that isn't a list;
• structure that isn't JSON, such as a missing comma or a bracket that doesn't match, named by its line;
• a document that ends before its JSON does;
• an object on the way to the list naming a member twice;
• an item longer than 16 MB.

Once the list closes, the rest of the document is walked to its end, so a document cut short, or followed by more text, stops the run after its last item.

Each runner walks the document from its start to the list once, as it reads a CSV's header. Each page starts from that walk, set to the position its run committed: the byte after the last item done, with its line and the next item's place. items_at: is part of the call's shape, so a deploy that changes it ends a running run failed, as one that changes what the run saves does.

gzip

A file whose first bytes are gzip's is read only when the helper is given max_unpacked_size:, and is unpacked as it is read (Files). The run's position, its rows' lines and its encoding are those of the unpacked text.

Its text is never read past max_unpacked_size:. The run stops at the row that would reach past it ("its file unpacks to more than 200 MB, its max_unpacked_size:"), with the rows wholly inside it done, so a small upload can't unpack to gigabytes. gzip can't be entered part way, so a runner that picks the run up unpacks the file again from its start to its position.

A file that isn't gzip is read as it is, whatever max_unpacked_size: says. max_unpacked_size: belongs to the build, not to what the run saves, so a deploy may change it under a running run.

Encodings

A file is read as UTF-8, with any byte-order mark removed, or as UTF-16 with its mark, little- or big-endian. The encoding is never guessed: anything else isn't text.

Not built yet

A file's run in an app with several databases, when neither the starting action nor each:'s or finished:'s actions write any of them, is toolchain.not_built_yet.