# Data formats: JSON, CSV, TSV and JSON Lines Four namespaces turn data in a format into your app's types and back, each in one shape. `json` reads and writes one document in memory, or a large JSON list in a file. `csv`, `tsv` and `json_lines` read and write files only. These namespaces are pure: they turn bytes into your app's values and back, wherever the bytes came from. Moving the bytes is a store's job, with `store.read` and `store.put` ([Files](files.md)). ## Reading and writing a file's rows A file's rows, however many, are read and written only through these operations' batch forms. They run as [batches](batches-work-over-many-items.md): background work over many items that keeps its progress. There is no loop over a file's rows and no reading a whole file into a list. How a run goes (batches, transactions, resuming, `batches.status` and `batches.cancel`, problems) is described with batches; the operations are here. - A batch form is called only in an action. It starts after the action commits, as `emit` does, and returns a `BatchRun` at once. If the action rolls back, the run never starts. - `key:` makes a repeated start return the same run. - `each:` and `finished:` take an inline function, written with `->` ([Passing functions](passing-functions.md)). Reading, `each:` calls one action per row. Writing, it is a pure expression that gives the row. The values they capture are saved with the run, so they may capture only values that can be stored. - Writing takes the store to write to, the file's `name:`, and `from:`. `from:` is a call to a named query that returns a `Page`, walked in keyset pages ([Pages](order-and-pages.md#pages)), or to a provider's list operation, walked by `page.next` ([Calling](calling.md)). - `finished:` receives an `ImportSummary` after a read, and after a write an `ExportSummary`, which holds the stored `File` ([Other built-in types](built-in-types.md#other-built-in-types)). ## `csv` | Operation | Gives | Fails | SQL form | |---|---|---|---| | `csv.read(file, as:, each:, finished:, key:, max_unpacked_size:)` | `BatchRun`: each row of a stored `File` decoded as an `Input` of `as:` and handed to `each:`; `finished:` receives an `ImportSummary`; a gzip file unpacked up to `max_unpacked_size:` | Never as a call: a row that can't be decoded, or whose action fails, is a `RowProblem` and the rest carry on; a file-level fault (a missing required column, an unterminated quote named by the line where it opened, an encoding the format doesn't declare, text that isn't text) stops the run | none | | `csv.write(store, name:, format:, from:, each:, finished:, key:)` | `BatchRun`: each item of `from:` turned into an `Output` row by `each:` and written to `name:` in `store`; `format:` a `CsvFormat`, required; `finished:` receives an `ExportSummary` | Never as a call: an item whose action fails is a `RowProblem` | none | - **Rows are typed both ways.** A row read is an `Input`, and a row written is an `Output`, with their rules, problems and JSON encoding ([HTTP](http.md)). A column maps to a field by the field's name, or by `column: "Email address"`. Cells written in other conventions use the `wire:` encodings, such as `Wire.day_month_year`, or a pure `decode:` function. A CSV row holds single values only: a list or an object in one is a compile error that points you to JSON Lines. - **A standard type in a cell is its canonical text,** as `text.from` writes it (`GBP 10.50`, `2.5 kg`, `2026-10-25T09:30:00Z`). It is read by `text.parse`, with its failures as the row's problems, so a cell keeps a currency, a unit and every decimal place. A standard type with no single text form (`PostalAddress`, `PersonName`, `Price`, a choice such as `Discount`) counts as an object here, so the compile error points to JSON Lines, whose rows carry the contract's JSON ([Every boundary through the contract](every-boundary-through-the-contract.md)). - **Writing names its reader.** `CsvFormat.spreadsheet` writes a byte-order mark and CRLF line endings, and escapes text that looks like a formula. `CsvFormat.standard` writes CRLF only. A write without `format:` is a compile error. - **Encodings.** Reading takes UTF-8 (a byte-order mark removed), UTF-16 with a mark, and legacy encodings the format declares, never guessing. Writing is UTF-8. gzip (`.csv.gz`) is read only with `max_unpacked_size:`, and no archives are opened. `max_unpacked_size:` is a size written as a `File` field's `max_size:` is (`200mb`), and every reader takes it. A file that isn't gzip is read as it is ([How a file's run is kept](file-batch-runs.md)). - **A `RowProblem`** holds the row as a spreadsheet numbers it (the header is row 1, the first data row row 2), the column, the field and the rule, never the value, so personal data stays out of problem lists and logs. - **No CSV in memory.** A provider's CSV answer goes into a store and through `csv.read`, so there is one way to read rows. How a row is written: - **The header and the cells.** A CSV or TSV file starts with its header, the row type's columns in declared order, each named by its `column:` or by its field's name, then one line for each row. A cell is its field's text as a query string holds it, and is read back the same way: a standard type is its canonical text, an enum its variant's name, and `none` an empty cell. - **Quoting.** A cell holding the delimiter, a double quote, a carriage return or a line feed is quoted, with its quotes doubled (RFC 4180), in TSV too. Every line, the last among them, ends with CRLF. - **Who reads it.** `CsvFormat.spreadsheet` writes a UTF-8 byte-order mark first. It also puts `'` before a cell of a field built on `Text` that starts with `=`, `+`, `-`, `@`, a tab or a carriage return, so a spreadsheet never reads it as a formula. A number's cell, a negative one among them, is never changed. `CsvFormat.standard` does neither. - **JSON.** A JSON Lines file is each row's JSON, as a response's body writes it (by its fields' names; `column:` names only a CSV column), with a line feed after each. `json.write`'s file form writes one compact list of the rows. An export of no items is its header alone, an empty file, or `[]`. - **The file.** UTF-8, named by `name:`, its `FileType` `csv`, `tsv`, `json_lines` (`application/jsonl`) or `json`. A bakery writes its price list for the shop's spreadsheet. `each:` turns each loaf into a `PriceLine`, and `finished:` keeps the file it wrote: ```jadpo application { connections: { price_lists: { driver: Driver.disk path: "/var/lib/bakery/price-lists" } } } ``` ```jadpo use bakery/types { LoafCode, LoafPrice } /// One line of the price list a spreadsheet reads. type PriceLine = Output { code: LoafCode { column: "Loaf code" } price: LoafPrice { column: "Price" } } ``` ```jadpo use app { price_lists } use bakery/entities { PriceList, PriceListNotFound } use bakery/types { LoafPage, PriceLine } use bakery/actions { prices_listed } use bakery/queries { loaves_by_id } /// Starts writing every loaf's price into a spreadsheet, once. action list_prices(list_id: PriceList.id) fails PriceListNotFound -> PriceList { var price_list: PriceList = attempt PriceList.get(list_id) var list_run: BatchRun = csv.write( price_lists, name: "prices.csv", format: CsvFormat.spreadsheet, from: loaves_by_id(LoafPage {}), each: loaf -> PriceLine { code: loaf.code price: loaf.price }, finished: done -> prices_listed(done, price_list.id), key: price_list.id ) return PriceList.update(price_list) { set: { list_run: list_run } } } ``` Three loaves, one of them with a code a spreadsheet would read as a formula, give this file: one line for each loaf, in the order of their ids, after the byte-order mark, each line ending in CRLF: ```text Loaf code,Price rye,3.50 spelt,4.20 '=sour,5.00 ``` A bakery reads its suppliers' price sheets into its loaves. The sheet's header names the columns, and `column:` names one that isn't the field's name. A row whose price isn't a price, or whose loaf's code is taken, is a problem, named by its row and column, and the rest carry on: ```jadpo application { connections: { sheets: { driver: Driver.disk path: "/var/lib/bakery/sheets" } } } ``` ```jadpo use bakery/types { LoafCode, LoafPrice } /// One sheet_line of a supplier's price sheet. type PriceRow = Input { code: LoafCode { column: "Loaf code" } price: LoafPrice { column: "Price" } } ``` ```jadpo use bakery/entities { PriceSheet, PriceSheetNotFound } use bakery/types { PriceRow } use bakery/actions { add_loaf, prices_read } /// Starts loaf_run a kept price sheet into the loaves, once. action read_prices(sheet_id: PriceSheet.id) fails PriceSheetNotFound -> PriceSheet { var supplier_sheet: PriceSheet = attempt PriceSheet.get(sheet_id) var loaf_run: BatchRun = csv.read( supplier_sheet.sheet, as: PriceRow, each: sheet_line -> add_loaf(sheet_line), finished: summary -> prices_read(summary, supplier_sheet.id), key: supplier_sheet.id ) return PriceSheet.update(supplier_sheet) { set: { loaf_run: loaf_run } } } ``` A sheet like this one adds two loaves. Row 3's price and row 4's code are problems, `(3, "Price", "type")` and `(4, "", "code_taken")`, and `finished:` hears of four rows, two of them with problems: ```text Loaf code,Price rye,3.50 spelt,cheap rye,3.80 oat,2.95 ``` ## `tsv` | Operation | Gives | Fails | SQL form | |---|---|---|---| | `tsv.read(file, as:, each:, finished:, key:, max_unpacked_size:)` | As `csv.read`, with a tab between cells | As `csv.read` | none | | `tsv.write(store, name:, format:, from:, each:, finished:, key:)` | As `csv.write`, with a tab between cells | As `csv.write` | none | TSV is CSV with a tab between cells, so every rule under `csv` applies. ## `json_lines` | Operation | Gives | Fails | SQL form | |---|---|---|---| | `json_lines.read(file, as:, each:, finished:, key:, max_unpacked_size:)` | `BatchRun`: each line of a stored `File` decoded as one JSON value, an `Input` of `as:`, and handed to `each:`; `finished:` receives an `ImportSummary` | As `csv.read`: a line that can't be decoded is a `RowProblem`; text that isn't text stops the run | none | | `json_lines.write(store, name:, from:, each:, finished:, key:)` | `BatchRun`: each item of `from:` turned into an `Output` by `each:` and written as one line; `finished:` receives an `ExportSummary` | As `csv.write` | none | A JSON Lines row may hold lists and objects, which is why a CSV row that needs them is pointed here. Encodings and gzip are as for `csv`, and a JSON Lines file takes no `format:`. > Not built yet: reading legacy encodings. They come when an application > needs one. > > The readers run as part of a `batches.each` run > ([How a file's run is kept](file-batch-runs.md)), > and the writers as an export's run does > ([How an export's run is kept](export-batch-runs.md)).