Reading gzip

Some uploads arrive as gzip files, such as a supplier's .csv.gz. This topic is how one is read (Compressed uploads).

Only the file helpers unpack, and only when told how far

A file whose first two bytes are gzip's magic number is unpacked by a format helper given max_unpacked_size: (Data formats), and by nothing else. store.read gives its bytes as they are stored, and a helper given no max_unpacked_size: stops the run before its first row.

The limit is the most text the run reads. The run stops at the row that would reach past it, with the rows wholly inside it done, and unpacks at most 64 KiB past it, so a small upload can't unpack to gigabytes.

gzip as RFC 1952 writes it, every member in turn

A gzip file is one member or more, one after another, as cat a.gz b.gz makes, and Jadpo reads them as one text.

• Each member's header is read whole, with every optional part its flags say it has (an extra field, a name, a comment), and its own CRC-16 checked when it has one.
• A reserved flag, a method other than deflate, or anything after a member that isn't another member is damage.
• Each member's trailer is held to what the member unpacked to: its CRC-32, and its size modulo 2³².

A damaged file, and one cut short inside a member, stop the run where the damage is found, with the rows before it done. No other compression, and no archive such as zip or tar, is opened.

Unpacked as it is read, never whole

The packed bytes are read from the store through the seam a window at a time, as a plain file's are, and unpacked (with miniz_oxide's streaming inflate) only as far as a page needs. gzip can't be entered part way, so a runner that picks up a run unpacks the file from its start to the run's position, discarding what comes before. Within one runner, each page carries on from the last.

A file that isn't gzip

A file that isn't gzip is read as it is, whatever max_unpacked_size: says, so one helper serves uploads that are packed and ones that aren't.