# Files

To store a file, such as a user's avatar, give an entity a field of type
`File`. The record holds the file's key, name, size and type. The bytes
live in a connection that provides the `files` capability: the built-in
S3 driver for any S3-compatible service, or the built-in disk driver on
the native target. Here an avatar is kept in a Cloudflare R2 bucket:

```jadpo
// app.jadpo
application {
    connections: {
        avatars: {
            driver: Driver.s3
            host: config.r2_host
            region: "auto"
            bucket: "avatars"
            credential: {
                key_id: config.r2_key_id
                secret: config.r2_secret
            }
        }
    }
}

// domains/accounts/entities/user.jadpo
entity User {
    ...
    avatar: File? {
        store: avatars
        max_size: 5mb
        types: [FileType.png, FileType.jpeg, FileType.webp]
    }
}
```

The field's settings:

- `store:` names the connection, as it does for entities and caches. Two
  buckets are two connections.
- `max_size:` is the largest file the field accepts.
- `types:` lists the file types it accepts.

Cloudflare R2, Amazon S3, Backblaze B2, DigitalOcean Spaces and MinIO all
use the S3 driver, each with its own host and settings, listed on the
driver's page. No package wraps the driver with presets.

## Files in development and tests

In `jadpo dev` and tests, files go to local disk with locally signed
links, so nothing leaves your machine. That local driver doesn't exist in
production builds. An app whose production store is the disk driver uses
the same driver in development, at a folder under `.jadpo/`, so upload
slots, ranges and signed links behave as they will in production.

## Writing a file your app makes: `put`

A file your app makes itself, such as a generated PDF, is written with
`attempt avatars.put(bytes, FileType.pdf)`, which returns a `File` to
store in a field. `put` is an outside call, so you make it before the
action's writes, while no transaction is open
([Outside calls hold no transaction](all-or-nothing-actions-and-concurrent-changes.md#outside-calls-hold-no-transaction)).
In an action with several changing calls it is a keyed step, like a
provider call, and its key is derived as any step's is, so a rerun writes
the same object again
([Several changing calls in one action](several-changing-calls-in-one-action.md)).
Bytes that are never attached to a record are deleted after a day, like
abandoned uploads.

A bakery keeps the price sheets its suppliers send. `sheets.put` keeps
the bytes before the action reads or writes anything, and gives the
`File` that the record keeps. A store with no room answers
`storage_full`, which only a failure of kind `unavailable` can map:

```jadpo
application {
    connections: {
        sheets: {
            driver: Driver.disk
            path: "/var/lib/bakery/sheets"
        }
    }
}
```

```jadpo
/// A supplier's price sheet, kept as it was sent.
entity PriceSheet {
    id: Uuid
    sheet: File {
        store: sheets
        max_size: 5mb
        types: [FileType.csv]
    }

    policy: {
        create: [Access.public]
        read: [Access.public, Access.system]
    }
}
```

```jadpo
use app { sheets }
use bakery/entities { PriceSheet }
use bakery/types { SheetText }
use bakery/failures { SheetNotKept }

/// Keeps a price sheet as it was sent.
action keep_prices(sheet_text: SheetText) fails SheetNotKept -> PriceSheet {
    var sheet: File = attempt sheets.put(encoding.to_utf8(sheet_text), FileType.csv) {
        storage_full: SheetNotKept
    }
    return PriceSheet.create {
        set: {
            sheet: sheet
        }
    }
}
```

## Reading a file: `read`, and formats

A store moves bytes in and out and knows nothing of what is inside.
Formats (CSV, TSV, JSON, JSON Lines) turn bytes into your app's types and
back, whatever store the bytes came from. Your code joins the two:

- `attempt store.read(file)` returns the file's contents as
  `FileContents` ([Other built-in types](built-in-types.md#other-built-in-types)),
  for passing the bytes on: to a provider as an upload, to a PDF service,
  or as an email attachment. The format helpers don't accept it. It reads
  the whole file into memory, so it is capped: reading a field whose
  `max_size` is over the whole-read limit of 10 MB is a compile error
  pointing at the format helpers or `batches.each`.
- Rows in a file, of any size, are read and written only through the
  batch helpers: `csv.read`, `csv.write`, `tsv`, `json_lines`,
  `json.read` and `json.write`
  ([Batches: work over many items](batches-work-over-many-items.md)).
  [Data formats](data-formats-json-csv-tsv-and-json-lines.md)
  lists their operations. There is no loop over a file's rows, and no
  parse of a whole file into a list. A provider's CSV report is put in a
  store and read the same way.
- `json.read(text, as:)` is only for single JSON documents in memory,
  such as request bodies and provider answers.

Media services, such as Cloudinary and Uploadcare, are ordinary packages.
You call them through their own operations and pure functions, such as
one that builds a transformation URL, because they can't do the whole
`files` job.

## Uploading

Files go straight from the client to the store, not through your app, so
the 1 MB limit on a request body ([What is refused](one-input.md#what-is-refused))
isn't in the way:

1. An action calls the generated `User.avatar_upload()`, which returns an
   `UploadSlot`: a signed address valid for 15 minutes that accepts only
   `max_size` bytes.
2. The client sends the file to that address.
3. The client sends the slot's token in an ordinary create or update, in
   an `Input` field of type `FileUpload`. Before the write commits, the
   runtime checks that the file is there, its size, and its type from its
   first bytes rather than what the client claimed. A mismatch is
   `invalid_value`.

Asking for a slot needs the `create` or `update` grant on the entity
([Grants](grants.md)). Uploads that are never attached are
deleted after a day.

## Large uploads go in parts

A field whose `max_size` is over 100 MB gets a multipart slot instead of
a single address, the same shape on S3, R2 and disk. The slot names a
part size, with equal parts as R2 requires, and gives an address for each
part: a signed `UploadPart` address on S3 and R2, and
`PUT /files/<store>/uploads/<slot>/parts/<n>` on disk. The client sends
the parts, and may resend any part after a dropped connection. Attaching
the `FileUpload` completes the upload and runs the same size and type
checks. A generated client helper hides the parts.

```text
POST /imports/slot                          an app route calling CustomerImport.sheet_upload()
<- 200 { "token": "upl_…", "part_size": 16777216, "parts_url": "/files/uploads/upl_…/parts" }
PUT /files/uploads/upl_…/parts/1            16 MiB
<- 204
PUT /files/uploads/upl_…/parts/2            the connection drops; the client resends part 2
<- 204
POST /imports { "sheet": "upl_…" }          the ordinary create joins the parts, checks size and type
<- 201
```

## Compressed uploads

An upload sent with a `Content-Encoding` is refused. A gzip file, such as
`.csv.gz`, is read only by a batch helper given `max_unpacked_size:`, and
going past that size fails the file, so a small upload can't unpack to
gigabytes ([Reading gzip](reading-gzip.md)). Jadpo unpacks no other
compression and no archives, such as zip.

## Downloading

An `Output` that picks a `File` field sends a signed link valid for five
minutes, with the file's name, size and type. The read has already passed
policy, field restrictions included, so there is no separate download
operation to protect.

## Deleting and erasing

Deleting the record, or clearing the field, deletes the bytes after the
change commits, through a generated subscriber, because an action writes
only one store ([One store per action](one-store-per-action.md)).
A `File` field can be marked `personal:`. Erasure and retention then
reach it like any field ([Personal data](personal-data.md)), and
`User.export` includes it as a link.

## The disk driver

`Driver.disk` keeps files on the server's own disk. It works in
production on the native target only, and on Workers it is a compile
error. The build manifest records that the app then runs on one server,
since a second server wouldn't see the files.

```jadpo
// app.jadpo
application {
    connections: {
        uploads: {
            driver: Driver.disk
            path: "/var/lib/acme/uploads"
            free_space_floor: 5%
        }
    }
}
```

- **Files never change.** Each is kept under a key the runtime
  generates, a UUID sharded into two directory levels, never one derived
  from a file's name. The driver refuses any key that doesn't match its
  own pattern, opens files relative to its root without following
  symbolic links, and writes them readable by the app's user only. The
  root must not be inside the app's source or build folders.
- **Uploads are streamed** to a temporary file as they arrive. They are
  counted, with a `413` at the first byte over `max_size`, hashed and
  type-checked, then synced and renamed into place, and the directory is
  synced.
- **Your app serves downloads** from its own origin. It answers
  single-range `Range` requests and `If-Range`, with an ETag from the
  file's hash, and reads in fixed sizes, so a 2 GB download uses the
  memory a 2 KB one does. Because the origin is your app's own, every
  download is sent with `Content-Disposition: attachment` (except for an
  allow-list of image types), `X-Content-Type-Options: nosniff` and
  `Content-Security-Policy: sandbox`. The client's file name only ever
  goes in `filename*` (RFC 6266).
- **Space.** A write that runs out of space fails as `storage_full`.
  Below `free_space_floor:`, 5% by default, new uploads are refused, so
  imports and the database aren't starved of space. The floor is reported
  as a monitoring threshold.
- **Commands.** `jadpo files ls`, `show`, `get`, `put` and `copy` work
  on any store, in development or production. `jadpo files copy` moves a
  store's files to another, which is how you move off one server.

A deployable built for Workers that uses `Driver.disk` is refused
([`deployable.target_unsupported`](../errors/deployable.target_unsupported.md)).
`jadpo dev`'s disk driver answers a full disk with `storage_full`.

## Image variants and scanning

Both come when an app needs them.

- **Image variants**, such as a thumbnail, are declared on a `File`
  field, `variants: { thumb: 200x200 }`, and made by the runtime over any
  store, so no store needs to offer them and the `files` job stays the
  same everywhere. The runtime checks an image's pixel dimensions before
  decoding it. A variant is read as `user.avatar.variants.thumb`, a
  `File` that becomes a signed link in an `Output` like the original.
- **Virus scanning** is the `scanning` capability, named in
  `application { }` as `scanning: { connection: clamav }`. A ClamAV
  container package fills it ([Services](what-package-code-may-do.md#services)),
  or a scanning API's package. With it set, Jadpo scans every upload
  itself. A file stays in a scanning state, and can't be attached, until
  it passes. An infected file is refused like one of the wrong type, and
  deleted.