How queues are received

This is the detail behind Queues: when a message is acknowledged, what is parked, how messages are read and fetched, and how a package provides receiving. The Cloudflare Queues driver is under The Cloudflare Queues driver.

• When a message is acknowledged. Once the transaction keeping its event commits: in one read-committed transaction in the events' store, its id is recorded under queue <connection> in jadpo_received_events and its outbox row and a delivery for each subscriber are written, as a webhook's are; then the message is deleted from the queue. Its subscribers run from the outbox with their usual retries, and one that gives up is parked as any delivery is, so "after the subscriber's transaction commits" is read as the transaction that hands the event to them; tying the message to each subscriber's run would keep a second retry budget beside the outbox's. A repeat, its id recorded already, is deleted and keeps nothing. Ids are deleted by the tick after 30 days, as a webhook's are.
• Parked, never dropped. A message whose type the queue doesn't list (unlisted_type), one that can't be read as its event (unreadable: not JSON, no id or type where the queue says, a field its event refuses, each problem kept by field and rule, never a value) and one over 64 KB as its event (too_large) are kept whole in jadpo_parked_messages, once per queue and body, and deleted from the queue. jadpo deliveries lists each as failed (queue warehouse_queue · OrderShipped, its reason as the code); retry makes it pending and the app's next tick reads it again with the build running then, keeping it or parking it again; dismiss sets it aside, deleted by the tick 30 days later. A message the store can't take now is left on the queue, which sends it again, and the queue's own dead-letter queue takes one that keeps failing, which on Cloudflare Queues the Worker parks; one whose event the store refuses as a statement, as it would every time, is parked unreadable.
• Reading a message. Its body is JSON; its id is the text or whole number at id_at:, its type the text at type_at:, matched against each listed event's wire_name:; a queue of one kind has no type. The event is read from the body's root, every member it doesn't declare left out, the queue's own id and type among them, as a webhook's are.
• Naming the events. events: names each event as code does: the app's own imported on a use line in app.jadpo (use stock/events { OrderShipped }), or a package's public one as Package.Event. With type_at: each gives a text wire_name:, no two alike; without it the queue lists one event. Anything else is receiving.invalid, and with no store to keep the events in, event.store_required.
• When it is received. On the tick: each queue is asked for up to ten messages at a time, again while a batch comes back full, at most ten times a tick, within the tick's budget; up to 20 parked messages an operator retried are read first. An app that receives a queue ticks at least every 10 seconds. On Workers the queue's consumer is handed batches instead ("The Cloudflare Queues driver").
• How a package provides it. provides/receiving.jadpo holds provide receiving with receive: (call:, input:, messages: response.<field>, body: item.<field>, handle: item.<field>) and acknowledge: (call:, input: with message.handle). Each input is keyed by the operation's input field and given settings.<name>, a setting the package declares, or text, a whole number or a flag; every required field is mapped, the list is a list of objects, and the body and the handle are text. An operation whose answer the package's own decode: reads can't be mapped. Problems are package.provider_invalid in the mapping's file. An installed package's resources a mapping calls are loaded with it.
• AWS's JSON protocol. Protocol.aws_json sends every call as a POST with Content-Type: application/x-amz-json-1.0, its body the input ({} without one), naming the operation in X-Amz-Target: <target>.<Operation>, its name in UpperCamelCase (receive_message is ReceiveMessage). A refusal's code at error_code: is AWS's error type, read after its # in lower_snake_case (com.amazonaws.sqs#QueueDoesNotExist is queue_does_not_exist). The provider key aws: { service: target: } names the service requests are signed for and the operations' prefix; aws_json needs target: and aws_sigv4 needs service: (package.provider_invalid).
• AWS Signature Version 4. CredentialScheme.aws_sigv4 signs every header but user-agent, with Host and X-Amz-Date, over the body's SHA-256, as AWS's published test suite and worked example sign, which its tests check. The connection gives credential: { key_id: config.<name> secret: config.<name> }, the secret a secret setting (connection.value_invalid otherwise). The region is read from an AWS host, <service>.<region>.amazonaws.com, and is us-east-1 for any other, as AWS's own tools sign for a local emulator; host placeholders wait for a package with fixed regional hosts, so the SQS package's connection gives its endpoint with host:. The secret appears in no header, problem or log line.
• A member a provider leaves out. On a provider's wire, a field its type may hold none in reads as none when the provider leaves its member out, as SQS leaves Messages out of an empty receive; the app's own contract still needs the member.
• The SQS package, AmazonSqs.v2012_11_05, is installed with jadpo package add. It maps ReceiveMessage and DeleteMessage, and marks ReceiveMessage safe to send again, since repeating it only hides other messages for a while. Its tests run against a local fake of SQS's JSON API that checks each signature.

Capability mappings other than receiving's and email's aren't built yet: one is toolchain.not_built_yet.