# Telemetry

The generated runtime reports what it does as OpenTelemetry traces,
metrics and logs, the open standard most monitoring tools accept. With
no `telemetry:` section, nothing is sent. To send it, add one to
`application { }`. It isn't a connection, because there is one protocol
and no vendor-specific code: a vendor is an endpoint and a header. Its
host is still listed in the network report
([The network report](connecting-in-app-jadpo.md#the-network-report)). Here
telemetry goes to Honeycomb:

```jadpo
// app.jadpo
application {
    telemetry: {
        endpoint: config.telemetry_endpoint
        credential: {
            header: "x-honeycomb-team"
            value: config.telemetry_key
        }
        send: {
            traces: true
            metrics: true
            logs: LogLevel.info
        }
        detail: {
            full_traces: [Trace.failures, Trace.faults, Trace.slow, Trace.new_paths]
            samples: {
                per_path: 1
                every: 1m
            }
        }
    }
}
```

## `send:`, which signals go out

`send:` says which signals go out, and the lowest level of log sent.
Leave it out and traces and metrics are sent, with logs from
`LogLevel.warn`.

## `detail:`, how much of each request

`detail:` says how much of each request is sent. `full_traces:` lists
when a request's whole trace is sent: it failed, it faulted, it crossed a
threshold, or its path is new to this process. `samples:` says how many
ordinary traces each path sends, `per_path:` within `every:`, so there is
always a normal request to compare a bad one with. The values in the
example are what you get by default, as
[Monitoring thresholds](monitoring-thresholds.md) describes.

## The standard variables

`endpoint:` and `credential:` come from settings bound to the standard
`OTEL_*` names
([Environment variable names](settings-config-jadpo.md#environment-variable-names)),
so a host that sets `OTEL_EXPORTER_OTLP_ENDPOINT` and
`OTEL_EXPORTER_OTLP_HEADERS` needs nothing else. Your app's code can't
read these settings.

## What is sent

- **Traces.** One span for each route, action, query, provider call,
  cache read, schedule run and event delivery, named after the
  declaration (`POST /todos`, `create_todo`, `todos_for_owner`).
  Incoming and outgoing `traceparent` headers are honoured, and a trace
  continues from the action that emitted an event into its subscriber.
- **Metrics.** Requests and durations for each route, failures for each
  code, query durations, provider calls and retries, cache hits,
  rate-limit refusals, the size of the queue and the age of the oldest
  waiting delivery.
- **No values from your app's data,** unless you add them deliberately.
  Spans carry names, identifiers and failure codes, never field values,
  as logs do ([Logging](logging.md)), and marked personal
  fields are hashed. The two deliberate ways to add a value are custom
  metric labels and `telemetry.note`, below, and both refuse personal
  fields.
- **Jadpo's own data, to every provider.** Path IDs, stable identities,
  source locations and the build id travel as attributes named
  `jadpo.…`, such as `jadpo.path` and `jadpo.path_key`. A monitoring
  product decodes them against the build manifest
  ([The manifest](the-manifest.md)), which a
  deploy uploads once for each build.

## Custom metrics

You declare your own metrics in a domain's `monitoring.jadpo`
([Project layout](project-layout.md)) and
record them in code:

```jadpo
// domains/billing/monitoring.jadpo
/// Checkouts that completed, by plan.
metric checkouts_completed = Counter {
    by: [plan]
}

// domains/billing/actions/complete_checkout.jadpo
telemetry.checkouts_completed.add(1, plan: account.plan)
```

A metric is one of three kinds:

- a `Counter`, a count that only rises, recorded with `add`;
- a `Measure`, a distribution such as a duration or an amount, recorded
  with `record`;
- a `Gauge`, a value at a moment, recorded with `set`.

A metric's labels, its `by:`, may only be enums and `Bool`, so the
number of series stays bounded.

`telemetry.note(name: value)` adds an attribute to the current trace,
for a fact no span carries. It takes text, numbers, `Bool` and enums. A
personal field given as a label or an attribute is a compile error.
There are no custom spans, because every action and query is already
one.

## In development

`jadpo dev` exports nothing unless you give it `--export-telemetry`. The
traces, metrics and logs stay on your machine, and you read them from the
command line with `jadpo traces`, `jadpo metrics` and `jadpo logs`,
filtering and watching them as you do the development call log
([Traces, metrics and logs in development](traces-metrics-and-logs-in-development.md)),
and in the developer console.