# Operations

> Health probes, graceful drain, scaling out, what a fleet costs, and dead-letter redrive.

There is nothing to operate but the binary and a bucket: no database, no
Raft, no coordination service.

## Health

A second listener (default `:9090`, `admin_addr`) serves orchestrator
probes:

| endpoint | meaning |
| --- | --- |
| `/healthz` | the process is up |
| `/readyz` | the process can reach its bucket |

Readiness is probed against the bucket every
`readiness_probe_interval_ms` (default 5000), fast enough that a bucket
outage shows up before a scheduler gives up on the node.

## Graceful drain

On shutdown the server drains: parked long polls are cut short and
in-flight work completes within `drain_deadline_ms`. The default is 30000,
matching Kubernetes' own default termination grace period, because a drain
that outlives the grace period is a drain that gets killed anyway.

## Scaling

Scaling out means starting another process against the same bucket; there
is no membership, rebalancing, or handoff step.

A fleet is not free, though, and okuri publishes what it costs. Measured
with `just cost` (seed 7, 240 messages through one queue, nothing
injected):

```
                             per message produced          per message consumed
setting                  plain   cas write   get  list plain   cas write   get  list
1 node                    0.10  0.01  0.11  0.01  0.00  0.00  1.10  1.10  3.60  0.10
2 nodes                   0.18  0.03  0.21  0.03  0.00  0.00  1.25  1.25  5.66  0.11
4 nodes                   0.31  0.05  0.36  0.05  0.00  0.00  1.38  1.38  8.61  0.12
```

At $5.00 per million PUT-class requests and $0.40 per million GETs, one
node is $7.99 per million messages and four nodes is $12.76: quadrupling
the fleet costs 60% more per message. Size the fleet for availability and
drain speed, not throughput reflex.

## Dead letters and redrive

A message that exhausts `max_receive_count` moves to the queue's
dead-letter companion at `{queue}-dlq`. The dead-letter queue is
receivable directly under its own name. Redrive, moving dead letters back
to the parent, is a deliberate manual operation through the native API:
`RedriveQueue` over gRPC or `POST /v1/queues/{queue}:redrive`.

## Purge

`PurgeQueue` (SQS) or `POST /v1/queues/{queue}:purge` takes effect the
moment it lands. Messages sent after the purge are untouched by
construction, and the response names the exact cut instant
(`PurgedBeforeUnixMs` on the SQS surface).

## Deduplication caveat

The producer-side deduplication window is in-memory and best-effort. A
node restart clears it, and a fleet does not share it. It reduces
duplicates without eliminating them, so consumers still need to be
idempotent.

## Coming in 0.1.0

The operability milestone (M6) adds Prometheus metrics, OpenTelemetry
tracing, `okuri queue` admin CLI commands (create, peek, redrive, purge),
and Grafana dashboards. Until then, observability is logs plus the stats
endpoints.