Monitor EDR

1. Embedded UI
2. Endpoints
3. The halted state
4. Health model
5. Triage CLI (edr-inspect)

For backfilling pre-existing or gapped data, see Historic fill and gap fill. For a Grafana dashboard built on the metrics below, see Grafana dashboard.

Embedded UI

Every EDR agent serves a web UI at /ui on its observability listener (--observability-listen, loopback 127.0.0.1:9091 by default): topology tree with health badges, per-channel metrics (bytes, lag, retry state), historic fill progress, and an activity log. Status is server-computed only—all thresholds come from config.

Endpoints

The observability endpoints below are served on --observability-listen (loopback by default—expose with --observability-listen 0.0.0.0:<PORT>). /health and the replication protocol are on the network --listen.

Outstanding work and row counts

The Prometheus gauge edr_owed_work reports work still owed to each destination. A value of 0 means the agent has no outstanding replication work for that destination at that observation. Use it with channel health when waiting for a backlog to drain before maintenance. New source writes can create more work after the observation. GA fixes a transient false zero while gaps move from pending to in progress.

rows_sent counts physical rows, not SQL count(*) results. Tables wider than 100 fields split a logical row across field families, so the counter can exceed the logical row count. For data validation, compare field-level values as well as counts. See ordering and wide tables.

The halted state

EDR never voluntarily skips deliverable data. When the downstream persistently refuses data that exists and could be delivered, the agent halts replication rather than replicate around the problem—a downstream that is stale-but-consistent as of the halt point beats one that looks live but silently misrepresents some tables.

Halting triggers:

• Schema conflict—immediately. Fix the downstream schema.
• Authentication failure—immediately. Fix the token or the config.
• Malformed batch (the receiver cannot parse what this agent sent)— immediately. Usually a version mismatch: upgrade the downstream agent.
• Persistent write errors or unclassified rejections—after the patience window (retry.halt_patience_secs, default 180). Within the window they are treated as transient and retried quietly.

What Halted means:

• Dispatch stops on every tier—no partial replication past the halt point. Upstream data keeps accumulating safely on the object store; the cursor does not move; nothing is lost.
• Underneath, the refused batch keeps retrying (every 60s for schema conflicts and malformed batches)—it is the probe. The moment the downstream accepts it, replication auto-resumes in order. No restart, no acknowledgement step.
• The JSON observability API reports replication_halted greater than 0 and sets halted_since_ns. The upstream agent includes the flag in its status report. The downstream agent exposes the reported value. See the halt field values.
• The agent writes the following:
  • An ERROR log entry that names the database and the reason.
  • A WARN heartbeat on every probe, with the halt duration.
  • An audit event.

Network outages and downstream restarts do not halt—they retry with backoff and show as degraded health; they self-correct.

Escape hatches (deliberately the only ones): fix the downstream (auto-resume), or exclude the offending table from the replication scope and restart the agent. There is intentionally no “skip after N hours” option.

Health model

Two layers, combined for display:

• Network health (silence duration vs the comms config): Healthy, Degraded (missed degraded_after intervals), Unreachable (missed unhealthy_after), or Unknown (never connected).
• Application health (last write result): Active, Unauthorised, SchemaConflict, Stalled (network unreachable), or Unknown.

Health status is the worst of (retry count vs config thresholds, silence duration vs config thresholds). Heartbeats and reports do not count as successful data sends—last_successful_send_ts only advances on real data. Schema and auth issues are local to the two directly connected nodes; they are not propagated up the chain.

Triage CLI (edr-inspect)

edr-inspect is a read-only triage tool shipped alongside the agent. Three independent subcommands, each reading one source:

state works even against a stopped or wedged agent (it reads the journals directly); it also accepts a cloud state-location (s3://, gs://, or az://, credentials from the environment). metrics and topology default to the loopback observability address ($INFLUXDB3_EDR_OBSERVABILITY_LISTEN). In Docker, run these through docker exec with no arguments—see Run EDR in Docker.