Troubleshoot common EDR issues

When something looks wrong, start here. The quick index maps a symptom to the first tool to run; the sections below go deeper.

Quick index

Symptom, what to run first, and what it tells you:

• Replication stopped (“halted”)—run edr-inspect metrics (live), or check the log’s last REPLICATION HALTED ... (after a crash). Tells you the halt class (auth, schema, or network). Fix the cause—the halted batch retries as a probe and auto-resumes; nothing is skipped.
• Lag climbing (“behind”)—run edr-inspect state, then edr-inspect metrics. Tells you whether anything is actually lost (rare) or just owed backlog that drains once the link recovers. Lag != loss.
• Data missing—run edr-inspect state (lost and gap counts). The only authority on loss. A non-zero lost(unrecoverable) here is real; a scary metrics reading is not.
• Agent crashed or restarted—capture the log, then follow the log post-mortem procedure below. In-memory metrics reset on restart—the log is the only witness.

Agent won’t start

• “config must include at least one of ‘downstream’ or ‘upstreams’”—config has neither section. Add one.
• “downstream.historic_fill must be set explicitly …”—no historic intent declared. Add historic_fill: { mode: none } (live-only), full, or since. See Historic fill.
• “historic_fill mode ‘since’ requires a ‘since’ value”, or “… is not parseable”—since missing or malformed. Set since to a whole-day duration (7d) or an ISO date (2026-06-01); sub-day units (30m or 6h) are rejected.
• “Multiple ingest nodes detected… idempotent_writes is false”—multi-node store without the guarantee. Add idempotent_writes: true or point at a single-node store.
• “failed to resolve token”—token file missing from store. Check --token-store path and file names.
• "… changed — requires full restart" (on reload)—a non-reloadable field changed. Restart the agent.

Data not flowing

1. First-line triage: edr-inspect state <STATE_LOCATION> (what’s owed and what’s in progress, even if the agent is wedged) and edr-inspect topology and edr-inspect metrics (live health and flow). See Triage CLI. For what the state files mean—and before deleting ANY of them—see State and recovery.
2. Check node info: curl http://127.0.0.1:9091/edr/v1/node_info (observability listener, loopback)
3. Check connectivity: curl http://destination:9090/health (network listener)
4. Check write-ahead log (WAL) files exist: ls <DATA_DIR>/<NODE_ID>/pt_wal/
5. Check scope includes the database and table.
6. Verbose logs: restart with RUST_LOG=debug.

Common log messages

• historic fill: MANIFEST GAP ...—expected: the compactor deleted old manifests; recovery falls back to WAL files, then to cv2. See Historic fill.
• cv2 over-replication warnings—expected when recovering compacted data; idempotent writes absorb it.
• ... POTENTIAL DATA LOSS—not expected: data was deleted from every tier (WAL, gen0, cv2) before EDR could read it. Investigate retention settings. Historic fill emits this as historic fill: MANIFEST GAP — ... no cv2 files found covering the gap — POTENTIAL DATA LOSS.
• Schema conflict, or channel blocked, retrying every 60s—destination rejected a write due to a type mismatch. Drop the conflicting table on the destination; the retry recreates it with the source schema.
• Unauthorised—destination rejected the auth token. Update the token file; the next retry picks it up.
• Cursor reset detected—InfluxDB WAL IDs reset (for example, after a data wipe with state retained). The agent auto-detects and resets its cursor.
• failed to connect to downstream followed by a burst of batch send failed, right after a restart—restart-ordering gap: the source came up before the destination’s listener was accepting. Clears at the next connected to downstream or edge connected. Benign and self-healing; not data loss.
• rejecting data: upstream has not called /connect (destination side)—momentary: data arrived a beat before the /connect handshake re-registered (typical just after a restart). Self-clears once the source re-issues /connect.
• Source REPLICATION HALTED: auth failure and destination batch write failed: auth failure on the same batch_id—both ends agree, so this is a genuine token mismatch on that hop, not a source-side phantom. Update the token (see Unauthorised above); the next probe resumes.

Protocol incompatible (agents of different versions)

Symptoms: nothing flows to one downstream; health shows Halted; the log carries protocol halt: replication to this downstream is STOPPED once at ERROR and a PROTOCOL-HALTED WARN every minute; the downstream’s log (or the upstream’s error) names both protocol versions and both minimums.

• This is deliberate: the two agents’ protocol versions cannot interoperate (or the downstream lacks a capability the config requires, for example encoding: pt). The message says which side must be upgraded.
• Upgrade the named agent. Negotiation re-runs on every report (default 10s), so the halted upstream resumes by itself within one report interval (protocol compatibility restored in the log); a config reload also re-negotiates immediately.
• Agents released before protocol versioning (0.1.x and 0.2.x) are always accepted as protocol v1—they can never be refused by current builds.

See Compatibility for the full protocol version matrix.

Stale binaries

After replacing a binary (for example, an EDR upgrade), make sure the old process is actually gone before concluding the new one misbehaves: stop the agent, start it again, and confirm the running binary’s modification time predates the process start time. A lingering old process is a common cause of “the new version didn’t take effect”.

Port conflicts and orphan processes

EDR agents, proxies, and InfluxDB instances hold their listen ports until fully stopped. If a start fails with “address in use”, find the holder with lsof -ti :<PORT>. Process managers that track PID files can lose track of processes across crashes—always verify with ps aux | grep influxdb3 | grep -v grep that no orphans remain before restarting. Orphaned old agents reading the same state file as a new agent will fight over the cursor.

Replication lag

• Lag is not loss. A high live-lag or pending_bytes is almost always owed backlog that drains once the destination catches up, not missing data. edr-inspect state (lost(unrecoverable), pending gaps) is the authority on what—if anything—is actually unrecoverable. Confirm there before treating lag as an incident.
• High pending_bytes indicates that the destination is slow or unreachable.
• If WAL files are deleted before replication, increase --wal-snapshots-to-keep on the source server. See Size WAL retention. Gap fill will recover, but at a cost.
• If WAL discovery is slow, lower --poll-interval-ms to list the object store more frequently.
• If the link is constrained, check whether a bandwidth_schedule window is active (limited or silent shows in the UI and logs).

Incorrect field values in wide tables

If a table has more than 100 fields and was replicated by 0.2.0, rc.1, or rc.2, upgrade to rc.3 or GA on a supported server line. Earlier builds can silently assign values to the wrong field names when types match, or halt with a schema conflict when types differ. Verify field-level values at the destination or re-replicate affected tables after upgrading. Matching row counts do not prove that field values are correct. See the rc.3 release warning.

Source WAL / disk growing

• The source server owns WAL lifecycle; a large or growing pt_wal footprint usually means --wal-snapshots-to-keep is set high (often to cover long outages). That is working as intended—WAL is retained until the source rolls it to gen0.
• If the buildup of already-replicated WAL is a problem, enable EDR’s opt-in --wal-cleanup-enabled so the agent prunes the replicated tail itself. See Size WAL retention. High server WAL retention plus opt-in agent cleanup is the supported production pattern for EDR 1.0.0. The sweep stays below all four floors: cursor, snapshot boundary minus margin, historic-fill floor, and queued or in-flight work plus unresolved gap ranges.
• After enabling, confirm gaps_pending and wal_files_lost_total stay flat—a rise is not expected (cleanup is designed never to create gaps) and warrants a look. A sweep that prunes anything logs wal cleanup: deleted replicated WAL files; delete failures warn and hold the watermark.

Log post-mortem (after a crash or restart)

When an agent has crashed, been killed, or restarted, the live views can’t help—its in-memory metrics reset on restart, so the logs are the only witness. EDR logs structured JSON to stdout; capture them first (a --rm container discards them on removal):

docker logs <CONTAINER> > edr.log 2>&1     # or: journalctl -u <EDR_UNIT> -o cat > edr.log

Then reconstruct the final run:

1. The last startup banner marks the final run (count banners to get the number of restarts).
2. The last successful send is last-known-good.
3. A shutdown signal received followed by a ... shutting down cascade means a clean stop; no shutdown markers means an unclean exit (crash, out-of-memory (OOM), or kill).
4. Identify the dominant ERROR or WARN class and its fields (table, upstream, error).
5. Write a one-line verdict and remediation.

For the complete rule set—delivery failures, data loss, cursor and state, connectivity, config, and infrastructure signatures—see the EDR log post-mortem playbook. The manual procedure above always works without any additional tooling.

Docker-specific troubleshooting

See Run EDR in Docker.