Historic fill and gap fill

1. Historic fill
2. Gap fill

Historic fill

Historic fill replicates data that existed on the source before EDR was deployed, or before this destination was added.

historic_fill is required—there is no default. This is deliberate: you must declare your intent for existing data before EDR attaches to a source. If historic_fill is absent, the agent refuses to start and prints the choices below.

downstream:
  # ... name, address, auth_token ...
  historic_fill:
    mode: none            # required -- choose: none | full | since
    # mode: full
    # mode: since
    # since: "7d"         # only with mode: since -- whole days (Nd) or ISO date (no sub-day units)

Start point. On a fresh start, every mode (including none) seeds the live cursor to the snapshot boundary—the frontier WAL ID near the leading edge—so live replication begins from the most recent WAL, never from WAL 1. The difference between the modes is only what (if anything) gets backfilled below that boundary. none does no backfill; full and since hand the below-boundary data to historic fill.

The boundary that divides historic fill from live replication is in WAL-ID (ingest) space and is computed once at startup. The since threshold filters by data timestamp (event time).

How historic fill works

On first startup (no prior cursor), the agent computes a snapshot boundary—the frontier WAL ID. The live WAL replicator handles everything after the frontier; historic fill handles everything before it:

1. Snapshot manifests (.ptsnap) are read to build a work list of gen0 files, each carrying its WAL range and time range.
2. Gen0 files on the work list are read and replicated at lower priority than live data unless a priority rule claims historic fill explicitly. Historic-fill concurrency follows concurrent_sends. With concurrent_sends: 1, fill processes one file at a time. Higher values allow parallel backfill and require idempotent_writes: true. The pipeline continues across snapshot boundaries.
3. cv2 files (compaction levels L1 and above) are the primary medium for genuinely old data—data whose gen0 files and snapshot manifests have already been compacted away. cv2 candidates are enumerated by time window and processed as first-class work list entries.

Progress is persisted in a manifest file, so historic fill survives agent restarts and resumes where it left off. When the work list is exhausted the fill is marked complete and never runs again (the manifest records completion).

What to expect in the logs

Historic fill is deliberately loud. These messages are normal:

• historic fill: starting — N snapshots, M files—work list built.
• historic fill: MANIFEST GAP — snapshot manifests ... deleted by compactor—expected whenever the compactor has cleaned up older snapshot manifests. The agent recovers the gap from surviving WAL files where possible, and from cv2 files for the evicted head.
• historic fill: MANIFEST GAP only partially covered by WAL files ...—the mixed three-tier situation: a suffix of the gap comes from WAL files, the evicted head from cv2. cv2 is the primary medium for that data.
• historic fill: added N cv2 work list entries ...—cv2 recovery in progress.
• Warnings about over-replication when sending cv2 files—expected whenever EDR recovers compacted data. cv2 files are compacted, so they contain data beyond the exact recovery window. Block-level time filters cut this down, but some over-replication is inherent. The destination’s idempotent writes absorb the duplicates, so correctness is unaffected. The warnings report the bandwidth cost; they don’t mean something is wrong.
• historic fill: completed or historic fill: already completed — skipping—done; restart-safe.

One historic fill message is not normal: POTENTIAL DATA LOSS. See Common log messages.

During a fill, the metrics API reports historic_fill_active, gapfill_files_done, and gapfill_files_total so you can track progress.

Gap fill

Gap fill recovers WAL ranges missed during live replication, including delivery failures, transient object-store reads, and server WAL eviction. The agent records each range in the gap ledger. Recovery first re-reads the exact raw WAL files if they still exist. A temporarily unreadable file causes recovery to wait briefly and retry the read. Only an evicted range falls back to snapshot manifests, gen0, and compacted cv2 files. gapfill_files_from_wal_total counts files recovered from raw WAL.

An agent behind the retention floor records the whole evicted span as one gap and resumes live replication above that floor. Gap fill recovers the span in the background. If a channel is halted, gapfill_blocked_on_halt and the log identify the blocked recovery.

Operationally:

• Gaps appear in metrics as gaps_pending and in the gap ledger file next to the state file. After recovery they are marked resolved.
• A recovering agent is visibly recovering. Gaps a round is actively working move from gaps_pending to gaps_in_progress (pending counts only untouched gaps), and the round’s phases publish progress: gapfill_discovery_candidates and gapfill_discovery_scanned while it searches the store for covering files, then gapfill_files_done and gapfill_files_total while it replicates them. If those gauges are moving, recovery is working—even when gaps_resolved_total hasn’t ticked yet (gaps settle when their round completes).
• gapfill_files_vanished_total counts covering files the server’s compactor removed mid-recovery (the agent re-discovers their replacements automatically). A high rate means recovery is racing compaction—expect rounds to take longer.
• Recovery from gen0 is precise. Recovery from cv2 over-replicates (see above)—expected, absorbed by idempotent writes.
• If you see gaps regularly, increase --wal-snapshots-to-keep on the source InfluxDB. See Size WAL retention; recovery from compacted files is a safety net, not the intended steady state.