Recover a crashed node
Use this procedure to recover a node that stopped ungracefully — for
example, the process crashed, was killed (kill -9), or its container was
force-stopped.
It also applies to a node wedged in the stopping state.
This procedure applies to both the Parquet engine and the upgraded storage engine (the default for new clusters in InfluxDB 3 Enterprise 3.11+).
Why ungraceful stops put writes at risk
A node stores recently acknowledged writes in its write-ahead log (WAL) until it
captures them in a snapshot.
These buffered writes are the node’s WAL tail.
Run the influxdb3 stop node
command against a running node to save the WAL tail before the node reports a
stopped state.
A node that dies without a graceful stop skips that drain:
- Running
stop nodeagainst the dead process cannot drain anything. - Removing the node purges its object-store file paths, permanently deleting any acknowledged writes not covered by its last snapshot.
- Sending the process a bare
SIGTERM(without callingstop node) does not force a WAL snapshot on the upgraded storage engine, so a plain process shutdown can also leave a WAL tail behind.
On clusters that have fully adopted the upgraded storage engine,
remove node refuses (HTTP 409) to remove a stopped
node that still has a WAL tail unless you pass --force-finalize.
Parquet clusters and clusters still mid-upgrade do not have this safeguard.
On those clusters, completing this recovery procedure before removal is the only
protection against losing the tail.
Recover the node
Restart a server process with the same
--node-id(environment variable:INFLUXDB3_NODE_ID) and the same object store configuration. This is functionally the same as restarting the original node: on startup, WAL recovery replays the WAL files that were not drained, so the acknowledged writes are safe again.influxdb3 serve \ --node-idNODE_ID\ --cluster-idCLUSTER_ID\ # ...same object store configuration as the original nodeStop the node gracefully and wait for it to reach
stopped. This drains the WAL tail (Parquet: WAL flush; upgraded engine: WAL snapshot):influxdb3 stop node --node-idNODE_IDThe command waits (up to
--timeout, default5m) for the node to confirm the stop.Remove the node if you intend to permanently remove it from the cluster. If you are recovering the node to keep using it, skip this step—after a graceful stop, you can start it again at any time.
influxdb3 remove node --node-idNODE_ID
If the node cannot be brought back
If the node cannot be restarted (for example, its data is
unrecoverable or the hardware is gone) and you accept the possible loss of
writes not yet captured in a snapshot, use the
--force-finalize option
to force the removal:
influxdb3 remove node --node-id NODE_ID --force-finalize--force-finalize can lose data: it removes the node using
whatever snapshot exists in object storage and permanently deletes any
acknowledged writes not covered by that snapshot.
If the node had finished snapshotting and only failed to report a clean
stop, nothing is lost.
Always attempt the recovery procedure first.
Verify node status
At any point, check node states with
influxdb3 show nodes:
influxdb3 show nodesWas this page helpful?
Thank you for your feedback!
Support and feedback
Thank you for being part of our community! We welcome and encourage your feedback and bug reports for InfluxDB 3 Enterprise and this documentation. To find support, use the following resources:
Customers with an annual or support contract can contact InfluxData Support. Customers using a trial license can email trial@influxdata.com for assistance.