Recover a crashed node
Use this procedure to recover a node that stopped ungracefully–for
example, the process crashed, was killed (kill -9), or its container was
force-stopped–or that is wedged in the stopping state.
This procedure applies to both the Parquet engine and the upgraded storage engine (the default for new clusters).
Why ungraceful stops put writes at risk
A node buffers recently acknowledged writes in its write-ahead log (WAL) and
only periodically captures them in a snapshot.
A graceful influxdb3 stop node
against the live node drains this WAL tail before the node reads as
stopped–the Parquet engine flushes the WAL and the upgraded storage
engine snapshots it.
A node that dies without a graceful stop skips that drain:
- Running
stop nodeagainst the dead process cannot drain anything. - Removing the node purges its object-store file paths, permanently deleting any acknowledged writes not covered by its last snapshot.
- Sending the process a bare
SIGTERM(without callingstop node) does not force a WAL snapshot on the upgraded storage engine, so a plain process shutdown can also leave a WAL tail behind.
On clusters that have fully adopted the upgraded storage engine,
remove node refuses (HTTP 409) to remove a stopped
node that still has an un-snapshotted WAL tail unless you pass
--force-finalize.
Parquet clusters and clusters still mid-upgrade do
not have this safeguard–on those clusters, completing this recovery
procedure before removal is the only protection against losing the tail.
Recover the node
Restart a server process with the same
--node-id(environment variable:INFLUXDB3_NODE_ID) and the same object store configuration. This is functionally the same as restarting the original node: on startup, WAL recovery finds and replays the un-drained WAL files, so the acknowledged writes are safe again.influxdb3 serve \ --node-idNODE_ID\ --cluster-idCLUSTER_ID\ # ...same object store configuration as the original nodeStop the node gracefully and wait for it to reach
stopped. This drains the WAL tail (Parquet: WAL flush; upgraded engine: WAL snapshot):influxdb3 stop node --node-idNODE_IDThe command waits (up to
--timeout, default5m) for the node to confirm the stop.Remove the node if you intend to permanently remove it from the cluster:
influxdb3 remove node --node-idNODE_IDIf you are recovering the node to keep using it, skip this step–after a graceful stop, you can start it again at any time.
If the node cannot be brought back
If the node genuinely cannot be restarted (for example, its data is
unrecoverable or the hardware is gone) and you accept the possible loss of
un-snapshotted writes, use the
--force-finalize option
to force the removal:
influxdb3 remove node --node-id NODE_ID --force-finalize--force-finalize can lose data: it removes the node using
whatever snapshot exists in object storage and permanently deletes any
acknowledged writes not covered by that snapshot.
If the node had finished snapshotting and only failed to report a clean
stop, nothing is lost.
Always attempt the recovery procedure first.
Verify node status
At any point, check node states with
influxdb3 show nodes:
influxdb3 show nodesWas this page helpful?
Thank you for your feedback!
Support and feedback
Thank you for being part of our community! We welcome and encourage your feedback and bug reports for InfluxDB 3 Enterprise and this documentation. To find support, use the following resources:
Customers with an annual or support contract can contact InfluxData Support. Customers using a trial license can email trial@influxdata.com for assistance.