Manage the node lifecycle

Every InfluxDB 3 Enterprise server process registers itself as a node in the catalog—the metadata store that tracks databases, tables, and nodes. The catalog is the source of truth for a node’s identity and state, and it persists in object storage—independently of the process, its container, or its host. Understanding how a node moves through its states helps you restart, upgrade, scale, and decommission InfluxDB 3 Enterprise safely.

• Node states
• Lifecycle overview
• Register a node
• Stop a node
• Remove a node
• Re-register a node
• Restart compared to removal
• Deploy with an orchestrator
• Verify node state
• Troubleshoot node lifecycle issues

Node states

The catalog records one of the following states for each node:

Node identity has two parts:

• Node ID: The name you assign with --node-id. It identifies the node across restarts and is the value you pass to node management commands.
• Instance ID: A UUID that InfluxDB 3 Enterprise generates the first time a node ID registers. A node that restarts with the same node ID reuses its existing instance ID.

Lifecycle overview

A node reaches stopped by one of two paths:

• Process shutdown (SIGTERM, SIGINT, or Ctrl-c) moves the node directly from running to stopped.
• influxdb3 stop node moves the node from running to stopping, and then to stopped after the node acknowledges its final snapshot.

Only the second path records a final snapshot sequence, and only the second path frees the node’s licensed cores for other nodes.

Register a node

When you start a server with influxdb3 serve, the node registers itself in the catalog and enters the running state:

influxdb3 serve \
  --node-id 
NODE_ID
 \
  --cluster-id 
CLUSTER_ID
 \
  --object-store s3 \
  --bucket 
BUCKET_NAME

Replace the following:

• NODE_ID: A unique identifier for this node
• CLUSTER_ID: The identifier shared by all nodes in the cluster
• BUCKET_NAME: The object storage bucket for the cluster

Registration is how a node claims its node ID. If the node ID already exists in the catalog, InfluxDB 3 Enterprise applies the re-registration rules before accepting the node.

Stop a node

How you stop a node depends on whether the node stays in the cluster.

Stop a node for a restart

To restart a node in place—for a rolling upgrade, a configuration change, or a node reschedule—signal the process (SIGTERM, SIGINT, or Ctrl-c). The node stops accepting writes, flushes its write-ahead log (WAL) buffer to object storage, waits for an in-progress snapshot to finish, and marks itself stopped in the catalog.

Because the final flush writes buffered data to the WAL in object storage, acknowledged writes survive the shutdown, and WAL replay restores them when the node restarts with the same node ID.

Stop a node before removing it

To take a node out of the cluster permanently, use influxdb3 stop node against the live node:

influxdb3 stop node --node-id 
NODE_ID

The stop proceeds in two phases:

1. InfluxDB 3 Enterprise marks the node stopping in the catalog.
2. The node completes its stop cascade, draining its WAL tail—the writes buffered since the last snapshot.
3. The node acknowledges the stop, reads as stopped, and its licensed cores are freed for other nodes.

By default, the command waits for the node to reach stopped (up to --timeout, default 5m). Use --no-wait to return as soon as the cluster accepts the request.

Stop a node that isn’t running

Stopping a node that has already stopped returns HTTP 400 Bad Request:

tried to stop a node (NODE_ID) that is already stopped

The error is safe to ignore—it reports that the node reached the state you asked for. Automation that stops nodes should treat this response as success rather than retrying.

Stop a node whose process is already gone

--host doesn’t have to point at the node you’re stopping. To stop a node whose process is dead, send the request to a running node in the same cluster:

influxdb3 stop node \
  --node-id 
NODE_ID
 \
  --host http://
RUNNING_NODE
:8181 \
  --token 
ADMIN_TOKEN

This clears a stale running entry that would otherwise keep the cluster expecting a node that never comes back. Remember that a dead process can’t drain its WAL tail, so treat the node as crashed and follow Recover a crashed node before you remove it.

Remove a node

Removal is permanent. After a node reaches stopped, remove it with influxdb3 remove node:

influxdb3 remove node --node-id 
NODE_ID

InfluxDB 3 Enterprise marks the node removing, drains its data up to its final snapshot, and then purges the node’s catalog entry and its object-store file paths.

Repeating a remove request for a node that’s already removing succeeds without changing state (HTTP 200 OK).

Removal completes on the compactor’s schedule

removing isn’t instantaneous, and you can’t force it to finish. The compactor drives removal: it waits until its per-node compaction floor passes the node’s recorded final snapshot sequence, and only then deletes the node’s object-store prefixes and purges the catalog entry. Nothing is deleted before it’s absorbed.

A node can therefore sit in removing for hours if the compactor is backed up, under-resourced, or restarting—this is the removal working as designed, waiting on compaction. Removing several nodes at once multiplies the backlog it has to absorb.

If nodes aren’t clearing, check the compactor before the nodes:

# Compare each node's compaction floor against its snapshots
influxdb3 query \
  --database _internal \
  --token 
AUTH_TOKEN
 \
  "SELECT node_id, node_prefix, last_snapshot_sequence, \
   last_compacted_wal_sequence_number FROM system.pt_compaction_nodes"

Nodes disappear from show nodes as each removal completes.

When removal is refused

InfluxDB 3 Enterprise refuses removal (HTTP 409 Conflict) in the following cases:

Remove a compactor node

A node running in compact mode holds the cluster’s single-writer compaction lease and can’t be removed. To retire the host running your compactor, start a replacement compactor node that reuses the same node ID, as described in Configure specialized cluster nodes.

Re-register a node

Whether a node ID can be claimed again depends on the current state of the node in the catalog:

Because a node restarting with the same node ID reuses its existing instance ID, an ordinary restart always satisfies these rules—even if the node still reads as running after an ungraceful stop.

Restart compared to removal

Choosing the wrong operation is the most common way to lose data during routine maintenance.

Restarting a node is not a removal: the node keeps its catalog entry, its instance ID, and its object-store files. Removal is the only operation that purges them.

For upgrade-specific sequencing and catalog version constraints, see Troubleshooting cluster upgrades.

Deploy with an orchestrator

Helm, Kubernetes, and Ansible deployments drive the node lifecycle on your behalf. The following guidance keeps orchestrated restarts on the graceful path.

Kubernetes and Helm

Give each node a stable node ID. Use a StatefulSet so pod names are stable and ordinal-based, and derive --node-id from the pod name.

The official InfluxDB 3 Enterprise Helm chart does this already—it runs a StatefulSet per node mode and sets --node-id=$(POD_NAME) from metadata.name.

A Deployment generates a new random pod name on every rollout, which registers a new node in the catalog on each restart and leaves the old entries behind.

Set a termination grace period that fits your WAL. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (default 30), and then sends SIGKILL. A node that’s still flushing when SIGKILL arrives stops ungracefully. Set terminationGracePeriodSeconds well above your observed shutdown time:

spec:
  template:
    spec:
      terminationGracePeriodSeconds: 300

Don’t remove nodes during a rolling update. A helm upgrade or kubectl rollout restart terminates and recreates pods with the same names. Each node re-registers with its existing node ID, which is exactly what you want. Never put influxdb3 remove node in a preStop hook—it turns every rollout into a permanent decommission.

Sequence rollouts across node modes yourself. The Helm chart runs a separate StatefulSet per node mode, but the image tag is a single chart-wide value, so one helm upgrade rolls every mode at once. Kubernetes doesn’t order rollouts across StatefulSets, so a plain upgrade doesn’t follow the recommended node upgrade order. Freeze the modes you aren’t upgrading yet with updateStrategy.rollingUpdate.partition and release them one at a time—see the Helm tab in Perform a rolling upgrade.

Protect nodes from concurrent drains. A node drain (kubectl drain, cluster autoscaler, or a managed node-pool upgrade) evicts pods the same way a rollout does, but nothing stops several nodes from draining at once. Enable the chart’s pod disruption budget so voluntary disruptions take one node at a time:

ingester:
  podDisruptionBudget:
    enabled: true
    maxUnavailable: 1

Set it per node mode (ingester, querier, compactor, and processingEngine). The chart creates a budget for a mode only when that mode runs more than one replica, so single-replica modes still drain without protection—make sure their termination grace period is long enough to finish the final flush.

Scale down deliberately. Reducing a StatefulSet’s replica count stops the pods but leaves the nodes in the catalog as stopped. Removing them is a separate, deliberate step:

# 1. Confirm the node reached the stopped state
influxdb3 show nodes

# 2. Remove the stopped node from the cluster
influxdb3 remove node --node-id 
NODE_ID

Disabling a node mode leaves its nodes in the catalog. Setting a mode’s enabled: false (for example, compactor.enabled=false) deletes that StatefulSet, but the nodes it ran keep their catalog entries. Stop and remove them deliberately, in that order, the same way you would after scaling down. A compactor node can’t be removed at all, so disable the compactor only when you’re retiring the cluster or replacing it with a node that reuses the same node ID.

Uninstalling doesn’t clean the catalog. helm uninstall removes Kubernetes resources, but the catalog and object storage persist, and the nodes remain as catalog entries. Reinstalling with the same node IDs and object store reattaches those nodes.

Ansible and systemd

Let systemd send SIGTERM. The systemd default KillSignal is SIGTERM, which starts a graceful shutdown—don’t override it with SIGKILL.

Raise TimeoutStopSec. If the node doesn’t exit within TimeoutStopSec, systemd escalates to SIGKILL. Set it above your observed shutdown time:

[Service]
KillSignal=SIGTERM
TimeoutStopSec=300

Roll one node at a time. Use serial: 1 in your playbook and confirm each node returns to running before proceeding. Group your inventory by node mode and order the plays to match the recommended node upgrade order—for a complete playbook, see the Ansible tab in Perform a rolling upgrade.

Don’t put remove node in a playbook. A restart keeps the node’s catalog entry and object-store files; removal purges them. Reserve influxdb3 remove node for deliberate decommissioning, never for routine configuration or version rollouts.

Never use kill -9. Ad hoc kill -9, docker kill, and force-stopped containers all skip the final flush.

Verify node state

Use influxdb3 show nodes to check the state of every node in the cluster:

influxdb3 show nodes

The output includes a state column for each node—abbreviated here to the lifecycle-relevant columns:

+---------+--------+------------+---------+
| node_id | mode   | core_count | state   |
+---------+--------+------------+---------+
| node-1  | ingest | 1          | running |
| node-2  | ingest | 1          | stopped |
+---------+--------+------------+---------+

Other nodes observe a state change after their catalog sync interval (default 10 seconds), so allow for that delay when scripting checks.

You can also query the system.nodes table in the _internal database:

influxdb3 query \
  --database _internal \
  --token 
AUTH_TOKEN
 \
  "SELECT node_id, mode, state, updated_at FROM system.nodes"

Replace AUTH_TOKEN with a token that has permission to query the _internal database.

Troubleshoot node lifecycle issues

A node is stuck in stopping

The node was marked stopping but never acknowledged the stop—usually because the process died mid-cascade. Follow Recover a crashed node: restart the node with the same node ID, stop it gracefully, and then remove it if you still intend to.

A node still reads as running after it crashed

A node that dies without a graceful shutdown never updates its catalog entry, so it keeps its last recorded state. Restarting the node with the same node ID re-registers it—the re-registration rules allow it because the instance ID matches.

Removal fails with a 409 conflict

See When removal is refused for each condition and its resolution. Prefer restarting and gracefully stopping the node over --force-finalize. Forcing removal discards the unsnapshotted writes the safeguard was protecting, and it starts an object-store cleanup for a node that never recorded a final snapshot. Restart the node, stop it gracefully, and then remove it.

A node stays in removing

Removal waits on the compactor, so a healthy cluster with a compaction backlog clears removing nodes slowly rather than never. See Removal completes on the compactor’s schedule for how to measure the remaining gap.

Check whether the compactor is making progress at all before assuming the removal is stuck:

• Confirm the compactor node is running and isn’t restarting or being OOM-killed.
• Watch the compaction floors in system.pt_compaction_nodes advance between queries.

Don’t scale a replacement node into a removing node’s ID while you wait—the registration is refused until removal finishes.

Nodes multiply in the catalog after each restart

Each restart registered a new node ID. Check that your deployment assigns a stable node ID—see Kubernetes and Helm. Stop and remove the stale entries after confirming which nodes are current.

Writes fail during a rolling upgrade

This is usually a catalog version constraint rather than a lifecycle problem. See Troubleshooting cluster upgrades.