---
title: Manage the node lifecycle
description: Understand how an InfluxDB 3 Enterprise node moves through the running, stopping, stopped, and removing states—how to restart, stop, and permanently remove nodes, and how to keep Helm, Kubernetes, and Ansible deployments on the graceful shutdown path.
url: https://docs.influxdata.com/influxdb3/enterprise/admin/node-lifecycle/
estimated_tokens: 6568
product: InfluxDB 3 Enterprise
version: enterprise
publisher: InfluxData
canonical: https://docs.influxdata.com/influxdb3/enterprise/admin/node-lifecycle/
date: '2026-08-06T17:50:34+00:00'
lastmod: '2026-08-06T17:50:34+00:00'
---

Every InfluxDB 3 Enterprise server process registers itself as a *node* in the
catalog—the metadata store that tracks databases, tables, and nodes.
The catalog is the source of truth for a node’s identity and state, and it
persists in object storage—independently of the process, its container, or its
host.
Understanding how a node moves through its states helps you restart, upgrade,
scale, and decommission InfluxDB 3 Enterprise safely.

* [Node states](#node-states)
* [Lifecycle overview](#lifecycle-overview)
* [Register a node](#register-a-node)
* [Stop a node](#stop-a-node)
* [Remove a node](#remove-a-node)
* [Re-register a node](#re-register-a-node)
* [Restart compared to removal](#restart-compared-to-removal)
* [Deploy with an orchestrator](#deploy-with-an-orchestrator)
* [Verify node state](#verify-node-state)
* [Troubleshoot node lifecycle issues](#troubleshoot-node-lifecycle-issues)

## Node states

The catalog records one of the following states for each node:

|  State   |                             Description                              |
|----------|----------------------------------------------------------------------|
|`running` |        The node started and registered itself in the catalog         |
|`stopping`|A graceful stop was requested, but the node hasn’t acknowledged it yet|
|`stopped` |   The node acknowledged its final snapshot and completed shutdown    |
|`removing`|      The node is marked for permanent removal from the cluster       |

Node identity has two parts:

* **Node ID**: The name you assign with[`--node-id`](/influxdb3/enterprise/reference/config-options/#node-id).
  It identifies the node across restarts and is the value you pass to node
  management commands.
* **Instance ID**: A UUID that InfluxDB 3 Enterprise generates the first time a
  node ID registers.
  A node that restarts with the same node ID reuses its existing instance ID.

## Lifecycle overview

stateDiagram-v2
[\*] --\> running: influxdb3 serve
running --\> stopped: SIGTERM or SIGINT
running --\> stopping: influxdb3 stop node
stopping --\> stopped: node acknowledges final snapshot
stopped --\> running: influxdb3 serve (same node ID)
stopped --\> removing: influxdb3 remove node
removing --\> [\*]: catalog entry and files purged

A node reaches `stopped` by one of two paths:

* **Process shutdown** (`SIGTERM`, `SIGINT`, or Ctrl-c) moves the node directly
  from `running` to `stopped`.
* **[`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)**moves the node from `running` to `stopping`, and then to `stopped` after the
  node acknowledges its final snapshot.

Only the second path records a final snapshot sequence, and only the second
path frees the node’s licensed cores for other nodes.

## Register a node

When you start a server with[`influxdb3 serve`](/influxdb3/enterprise/reference/cli/influxdb3/serve/), the node
registers itself in the catalog and enters the `running` state:

```bash
influxdb3 serve \
  --node-id NODE_ID \
  --cluster-id CLUSTER_ID \
  --object-store s3 \
  --bucket BUCKET_NAME
```

Replace the following:

* `NODE_ID`:
  A unique identifier for this node
* `CLUSTER_ID`:
  The identifier shared by all nodes in the cluster
* `BUCKET_NAME`:
  The object storage bucket for the cluster

Registration is how a node claims its node ID.
If the node ID already exists in the catalog, InfluxDB 3 Enterprise applies the[re-registration rules](#re-register-a-node) before accepting the node.

## Stop a node

How you stop a node depends on whether the node stays in the cluster.

### Stop a node for a restart

To restart a node in place—for a rolling upgrade, a configuration change, or a
node reschedule—signal the process (`SIGTERM`, `SIGINT`, or Ctrl-c).
The node stops accepting writes, flushes its write-ahead log (WAL) buffer to
object storage, waits for an in-progress snapshot to finish, and marks itself`stopped` in the catalog.

Because the final flush writes buffered data to the WAL in object storage,
acknowledged writes survive the shutdown, and WAL replay restores them when the
node restarts with the same node ID.

### Stop a node before removing it

To take a node out of the cluster permanently, use[`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)against the **live** node:

```bash
influxdb3 stop node --node-id NODE_ID
```

The stop proceeds in two phases:

1. InfluxDB 3 Enterprise marks the node `stopping` in the catalog.
2. The node completes its stop cascade, draining its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail)—the
   writes buffered since the last snapshot.
3. The node acknowledges the stop, reads as `stopped`, and its licensed cores
   are freed for other nodes.

By default, the command waits for the node to reach `stopped`(up to `--timeout`, default `5m`).
Use `--no-wait` to return as soon as the cluster accepts the request.

> [!Important]
> #### Run stop node against the live node—don’t kill it first
>
> `stop node` is how a node drains its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail) and
> records a final snapshot.
> If you kill the process first and run `stop node` afterward, the dead process
> can’t drain anything, and[removing the node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)can permanently delete the stranded writes.
> If a node already stopped ungracefully, follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/).

> [!Note]
> #### A flush isn’t a snapshot
>
> A bare `SIGTERM` flushes the WAL buffer, but it doesn’t force a new snapshot.
> The writes are durable and replay on restart, but they remain part of the WAL
> tail.
> That distinction matters only before[removing a node](#remove-a-node), because removal purges the node’s WAL
> files.

### Stop a node that isn’t running

Stopping a node that has already stopped returns HTTP `400 Bad Request`:

```
tried to stop a node (NODE_ID) that is already stopped
```

The error is safe to ignore—it reports that the node reached the state you
asked for.
Automation that stops nodes should treat this response as success rather than
retrying.

### Stop a node whose process is already gone

`--host` doesn’t have to point at the node you’re stopping.
To stop a node whose process is dead, send the request to a **running** node in
the same cluster:

```bash
influxdb3 stop node \
  --node-id NODE_ID \
  --host http://RUNNING_NODE:8181 \
  --token ADMIN_TOKEN
```

This clears a stale `running` entry that would otherwise keep the cluster
expecting a node that never comes back.
Remember that a dead process can’t drain its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail), so
treat the node as crashed and follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) before you
remove it.

## Remove a node

Removal is permanent.
After a node reaches `stopped`, remove it with[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/):

```bash
influxdb3 remove node --node-id NODE_ID
```

InfluxDB 3 Enterprise marks the node `removing`, drains its data up to its final
snapshot, and then purges the node’s catalog entry and its object-store file
paths.

> [!Warning]
> #### Removal deletes the node’s files
>
> Removal permanently deletes the node’s object-store file paths, including its
> WAL files.
> Any acknowledged writes not covered by the node’s final snapshot are deleted
> with them.
> Complete a graceful [`stop node`](#stop-a-node-before-removing-it) first.

Repeating a remove request for a node that’s already `removing` succeeds
without changing state (HTTP `200 OK`).

### Removal completes on the compactor’s schedule

`removing` isn’t instantaneous, and you can’t force it to finish.
The compactor drives removal: it waits until its per-node compaction floor
passes the node’s recorded final snapshot sequence, and only then deletes the
node’s object-store prefixes and purges the catalog entry.
Nothing is deleted before it’s absorbed.

A node can therefore sit in `removing` for hours if the compactor is backed up,
under-resourced, or restarting—this is the removal working as designed, waiting
on compaction.
Removing several nodes at once multiplies the backlog it has to absorb.

If nodes aren’t clearing, check the compactor before the nodes:

```bash
# Compare each node's compaction floor against its snapshots
influxdb3 query \
  --database _internal \
  --token AUTH_TOKEN \
  "SELECT node_id, node_prefix, last_snapshot_sequence, \
   last_compacted_wal_sequence_number FROM system.pt_compaction_nodes"
```

Nodes disappear from `show nodes` as each removal completes.

> [!Warning]
> #### Removal keeps the cluster expecting the node until it completes
>
> While a node is `removing`, its node ID can’t be reclaimed—`removing` is
> terminal.
> A replacement that reuses the ID (for example, a StatefulSet pod recreated
> with the same ordinal name) starts but is refused registration.
> Keep your replica count pinned below the removing ordinals until removal
> finishes.

### When removal is refused

InfluxDB 3 Enterprise refuses removal (HTTP `409 Conflict`) in the following
cases:

|          Condition          |                                 Message                                  |                                                         Resolution                                                         |
|-----------------------------|--------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|
|    Node isn’t `stopped`     |       `node 'NODE_ID' is not fully stopped (current state: STATE)`       |                     [Stop the node gracefully](#stop-a-node-before-removing-it) and wait for `stopped`                     |
| Node runs in `compact` mode |         `node 'NODE_ID' has compact mode and cannot be removed`          |                  Compactor nodes can’t be removed—see [Remove a compactor node](#remove-a-compactor-node)                  |
|Node belongs to a query group| `cannot remove node 'NODE_ID' because it is a member of query group ...` |                                         Remove the node from the query group first                                         |
|  Unsnapshotted WAL remains  |`node 'NODE_ID' has unsnapshotted WAL (wal file N, snapshotted through M)`|Restart the node, stop it gracefully, then remove it—see [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/)|

> [!Note]
> #### The unsnapshotted WAL safeguard requires the upgraded storage engine
>
> The unsnapshotted WAL check applies only to clusters that fully adopted the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters in InfluxDB 3 Enterprise 3.11+).
> Clusters on the Parquet engine, or still mid-upgrade, aren’t guarded—on those
> clusters, a graceful stop before removal is your only protection against
> losing the[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail).
>
> Clusters that started on 3.10 or earlier keep the Parquet engine until you
> restart them with `--upgrade-pacha-tree`.
> If you started a storage engine upgrade, confirm it finished—query`system.upgrade_parquet_node` and check that every node reports `completed`.
> See [Query system data](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables).

### Remove a compactor node

A node running in `compact` mode holds the cluster’s single-writer compaction
lease and can’t be removed.
To retire the host running your compactor, start a replacement compactor node
that reuses the same node ID, as described in[Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/#from-single-node-to-specialized-cluster).

## Re-register a node

Whether a node ID can be claimed again depends on the current state of the
node in the catalog:

|     Current state     |                    Can register again?                     |
|-----------------------|------------------------------------------------------------|
|       `stopped`       |         Yes—any instance can take over the node ID         |
|`running` or `stopping`|Only the same instance ID (an idempotent retry or a restart)|
|      `removing`       |         No—`removing` is terminal for the node ID          |

Because a node restarting with the same node ID reuses its existing instance
ID, an ordinary restart always satisfies these rules—even if the node still
reads as `running` after an ungraceful stop.

> [!Warning]
> #### Don’t reuse the node ID of a removed node
>
> After removal completes, InfluxDB 3 Enterprise purges the node’s catalog entry
> and its object-store file paths.
> While the node is `removing`, registration with that node ID is rejected.
> Assign a new node ID to a replacement node instead of racing the removal.

## Restart compared to removal

Choosing the wrong operation is the most common way to lose data during routine
maintenance.

|                           Goal                            |                             Operation                             |
|-----------------------------------------------------------|-------------------------------------------------------------------|
|Rolling upgrade, config change, pod reschedule, host reboot|Restart the process with the **same** node ID—don’t remove the node|
|      Permanently scale down or decommission hardware      |         `stop node`, verify `stopped`, then `remove node`         |
|          Replace a failed host, keeping its data          | Start a node with the same node ID and object store configuration |

Restarting a node is not a removal: the node keeps its catalog entry, its
instance ID, and its object-store files.
Removal is the only operation that purges them.

For upgrade-specific sequencing and catalog version constraints, see[Troubleshooting cluster upgrades](/influxdb3/enterprise/admin/upgrade/#troubleshooting-cluster-upgrades).

## Deploy with an orchestrator

Helm, Kubernetes, and Ansible deployments drive the node lifecycle on your
behalf.
The following guidance keeps orchestrated restarts on the graceful path.

### Kubernetes and Helm

**Give each node a stable node ID.**Use a StatefulSet so pod names are stable and ordinal-based, and derive`--node-id` from the pod name.

The official[InfluxDB 3 Enterprise Helm chart](https://github.com/influxdata/helm-charts/tree/master/charts/influxdb3-enterprise)does this already—it runs a StatefulSet per[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes) and sets`--node-id=$(POD_NAME)` from `metadata.name`.

A Deployment generates a new random pod name on every rollout, which registers a
new node in the catalog on each restart and leaves the old entries behind.

**Set a termination grace period that fits your WAL.**Kubernetes sends `SIGTERM`, waits `terminationGracePeriodSeconds`(default `30`), and then sends `SIGKILL`.
A node that’s still flushing when `SIGKILL` arrives stops ungracefully.
Set `terminationGracePeriodSeconds` well above your observed shutdown time:

```yaml
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 300
```

> [!Important]
> #### The Helm chart doesn’t set a grace period
>
> The InfluxDB 3 Enterprise Helm chart doesn’t set`terminationGracePeriodSeconds`, so pods inherit the Kubernetes default of
> 30 seconds.
> For nodes with a large WAL, raise it in your `values.yaml` overrides and
> confirm the shutdown completes in the pod logs.

**Don’t remove nodes during a rolling update.**A `helm upgrade` or `kubectl rollout restart` terminates and recreates pods
with the same names.
Each node re-registers with its existing node ID, which is exactly what you
want.
Never put[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)in a `preStop` hook—it turns every rollout into a permanent decommission.

**Sequence rollouts across node modes yourself.**The Helm chart runs a separate StatefulSet per[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes), but the
image tag is a single chart-wide value, so one `helm upgrade` rolls every mode
at once.
Kubernetes doesn’t order rollouts across StatefulSets, so a plain upgrade
doesn’t follow the[recommended node upgrade order](/influxdb3/enterprise/admin/upgrade/#recommended-node-upgrade-order).
Freeze the modes you aren’t upgrading yet with`updateStrategy.rollingUpdate.partition` and release them one at a time—see the**Helm** tab in[Perform a rolling upgrade](/influxdb3/enterprise/admin/upgrade/#perform-a-rolling-upgrade).

**Protect nodes from concurrent drains.**A node drain (`kubectl drain`, cluster autoscaler, or a managed node-pool
upgrade) evicts pods the same way a rollout does, but nothing stops several
nodes from draining at once.
Enable the chart’s pod disruption budget so voluntary disruptions take one node
at a time:

```yaml
ingester:
  podDisruptionBudget:
    enabled: true
    maxUnavailable: 1
```

Set it per[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes)(`ingester`, `querier`, `compactor`, and `processingEngine`).
The chart creates a budget for a mode only when that mode runs more than one
replica, so single-replica modes still drain without protection—make sure their
termination grace period is long enough to finish the final flush.

**Scale down deliberately.**Reducing a StatefulSet’s replica count stops the pods but leaves the nodes in
the catalog as `stopped`.
Removing them is a separate, deliberate step:

```bash
# 1. Confirm the node reached the stopped state
influxdb3 show nodes

# 2. Remove the stopped node from the cluster
influxdb3 remove node --node-id NODE_ID
```

**Disabling a[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes) leaves
its nodes in the catalog.**Setting a mode’s `enabled: false` (for example, `compactor.enabled=false`)
deletes that StatefulSet, but the nodes it ran keep their catalog entries.
Stop and remove them deliberately, in that order, the same way you would after
scaling down.
A [compactor node can’t be removed](#remove-a-compactor-node) at all, so disable
the compactor only when you’re retiring the cluster or replacing it with a node
that reuses the same node ID.

**Uninstalling doesn’t clean the catalog.**`helm uninstall` removes Kubernetes resources, but the catalog and object
storage persist, and the nodes remain as catalog entries.
Reinstalling with the same node IDs and object store reattaches those nodes.

### Ansible and systemd

**Let systemd send `SIGTERM`.**The systemd default `KillSignal` is `SIGTERM`, which starts a graceful
shutdown—don’t override it with `SIGKILL`.

**Raise `TimeoutStopSec`.**If the node doesn’t exit within `TimeoutStopSec`, systemd escalates to`SIGKILL`.
Set it above your observed shutdown time:

```ini
[Service]
KillSignal=SIGTERM
TimeoutStopSec=300
```

**Roll one node at a time.**Use `serial: 1` in your playbook and confirm each node returns to `running`before proceeding.
Group your inventory by[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes) and order
the plays to match the[recommended node upgrade order](/influxdb3/enterprise/admin/upgrade/#recommended-node-upgrade-order)—for
a complete playbook, see the **Ansible** tab in[Perform a rolling upgrade](/influxdb3/enterprise/admin/upgrade/#perform-a-rolling-upgrade).

**Don’t put `remove node` in a playbook.**A restart keeps the node’s catalog entry and object-store files; removal purges
them.
Reserve[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)for deliberate decommissioning, never for routine configuration or version
rollouts.

**Never use `kill -9`.**Ad hoc `kill -9`, `docker kill`, and force-stopped containers all skip the
final flush.

## Verify node state

Use [`influxdb3 show nodes`](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/)to check the state of every node in the cluster:

```bash
influxdb3 show nodes
```

The output includes a `state` column for each node—abbreviated here to the
lifecycle-relevant columns:

```
+---------+--------+------------+---------+
| node_id | mode   | core_count | state   |
+---------+--------+------------+---------+
| node-1  | ingest | 1          | running |
| node-2  | ingest | 1          | stopped |
+---------+--------+------------+---------+
```

Other nodes observe a state change after their catalog sync interval
(default 10 seconds), so allow for that delay when scripting checks.

You can also query the `system.nodes` table in the `_internal` database:

```bash
influxdb3 query \
  --database _internal \
  --token AUTH_TOKEN \
  "SELECT node_id, mode, state, updated_at FROM system.nodes"
```

Replace `AUTH_TOKEN`with a token that has permission to query the `_internal` database.

## Troubleshoot node lifecycle issues

### A node is stuck in stopping

The node was marked `stopping` but never acknowledged the stop—usually because
the process died mid-cascade.
Follow [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/):
restart the node with the same node ID, stop it gracefully, and then remove it
if you still intend to.

### A node still reads as running after it crashed

A node that dies without a graceful shutdown never updates its catalog entry,
so it keeps its last recorded state.
Restarting the node with the same node ID re-registers it—the[re-registration rules](#re-register-a-node) allow it because the instance ID
matches.

### Removal fails with a 409 conflict

See [When removal is refused](#when-removal-is-refused) for each condition and
its resolution.
Prefer restarting and gracefully stopping the node over[`--force-finalize`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/#force-removal-of-a-node-that-did-not-shut-down-cleanly).
Forcing removal discards the unsnapshotted writes the safeguard was protecting,
and it starts an object-store cleanup for a node that never recorded a final
snapshot.
Restart the node, stop it gracefully, and then remove it.

### A node stays in removing

Removal waits on the compactor, so a healthy cluster with a compaction backlog
clears `removing` nodes slowly rather than never.
See [Removal completes on the compactor’s schedule](#removal-completes-on-the-compactors-schedule)for how to measure the remaining gap.

Check whether the compactor is making progress at all before assuming the
removal is stuck:

* Confirm the compactor node is `running` and isn’t restarting or being
  OOM-killed.
* Watch the compaction floors in `system.pt_compaction_nodes` advance between
  queries.

Don’t scale a replacement node into a removing node’s ID while you wait—the
registration is refused until removal finishes.

### Nodes multiply in the catalog after each restart

Each restart registered a new node ID.
Check that your deployment assigns a stable node ID—see[Kubernetes and Helm](#kubernetes-and-helm).
Stop and remove the stale entries after confirming which nodes are current.

### Writes fail during a rolling upgrade

This is usually a catalog version constraint rather than a lifecycle problem.
See [Troubleshooting cluster upgrades](/influxdb3/enterprise/admin/upgrade/#troubleshooting-cluster-upgrades).

#### Related

* [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/)
* [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/)
* [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/)
* [Deploy InfluxDB 3 Enterprise on Kubernetes](/influxdb3/enterprise/install/kubernetes/)
* [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)
* [influxdb3 remove node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)
* [influxdb3 show nodes](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/)
* [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/)

[clustering](/influxdb3/enterprise/tags/clustering/)[nodes](/influxdb3/enterprise/tags/nodes/)[lifecycle](/influxdb3/enterprise/tags/lifecycle/)[kubernetes](/influxdb3/enterprise/tags/kubernetes/)[wal](/influxdb3/enterprise/tags/wal/)
| State | Description |
| --- | --- |
| State | Description |
| running | The node started and registered itself in the catalog |
| stopping | A graceful stop was requested, but the node hasn’t acknowledged it yet |
| stopped | The node acknowledged its final snapshot and completed shutdown |
| removing | The node is marked for permanent removal from the cluster |

| Condition | Message | Resolution |
| --- | --- | --- |
| Condition | Message | Resolution |
| Node isn’t  stopped | node 'NODE_ID' is not fully stopped (current state: STATE) | Stop the node gracefully  and wait for  stopped |
| Node runs in  compact  mode | node 'NODE_ID' has compact mode and cannot be removed | Compactor nodes can’t be removed—see  Remove a compactor node |
| Node belongs to a query group | cannot remove node 'NODE_ID' because it is a member of query group ... | Remove the node from the query group first |
| Unsnapshotted WAL remains | node 'NODE_ID' has unsnapshotted WAL (wal file N, snapshotted through M) | Restart the node, stop it gracefully, then remove it—see  Recover a crashed node |

| Current state | Can register again? |
| --- | --- |
| Current state | Can register again? |
| stopped | Yes—any instance can take over the node ID |
| running  or  stopping | Only the same instance ID (an idempotent retry or a restart) |
| removing | No— removing  is terminal for the node ID |

| Goal | Operation |
| --- | --- |
| Goal | Operation |
| Rolling upgrade, config change, pod reschedule, host reboot | Restart the process with the  same  node ID—don’t remove the node |
| Permanently scale down or decommission hardware | stop node , verify  stopped , then  remove node |
| Replace a failed host, keeping its data | Start a node with the same node ID and object store configuration |