Configure a standalone process node

A standalone process node is a node that runs only in process mode. It runs Processing Engine plugins. It doesn’t store writes or answer data queries; the other nodes in the cluster do that work.

In a cluster, we recommend running the Processing Engine on a standalone process node. This keeps plugin CPU and memory use off your ingest and query nodes. You can also restart the process node, or limit its resources, without affecting the other nodes.

This page covers the standalone process node only. For an overview of process node architectures, see Configure a process node.

• How a process node runs plugins
• Plan the cluster
• Start the nodes
• Verify the cluster
• Add plugins and packages
• Create triggers
• Send each request to the right node
• Manage triggers
• Stop and restart the process node
• Troubleshoot

How a process node runs plugins

The process node starts each trigger type differently:

• WAL triggers (table: or all_tables): The process node reads each ingest node’s WAL files from object storage and passes new rows to the plugin’s process_writes() function.
• Schedule triggers (every: or cron:): The process node calls the plugin’s process_scheduled_call() function on the schedule.
• Request triggers (request:): The process node serves /api/v3/engine/<path> and calls the plugin’s process_request() function for each request.

For every trigger type, the plugin reaches data through other nodes over the internode protocol:

• influxdb3_local.query() runs the query on a query node.
• influxdb3_local.write_sync() and the other write methods send data to an ingest node.
• influxdb3_local.info(), warn(), and error() messages are written to system.processing_engine_logs through an ingest node.

Plan the cluster

A cluster with a process node needs at least two nodes: the process node and one or more nodes that run ingest and query. All nodes share one object store, one --cluster-id, and one license.

Start every ingest node and every query node with --internode-bind-addr:

• Ingest nodes: Receive plugin writes and plugin log messages.
• Query nodes: Run plugin queries.

The process node doesn’t need --internode-bind-addr.

The smallest cluster is one combined node and the process node:

To scale ingest, query, and compaction separately, run each mode on its own node:

Secure the process node

A process node isn’t a security boundary: it needs the same credentials as the other nodes.

• Object storage credentials: The process node reads the catalog and saves its trigger state in object storage. For WAL triggers, it also reads each ingest node’s WAL files from object storage.
• License: The process node needs the cluster’s license file.
• Internode credentials: Plugin queries, writes, and log messages use the internode protocol. If you set --internode-jwt-primary or the --internode-tls-* options on the cluster, set them on the process node too. Without these options, internode traffic is unencrypted.

Start the nodes

Start each node with influxdb3 serve. The following examples start the four-node cluster from Plan the cluster, with each node on its own host. To keep the commands short, they use a file object store in a directory that every host can reach. In production, use an object store such as Amazon S3. See Connect to your object store.

# Ingest node
influxdb3 serve \
  --cluster-id 
CLUSTER_ID
 \
  --node-id ingest0 \
  --mode ingest \
  --object-store file \
  --data-dir 
DATA_DIR
 \
  --internode-bind-addr 0.0.0.0:8083 \
  --conn-info 
INGEST_HOST
:8083

# Query node
influxdb3 serve \
  --cluster-id 
CLUSTER_ID
 \
  --node-id query0 \
  --mode query \
  --object-store file \
  --data-dir 
DATA_DIR
 \
  --internode-bind-addr 0.0.0.0:8083 \
  --conn-info 
QUERY_HOST
:8083

# Compact node
influxdb3 serve \
  --cluster-id 
CLUSTER_ID
 \
  --node-id compact0 \
  --mode compact \
  --object-store file \
  --data-dir 
DATA_DIR

# Process node
influxdb3 serve \
  --cluster-id 
CLUSTER_ID
 \
  --node-id process0 \
  --mode process \
  --object-store file \
  --data-dir 
DATA_DIR
 \
  --plugin-dir 
PLUGIN_DIR

Replace the following:

• CLUSTER_ID: the same cluster ID on every node
• DATA_DIR: the shared data directory
• PLUGIN_DIR: the directory on the process node that holds your plugin files
• INGEST_HOST, QUERY_HOST: the host name or IP address that the process node uses to reach the ingest or query node

--conn-info sets the address that other nodes use to reach the node’s internode port. Without it, a node bound to a wildcard address such as 0.0.0.0 advertises <hostname>:<port>. If the process node can’t resolve that host name, plugin queries, plugin writes, and plugin log messages fail.

On first start, the process node creates a Python virtual environment at PLUGIN_DIR/.venv from the Python bundled with InfluxDB 3 Enterprise.

Verify the cluster

Query system.nodes on a query node:

influxdb3 query \
  --host http://
QUERY_HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --database _internal \
  "SELECT node_id, mode, state, conn_info FROM system.nodes"

Check the following:

• process0 has mode [process] and state running.
• Each ingest and query node has a conn_info that the process node can reach.

Add plugins and packages

Put plugin files in the process node’s plugin directory. For the plugin API, see the Python API reference.

To install Python packages, run influxdb3 install package against the process node:

influxdb3 install package \
  --host http://
PROCESS_HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  
PACKAGE_NAME

Packages are installed only on the node that receives the request.

Create triggers

1. Create the database on any node.
2. Create the trigger on the process node. Set --node-spec to the process node’s ID. Other nodes reject the request with 405 Current node mode does not use the processing engine.

influxdb3 create trigger \
  --host http://
PROCESS_HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --database 
DATABASE_NAME
 \
  --path 
PLUGIN_FILE
 \
  --trigger-spec "table:
TABLE_NAME
" \
  --node-spec "nodes:process0" \
  
TRIGGER_NAME

Always set --node-spec. The default, all, makes every node that runs the Processing Engine own the trigger. You can’t change a trigger’s --node-spec after you create it.

If you use --upload to send a local plugin file, the file is stored on the process node.

Send each request to the right node

A request sent to the wrong node returns an error. See HTTP errors.

Manage triggers

Enable, disable, and delete triggers from any node. The process node applies the change within a few seconds.

influxdb3 disable trigger \
  --host http://
HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --database 
DATABASE_NAME
 \
  
TRIGGER_NAME

influxdb3 enable trigger \
  --host http://
HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --database 
DATABASE_NAME
 \
  
TRIGGER_NAME

influxdb3 delete trigger \
  --host http://
HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --database 
DATABASE_NAME
 \
  --force \
  
TRIGGER_NAME

To list triggers, query system.processing_engine_triggers on a query node:

influxdb3 show system \
  --host http://
QUERY_HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --database 
DATABASE_NAME
 \
  table processing_engine_triggers

Stop and restart the process node

Stop the process node with influxdb3 stop node from any node:

influxdb3 stop node \
  --host http://
HOST
:8181 \
  --token 
AUTH_TOKEN
 \
  --node-id process0

To restart it, run influxdb3 serve with the same options.

The process node saves its trigger state to <cluster-id>/<node-id>/processing_engine/restart-state.postcard in object storage. It saves the state every --processing-engine-restart-state-snapshot-interval (default 1s) and at shutdown. After a restart, the process node does the following:

• WAL triggers: Resumes reading each ingest node’s WAL where it stopped, so it processes writes that arrived while it was down.
• Schedule triggers: Runs each scheduled call that came due while it was down.
• Request triggers: Doesn’t replay requests. During a graceful shutdown, in-progress requests return 503 with {"error": "server is shutting down"}. After a forced stop, the connection drops. Clients can retry either way.
• Retries: Keeps pending retries.

A trigger that you disable or delete while the process node is down stays disabled or deleted.

Retry and queue limits

• --trigger-retry-max-attempts (default 5): Sets how many times a failed run is attempted for triggers with --error-behavior retry.
• Each trigger queues up to 60 pending runs. When a request trigger’s queue is full, /api/v3/engine/<path> returns 503 with {"error": "trigger queue is full"}.

Troubleshoot

• Startup errors
• Plugin query and write errors
• HTTP errors
• Trigger problems

Startup errors

The process node doesn’t start

--mode process without --plugin-dir fails with:

starting a node with mode 'process' requires a configured plugin dir for the python environment

Add --plugin-dir.

A node runs the Processing Engine unexpectedly

--plugin-dir or INFLUXDB3_PLUGIN_DIR adds process mode to a node, whatever its --mode. Remove the option or variable from every node except the process node. For Docker, see Start the nodes.

The process node doesn’t start after a crash

If restart-state.postcard is corrupt, the process node doesn’t start. Move the file out of the way and start the node. The node starts with no saved trigger state: queued runs, pending retries, and its position in each WAL are lost.

Plugin query and write errors

influxdb3_local.query(), write_sync(), and write_sync_to_db() raise these errors in the plugin. write() and write_to_db() don’t: they queue rows until the plugin returns, so a failed write loses the rows and the error appears only in the process node’s log. We recommend write_sync() or write_sync_to_db() so that your plugin can detect and handle failed writes.

Cannot query: no remote query client found

No running query node has an internode address. Restart a query node with --internode-bind-addr. The process node detects it within a few seconds; you don’t need to restart the process node.

Cannot write: no remote write client found

No running ingest node has an internode address. Until you fix it, plugin writes and plugin log messages fail. Failed log messages don’t raise an error in the plugin; look for this line in the process node’s log:

failed to persist processing engine log error=Cannot log: Cannot write: no remote write client found

Restart an ingest node with --internode-bind-addr. The process node detects it within a few seconds.

dns error, then circuit-open

A plugin query or write_sync() call fails with dns error. After a few failures, it fails with all remote query clients are circuit-open or all remote write clients are circuit-open. Plugin log messages are lost with the same error.

An ingest or query node advertises a conn_info that the process node can’t resolve, usually the host name from a 0.0.0.0 bind. Check conn_info in system.nodes, then restart that node with --conn-info set to an address the process node can reach. The process node detects the new address without a restart.

ModuleNotFoundError for a package you installed

The package isn’t installed on the process node. Run influxdb3 install package against the process node.

HTTP errors

Nodes load catalog changes every --catalog-sync-interval (default 1s).

Trigger problems

A trigger disables itself

With --error-behavior disable, the first failed run disables the trigger. The process node logs trigger execution failed; disabling trigger. Fix the plugin, then enable the trigger.

A trigger stops running after you replace the process node

Only the node in a trigger’s --node-spec owns the trigger. If that node no longer runs process mode, the trigger never runs, but system.processing_engine_triggers still shows it as enabled. Delete the trigger and create it again with the new node ID in --node-spec.

A plugin error doesn’t appear in the logs table

Python tracebacks appear only in the process node’s log, in lines that contain trigger execution failed. They don’t appear in system.processing_engine_logs.

Rows in system.processing_engine_logs can appear up to 50 seconds after the run. The node_id column shows the node that ran the plugin.