# InfluxDB 3 Enterprise — full Markdown corpus > Generated 2026-11-17T23:22:51.679Z from https://docs.influxdata.com/influxdb3/enterprise/ > See https://docs.influxdata.com/llms.txt for the cross-product table of contents. > Publisher: InfluxData (https://www.influxdata.com) > sameAs: https://github.com/influxdata, https://hub.docker.com/u/influxdata, https://www.crunchbase.com/organization/influxdb, https://www.linkedin.com/company/influxdb, https://twitter.com/influxdb, https://www.youtube.com/@InfluxData, https://www.facebook.com/influxdb/ --- # InfluxDB 3 Enterprise documentation Source: https://docs.influxdata.com/influxdb3/enterprise/ > [!Tip] > Comparing InfluxDB 3 products or planning a migration from InfluxDB 1 or 2? > See [Which InfluxDB 3 should I use?](/influxdb3/which-influxdb-3/) for a > full decision guide. InfluxDB 3 Enterprise is a database built to collect, process, transform, and store event and time series data, and is ideal for use cases that require real-time ingest and fast query response times to build user interfaces, monitoring, and automation solutions. Common use cases include: * Monitoring sensor data * Server monitoring * Application performance monitoring * Network monitoring * Financial market and trading analytics * Behavioral analytics InfluxDB is optimized for scenarios where near real-time data monitoring is essential and queries need to return quickly to support user experiences such as dashboards and interactive user interfaces. InfluxDB 3 Enterprise is built on InfluxDB 3 Core, the InfluxDB 3 open source release. Core’s feature highlights include: * Diskless architecture with object storage support (or local disk with no dependencies) * Fast query response times (under 10ms for last-value queries, or 30ms for distinct metadata) * Embedded Python VM for plugins and triggers * Parquet file persistence * Compatibility with InfluxDB 1.x and 2.x write APIs The Enterprise version adds the following features to Core: * Historical query capability and single series indexing * High availability * Read replicas * Enhanced security (coming soon) * Row-level delete support (coming soon) * Integrated admin UI (coming soon) [Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/) --- # Administer InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/admin/ The following articles provide information about managing your InfluxDB 3 Enterprise resources: > [!Note] > #### Replicating data from the edge? > > [EDR (Edge Data Replication) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/)replicates time series data from InfluxDB 3 Enterprise to another > Enterprise instance, to InfluxDB 3 Cloud, or to AWS Timestream for > InfluxDB 3–over connections that may be intermittent or > bandwidth-constrained. See[Get started with EDR](/influxdb3/enterprise/edr/get-started/). ### [Identify InfluxDB 3 Enterprise version](/influxdb3/enterprise/admin/identify-version/) Learn how to identify your InfluxDB 3 Enterprise version using command-line tools, HTTP endpoints, and other methods. ### [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) Learn how to configure specialized nodes in your InfluxDB 3 Enterprise cluster for ingest, query, compaction, and processing workloads with optimal thread allocation. ### [Manage your InfluxDB 3 Enterprise license](/influxdb3/enterprise/admin/license/) InfluxDB 3 Enterprise licenses authorize the use of the InfluxDB 3 Enterprise software. Learn how licenses work, how to activate and renew licenses, and more. ### [Configure a standalone process node](/influxdb3/enterprise/admin/processing-engine/) Configure a standalone process node, a node that runs only the Processing Engine, in an InfluxDB 3 Enterprise cluster: plan the cluster, start the nodes, create triggers, send each request to the right node, and troubleshoot common errors. ### [Manage tables](/influxdb3/enterprise/admin/tables/) Tables in InfluxDB 3 Enterprise are synonymous with measurements and contain time series data. Each table has a defined schema with tag and field columns. ### [Manage databases](/influxdb3/enterprise/admin/databases/) An InfluxDB 3 Enterprise database is a named location where time series data is stored. Each database can contain multiple tables. ### [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) Understand how an InfluxDB 3 Enterprise node moves through the `running`, `stopping`, `stopped`, and `removing` states—how to restart, stop, and permanently remove nodes, and how to keep Helm, Kubernetes, and Ansible deployments on the graceful shutdown path. ### [Manage tokens](/influxdb3/enterprise/admin/tokens/) Manage tokens to authenticate and authorize access to server actions, resources, and data in an InfluxDB 3 Enterprise instance. ### [Manage the Last Value Cache](/influxdb3/enterprise/admin/last-value-cache/) The InfluxDB 3 Enterprise Last Value Cache (LVC) lets you cache the most recent values for specific fields in a table, improving the performance of queries that return the most recent value of a field for specific time series or the last N values of a field. ### [Delete data](/influxdb3/enterprise/admin/delete-data/) Delete rows from InfluxDB 3 Enterprise by time range and tag predicate using the influxdb3 delete rows command. Deletes are asynchronous and applied by the compactor. ### [Manage the Distinct Value Cache](/influxdb3/enterprise/admin/distinct-value-cache/) The InfluxDB 3 Enterprise Distinct Value Cache (DVC) lets you cache distinct values of one or more columns in a table, improving the performance of queries that return distinct tag and field values. ### [Import data](/influxdb3/enterprise/admin/import-data/) Bulk import your existing Parquet files into a database and table in InfluxDB 3 Enterprise using the influxdb3 import upload or influxdb3 import from-object-store command, and map columns to InfluxDB types. Imported data is stored in your object storage. ### [Manage file indexes](/influxdb3/enterprise/admin/file-index/) Customize the indexing strategy of a database or table in InfluxDB 3 Enterprise to optimize the performance of single-series queries. ### [Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/) Run compaction jobs across every compact node in a cluster instead of only the node that holds the compactor lease, using `influxdb3 serve --compactor-dispatch-target`. ### [Gen1 file cleanup](/influxdb3/enterprise/admin/gen1-file-cleanup/) Delete Gen1 Parquet files that compaction has already merged into later generations in InfluxDB 3 Enterprise clusters that use the legacy Parquet storage engine. ### [Orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/) Learn how InfluxDB 3 Enterprise finds and deletes compacted files that nothing references, when cleanup passes run, and how to inspect cleanup activity with the influxdb3 debug sweep and influxdb3 debug verify-references commands. ### [Configure object storage](/influxdb3/enterprise/admin/object-storage/) Configure InfluxDB 3 Enterprise to connect to and use different object storage providers. ### [Query system data](/influxdb3/enterprise/admin/query-system-data/) Query system tables to see data related to the server, queries, and tables in an InfluxDB 3 Enterprise instance. Use the HTTP SQL query API to retrieve information about your database server and table schemas. ### [Back up and restore data](/influxdb3/enterprise/admin/backup-restore/) Back up and restore your InfluxDB 3 Enterprise cluster by copying object storage files in the recommended order for each node type. ### [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) Safely recover an InfluxDB 3 Enterprise node that crashed or was stopped without a graceful `stop node`: restart it with the same node ID, let it drain its write-ahead log (WAL), stop it gracefully, and then remove it. ### [Performance tuning](/influxdb3/enterprise/admin/performance-tuning/) Optimize InfluxDB 3 Enterprise performance by tuning thread allocation, memory settings, and other configuration options for your specific workload. ### [Security](/influxdb3/enterprise/admin/security/) Secure InfluxDB 3 Enterprise: authenticate clients with tokens, enable multi-user authentication and role-based access control (RBAC), and harden the host. ### [Upgrade from InfluxDB 3 Core](/influxdb3/enterprise/admin/upgrade-from-core/) Upgrade from InfluxDB 3 Core to InfluxDB 3 Enterprise. Your existing data and plugins are compatible–no data migration is required. ### [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/) Learn how to upgrade your InfluxDB 3 Enterprise cluster across multiple versions, including rolling upgrades, catalog version constraints, and recommended upgrade order for different node modes. ### [Use the InfluxDB 3 MCP server](/influxdb3/enterprise/admin/mcp-server/) Use the **InfluxDB Model Context Protocol (MCP) server** to interact with and manage InfluxDB 3 Enterprise using natural language with LLM agents to query and analyze data, manage databases and more. Query InfluxDB 3 Enterprise documentation from your IDE using the InfluxDB documentation Model Context Protocol (MCP) server. ### [Capture workload data](/influxdb3/enterprise/admin/load-capture/) Use load capture to record an anonymized profile of write and query traffic on an InfluxDB 3 Enterprise query node, inspect it, and share it with InfluxData to troubleshoot performance. ### [Use the integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/) Enable the Explorer UI embedded in InfluxDB 3 Enterprise: quick start without authentication, sign in with multi-user authentication, configure SSO, and understand sessions and default connections. --- # Back up and restore data Source: https://docs.influxdata.com/influxdb3/enterprise/admin/backup-restore/ InfluxDB 3 Enterprise persists all data and metadata to object storage. How you back up and restore that data depends on your storage engine: * **Enterprise on the upgraded storage engine** (the default for new clusters, or after running the storage engine upgrade with`--upgrade-pacha-tree`): use the built-in [`influxdb3` backup and restore commands](#back-up-and-restore-with-the-influxdb3-cli). This is the recommended path on the upgraded engine. * **Enterprise on the Parquet engine** (clusters that started on 3.10 or earlier that have not run the storage engine upgrade): use the[manual object-storage procedure](#manual-backup-process) to copy object storage files in a specific order. ## Supported object storage InfluxDB 3 supports the following object storage backends for data persistence: * **File system** (local directory) * **AWS S3** and S3-compatible storage ([MinIO](/influxdb3/enterprise/object-storage/minio/)) * **Azure Blob Storage** * **Google Cloud Storage** > [!Note] > Backup and restore procedures don’t apply to memory-based [object stores](/influxdb3/enterprise/reference/config-options/#object-store). ## File structure | Location | Description | |-----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **Cluster files** | | | `/_catalog_checkpoint` | Catalog state checkpoint file | | `/catalog/` | Catalog log files tracking catalog state changes | | `/commercial_license` | Commercial [license](/influxdb3/enterprise/admin/license/) file (if applicable) | | `/trial_or_home_license` | Trial or home [license](/influxdb3/enterprise/admin/license/) file (if applicable) | | `/enterprise` | Enterprise configuration file | | **Node files** | | | `/wal/` | [Write-ahead log files](/influxdb3/enterprise/reference/internals/durability/#write-ahead-log-wal-persistence) containing written data | | `/snapshots/` | Snapshot files | | `/dbs////` |[Parquet files](/influxdb3/enterprise/reference/internals/durability/#parquet-storage) organized by [database](/influxdb3/enterprise/admin/databases/), [table](/influxdb3/enterprise/admin/tables/), and time| |`/table-snapshots//
/`| Table snapshot files (regenerated on restart, optional for backup) | | **Compactor node additional files** | | | `/cs` | Compaction summary files | | `/cd` | Compaction detail files | | `/c` | Generation detail and [Parquet files](/influxdb3/enterprise/reference/internals/durability/#parquet-storage) | ## Back up and restore with the influxdb3 CLI On the upgraded storage engine, InfluxDB 3 Enterprise provides built-in`influxdb3` backup and restore commands. Before you use them, ensure the following: * The cluster uses the upgraded storage engine (the default for new clusters; older clusters must first run the[storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree)). * You run the commands against any node that acts as the **compactor**, including `mode=all`. * You authenticate with an **admin token** or as an **admin user**. > [!Note] > #### Availability and node-role requirements > > * The backup and restore commands and endpoints are available **only when the > storage engine upgrade is enabled**. On the default Parquet engine, the > backup endpoints return `404`. > * Backup and restore run on a node running **compaction**. Query-only nodes return`503` and ingest-only nodes return `404`. The backup and restore subcommands map to the`/api/v3/enterprise/backup[/{name}]` and `/api/v3/enterprise/restore[/{id}]` HTTP API endpoints. For complete command syntax and flags, see the[`influxdb3` CLI reference](/influxdb3/enterprise/reference/cli/influxdb3/). ### Create a backup Use `influxdb3 create backup` to create a full backup: ```bash influxdb3 create backup --name base ``` If you omit `--name`, InfluxDB 3 Enterprise generates one from a UTC timestamp. > [!Note] > * `create backup` **refuses to overwrite an existing backup name** unless > you pass `--force`. > * **A backup only includes data already persisted to object storage**–your > most recent writes may still be buffered and not yet captured. See[What a backup includes](#what-a-backup-includes). #### What a backup includes A backup captures data already persisted to object storage. Writes still buffered in memory and not yet flushed to the[write-ahead log (WAL)](/influxdb3/enterprise/reference/internals/durability/#write-ahead-log-wal-persistence)aren’t included. To minimize the window of unpersisted writes before a backup, use the default[`no_sync=false`](/influxdb3/enterprise/write-data/http-api/v3-write-lp/#use-no_sync-for-immediate-write-responses)write behavior, which acknowledges writes only after WAL persistence completes. For the full ingest and persistence path, see[InfluxDB 3 internals: durability](/influxdb3/enterprise/reference/internals/durability/). ### Create an incremental backup An incremental backup captures only the data that changed since its parent backup, which saves time and storage space compared to a full backup. Each incremental backup names the backup it chains from with `--parent`. `create backup` returns as soon as the backup starts, not when it finishes.`influxdb3 status backup` reports `in_progress`, `completed`, or `failed`. Check that a backup’s status is `completed` before creating a child incremental backup that names it as `--parent`: ```bash # Full backup first influxdb3 create backup --name base # Wait for it to complete before using it as a parent influxdb3 status backup --name base # Then create an incremental, naming its parent influxdb3 create backup --name inc-1 --incremental --parent base # Wait for inc-1 to complete before chaining the next incremental off it influxdb3 status backup --name inc-1 influxdb3 create backup --name inc-2 --incremental --parent inc-1 ``` Restoring an incremental backup walks the parent chain back to the full backup and produces a complete restore–you don’t need to restore each link in the chain separately. Deleting an incremental backup also deletes every incremental backup that depends on it: ```bash influxdb3 delete backup --name inc-1 --incremental ``` > [!Caution] > Deleting an incremental backup with `--incremental` deletes it and all of > its child incremental backups. This can remove multiple backups at once. ### Inspect and manage backups | Command | Description | |-------------------------|----------------------------| |`influxdb3 status backup`|Show the status of a backup | |`influxdb3 show backups` | List existing backups | |`influxdb3 delete backup`| Delete a backup | |`influxdb3 cancel backup`|Cancel an in-progress backup| ### Restore from a backup Use `influxdb3 create restore` to start a restore, and the related commands to monitor or cancel it: ```bash influxdb3 create restore --backup base ``` To restore from an incremental backup, pass its name–the restore walks the chain of parent backups automatically: ```bash influxdb3 create restore --backup inc-2 ``` | Command | Description | |--------------------------|-----------------------------| |`influxdb3 create restore`|Start a restore from a backup| |`influxdb3 status restore`|Show the status of a restore | |`influxdb3 show restores` | List restores | |`influxdb3 cancel restore`|Cancel an in-progress restore| A restore is **asynchronous** and runs **in place on the live cluster**–no restart required. **Only one restore can run at a time across the cluster**; concurrent restore attempts return `409`. Restore is a **point-in-time rollback**, not an additive merge: it restores the catalog, copies the backup’s files, and writes a new checkpoint above the existing checkpoint sequence. Rolling back also truncates the WAL down to the backup’s watermark, so a subsequent node restart doesn’t replay and resurrect the rolled-back writes. Restore is **not a full pre-clean** of post-backup object storage output. It removes newer transient and pre-compaction artifacts–PachaTree snapshots, `gen0` files, PachaTree WAL files, and newer compactor checkpoints. But already-**compacted** post-backup files (compactor run-set files and their indexes) stay in object storage, unreferenced by the restored catalog, pending later garbage collection rather than immediate deletion. A restore also discards any persisted compactor delete queue newer than the restored checkpoint, so stale cleanup work can’t delete files the restored state still references. > [!Note] > #### Scripting a restore > > `influxdb3 create restore` and `influxdb3 status restore` print > human-readable text only–neither supports `--format json`. To script a > restore, call the HTTP API directly: `POST /api/v3/enterprise/restore`returns `202` with a JSON body containing `restore_id`, which you poll with`GET /api/v3/enterprise/restore/{id}`. > [!Warning] > #### Row deletes may persist across restores > > Backup doesn’t currently pick up row-delete state files in object storage, > so row deletes may persist across a restore. ### Disaster recovery To recover into a new object store, copy the backup directory`{cluster_id}/backups/{name}/` into an empty object store that uses the **same cluster ID and node ID** as the original deployment, then restart the node(s). ## Manual backup process Use this manual object-storage procedure to back up InfluxDB 3 Enterprise running on the Parquet engine—clusters that started on 3.10 or earlier that have not run the storage engine upgrade (`--upgrade-pacha-tree`) . It copies object storage files in a specific order to ensure consistency. > [!Important] > Copy files in the recommended order to reduce risk of creating inconsistent backups. Perform backups during downtime or minimal load periods when possible. **Recommended backup order:** 1. Compactor node directories (cs, cd, c) 2. All nodes’ snapshots, dbs, wal directories 3. Cluster catalog and checkpoint 4. License files [S3](#s3-2)[File system](#file-system-2) ``` #!/bin/bash CLUSTER_ID="CLUSTER_ID" COMPACTOR_NODE="COMPACTOR_NODE" DATA_NODES=("NODE1" "NODE2" "NODE3") SOURCE_BUCKET="SOURCE_BUCKET" BACKUP_BUCKET="BACKUP_BUCKET" BACKUP_PREFIX="backup-$(date +%Y%m%d-%H%M%S)" # 1. Backup compactor node first echo "Backing up compactor node..." aws s3 sync s3://${SOURCE_BUCKET}/${COMPACTOR_NODE}/cs \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${COMPACTOR_NODE}/cs/ aws s3 sync s3://${SOURCE_BUCKET}/${COMPACTOR_NODE}/cd \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${COMPACTOR_NODE}/cd/ aws s3 sync s3://${SOURCE_BUCKET}/${COMPACTOR_NODE}/c \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${COMPACTOR_NODE}/c/ # 2. Backup all nodes (including compactor) ALL_NODES=("${DATA_NODES[@]}" "$COMPACTOR_NODE") for NODE_ID in "${ALL_NODES[@]}"; do echo "Backing up node: ${NODE_ID}" aws s3 sync s3://${SOURCE_BUCKET}/${NODE_ID}/snapshots \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${NODE_ID}/snapshots/ aws s3 sync s3://${SOURCE_BUCKET}/${NODE_ID}/dbs \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${NODE_ID}/dbs/ aws s3 sync s3://${SOURCE_BUCKET}/${NODE_ID}/wal \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${NODE_ID}/wal/ done # 3. Backup cluster catalog echo "Backing up cluster catalog..." aws s3 sync s3://${SOURCE_BUCKET}/${CLUSTER_ID}/catalog \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/catalog/ aws s3 cp s3://${SOURCE_BUCKET}/${CLUSTER_ID}/_catalog_checkpoint \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/ aws s3 cp s3://${SOURCE_BUCKET}/${CLUSTER_ID}/enterprise \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/ # 4. Backup license files (may not exist) aws s3 cp s3://${SOURCE_BUCKET}/${CLUSTER_ID}/commercial_license \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/ 2>/dev/null || true aws s3 cp s3://${SOURCE_BUCKET}/${CLUSTER_ID}/trial_or_home_license \ s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/ 2>/dev/null || true echo "Backup completed to s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}" ``` Replace the following: * `CLUSTER_ID`: your [cluster ID](/influxdb3/enterprise/reference/config-options/#cluster-id) * `COMPACTOR_NODE`: your [compactor](/influxdb3/enterprise/get-started/multi-server/#high-availability-with-a-dedicated-compactor) node ID * `NODE1`, `NODE2`, `NODE3`: your data [node IDs](/influxdb3/enterprise/reference/config-options/#node-id) * `SOURCE_BUCKET`: your InfluxDB data bucket * `BACKUP_BUCKET`: your backup destination bucket ```bash #!/bin/bash CLUSTER_ID="CLUSTER_ID" COMPACTOR_NODE="COMPACTOR_NODE" DATA_NODES=("NODE1" "NODE2" "NODE3") DATA_DIR="/path/to/data" BACKUP_DIR="/backup/$(date +%Y%m%d-%H%M%S)" mkdir -p "$BACKUP_DIR" # 1. Backup compactor node first echo "Backing up compactor node..." cp -r $DATA_DIR/${COMPACTOR_NODE}/cs "$BACKUP_DIR/${COMPACTOR_NODE}/" cp -r $DATA_DIR/${COMPACTOR_NODE}/cd "$BACKUP_DIR/${COMPACTOR_NODE}/" cp -r $DATA_DIR/${COMPACTOR_NODE}/c "$BACKUP_DIR/${COMPACTOR_NODE}/" # 2. Backup all nodes ALL_NODES=("${DATA_NODES[@]}" "$COMPACTOR_NODE") for NODE_ID in "${ALL_NODES[@]}"; do echo "Backing up node: ${NODE_ID}" mkdir -p "$BACKUP_DIR/${NODE_ID}" cp -r $DATA_DIR/${NODE_ID}/snapshots "$BACKUP_DIR/${NODE_ID}/" cp -r $DATA_DIR/${NODE_ID}/dbs "$BACKUP_DIR/${NODE_ID}/" cp -r $DATA_DIR/${NODE_ID}/wal "$BACKUP_DIR/${NODE_ID}/" done # 3. Backup cluster catalog echo "Backing up cluster catalog..." mkdir -p "$BACKUP_DIR/${CLUSTER_ID}" cp -r $DATA_DIR/${CLUSTER_ID}/catalog "$BACKUP_DIR/${CLUSTER_ID}/" cp $DATA_DIR/${CLUSTER_ID}/_catalog_checkpoint "$BACKUP_DIR/${CLUSTER_ID}/" cp $DATA_DIR/${CLUSTER_ID}/enterprise "$BACKUP_DIR/${CLUSTER_ID}/" # 4. Backup license files (if they exist) [ -f "$DATA_DIR/${CLUSTER_ID}/commercial_license" ] && \ cp $DATA_DIR/${CLUSTER_ID}/commercial_license "$BACKUP_DIR/${CLUSTER_ID}/" [ -f "$DATA_DIR/${CLUSTER_ID}/trial_or_home_license" ] && \ cp $DATA_DIR/${CLUSTER_ID}/trial_or_home_license "$BACKUP_DIR/${CLUSTER_ID}/" echo "Backup completed to $BACKUP_DIR" ``` Replace the following: * `CLUSTER_ID`: your [cluster ID](/influxdb3/enterprise/reference/config-options/#cluster-id) * `COMPACTOR_NODE`: your [compactor](/influxdb3/enterprise/get-started/multi-server/#high-availability-with-a-dedicated-compactor) node ID * `NODE1`, `NODE2`, `NODE3`: your data [node IDs](/influxdb3/enterprise/reference/config-options/#node-id) ## Manual restore process Use this manual object-storage procedure to restore InfluxDB 3 Enterprise running on the Parquet engine—clusters that started on 3.10 or earlier that have not run the storage engine upgrade (`--upgrade-pacha-tree`) . > [!Warning] > Restoring overwrites existing data. Always verify you have correct backups before proceeding. **Recommended restore order:** 1. Cluster catalog and checkpoint 2. License files 3. All nodes’ snapshots, dbs, wal directories 4. Compactor node directories (cs, cd, c) #### S3 restore example ```bash #!/bin/bash CLUSTER_ID="CLUSTER_ID" COMPACTOR_NODE="COMPACTOR_NODE" DATA_NODES=("NODE1" "NODE2" "NODE3") BACKUP_BUCKET="BACKUP_BUCKET" BACKUP_PREFIX="backup-BACKUP_DATE" TARGET_BUCKET="TARGET_BUCKET" # 1. Stop all InfluxDB 3 Enterprise nodes # Implementation depends on your orchestration # 2. Restore cluster catalog and license first aws s3 cp s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/_catalog_checkpoint \ s3://${TARGET_BUCKET}/${CLUSTER_ID}/ aws s3 sync s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/catalog \ s3://${TARGET_BUCKET}/${CLUSTER_ID}/catalog/ aws s3 cp s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/enterprise \ s3://${TARGET_BUCKET}/${CLUSTER_ID}/ # Restore license files if they exist aws s3 cp s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/commercial_license \ s3://${TARGET_BUCKET}/${CLUSTER_ID}/ 2>/dev/null || true aws s3 cp s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${CLUSTER_ID}/trial_or_home_license \ s3://${TARGET_BUCKET}/${CLUSTER_ID}/ 2>/dev/null || true # 3. Restore all nodes ALL_NODES=("${DATA_NODES[@]}" "$COMPACTOR_NODE") for NODE_ID in "${ALL_NODES[@]}"; do echo "Restoring node: ${NODE_ID}" aws s3 sync s3://${BACKUP_BUCKET}/${BACKUP_PREFIX}/${NODE_ID}/ \ s3://${TARGET_BUCKET}/${NODE_ID}/ done # 4. Start InfluxDB Enterprise nodes # Start in order: data nodes, compactor ``` Replace the following: * `CLUSTER_ID`: your [cluster ID](/influxdb3/enterprise/reference/config-options/#cluster-id) * `COMPACTOR_NODE`: your [compactor](/influxdb3/enterprise/get-started/multi-server/#high-availability-with-a-dedicated-compactor) node ID * `NODE1`, `NODE2`, `NODE3`: your data [node IDs](/influxdb3/enterprise/reference/config-options/#node-id) * `BACKUP_DATE`: backup timestamp * `BACKUP_BUCKET`: bucket containing backup * `TARGET_BUCKET`: target bucket for restoration #### File system restore example ```bash #!/bin/bash CLUSTER_ID="CLUSTER_ID" COMPACTOR_NODE="COMPACTOR_NODE" DATA_NODES=("NODE1" "NODE2" "NODE3") BACKUP_DIR="/backup/BACKUP_DATE" DATA_DIR="/path/to/data" # 1. Stop all InfluxDB 3 Enterprise nodes # Implementation depends on your deployment method # 2. Optional: Clear existing data ALL_NODES=("${DATA_NODES[@]}" "$COMPACTOR_NODE") for NODE_ID in "${ALL_NODES[@]}"; do rm -rf ${DATA_DIR}/${NODE_ID}/* done rm -rf ${DATA_DIR}/${CLUSTER_ID}/* # 3. Restore cluster catalog and license mkdir -p ${DATA_DIR}/${CLUSTER_ID} cp ${BACKUP_DIR}/${CLUSTER_ID}/_catalog_checkpoint ${DATA_DIR}/${CLUSTER_ID}/ cp -r ${BACKUP_DIR}/${CLUSTER_ID}/catalog ${DATA_DIR}/${CLUSTER_ID}/ cp ${BACKUP_DIR}/${CLUSTER_ID}/enterprise ${DATA_DIR}/${CLUSTER_ID}/ # Restore license files if they exist [ -f "${BACKUP_DIR}/${CLUSTER_ID}/commercial_license" ] && \ cp ${BACKUP_DIR}/${CLUSTER_ID}/commercial_license ${DATA_DIR}/${CLUSTER_ID}/ [ -f "${BACKUP_DIR}/${CLUSTER_ID}/trial_or_home_license" ] && \ cp ${BACKUP_DIR}/${CLUSTER_ID}/trial_or_home_license ${DATA_DIR}/${CLUSTER_ID}/ # 4. Restore all nodes for NODE_ID in "${ALL_NODES[@]}"; do echo "Restoring node: ${NODE_ID}" mkdir -p ${DATA_DIR}/${NODE_ID} cp -r ${BACKUP_DIR}/${NODE_ID}/* ${DATA_DIR}/${NODE_ID}/ done # 5. Set correct permissions # NOTE: Adjust 'influxdb:influxdb' to match your actual deployment user/group configuration. chown -R influxdb:influxdb ${DATA_DIR} # 6. Start InfluxDB Enterprise nodes # Start in order: data nodes, compactor ``` Replace the following: * `CLUSTER_ID`: your [cluster ID](/influxdb3/enterprise/reference/config-options/#cluster-id) * `COMPACTOR_NODE`: your [compactor](/influxdb3/enterprise/get-started/multi-server/#high-availability-with-a-dedicated-compactor) node ID * `NODE1`, `NODE2`, `NODE3`: your data [node IDs](/influxdb3/enterprise/reference/config-options/#node-id) * `BACKUP_DATE`: backup directory timestamp ## Recovery expectations > [!Warning] > Recovery succeeds to a consistent point in time, which is the **latest snapshot included** in the backup. Data written after that snapshot may not be present if its WAL was deleted after the backup. Any Parquet files without a snapshot reference are ignored. ## Object storage features and backup planning ### Versioning If your object storage provider supports it, consider enabling short-term object versioning on the object store backing InfluxDB 3 Enterprise — typically 1–2 days — to protect against errant writes or accidental deletions. With versioning enabled, the object store retains distinct versions of each updated object, which can be used to recover from these incidents. InfluxDB 3 Enterprise writes mostly immutable Parquet files and rarely updates existing objects, so versioning helps most with recovering specific accidentally deleted files. If many files are affected at once, recovering them through individual version restores becomes impractical. Versioning costs grow with object update frequency. InfluxDB 3 Enterprise’s compaction process generates new Parquet files over time, so each compaction pass adds to the retained version count in a versioned bucket. Use the shortest retention window that meets your recovery needs, and treat versioning as a short-window safety net rather than a long-term backup substitute. ### Periodic backups Provider-side features like versioning, soft delete, and point-in-time recovery protect against many failure modes but do **not** replace backups. Periodic backups to a separate object store remain the only predictable, repeatable way to recover from: * Corruption or compromise of the primary object store. * Accidental or malicious deletion of data. * Bad data written into the primary store (a replica of bad data is not a valid recovery point). The [backup process](#manual-backup-process) above is provider-agnostic — you can use any provider-native tool to copy to a separate bucket, container, or account, as long as you preserve the documented directory structure and file ordering. Schedule copies during low-load periods or downtime windows when possible. > [!Note] > Restoring to a different bucket, container, or storage account requires a new license — see [Licensing and disaster recovery](#licensing-and-disaster-recovery) below for binding details and DR planning. ## Licensing and disaster recovery > [!Important] > License files are tied to the complete object store configuration and the cluster ID, not just the bucket or container name. > Changing any component of your object store configuration or restoring to a different cluster invalidates your license. License binding varies by cloud provider: | Provider | License bound to | |------------------------|--------------------------------------------------| | **AWS S3** | Endpoint + bucket name + region + cluster ID | | **Azure Blob Storage** |Storage account name + container name + cluster ID| |**Google Cloud Storage**| GCS base URL + bucket name + cluster ID | | **File storage** | Exact file path + cluster ID | For disaster recovery planning: * **Pre-provision a DR license**: Request a license for your DR storage configuration before an incident occurs. * **Contact support for migrations**: If you need to restore to a different storage account, endpoint, bucket, or cluster, [contact InfluxData support](https://support.influxdata.com) to obtain a new license. ## Containerized deployments When running InfluxDB 3 Enterprise in containers: * **Volume consistency**: Use the same volume mounts for backup and restore operations * **File permissions**: Ensure container user can read restored files (use `chown` if needed) * **Backup access**: Mount a backup directory to copy files from containers to the host * **Node coordination**: Stop and start all Enterprise nodes (querier, ingester, compactor) in the correct order ## Excluded files Files in `/table-snapshots/` are intentionally excluded from backup: * These files are periodically overwritten * They regenerate automatically on server restart * Including them doesn’t harm but increases backup size unnecessarily ## Backup timing * Perform backups during downtime or minimal load periods * Copying files while the database is active may create inconsistent backups * Consider using filesystem or storage snapshots if available * Compression is optional but recommended for long-term storage #### Related * [Manage databases](/influxdb3/enterprise/admin/databases/) * [Manage your InfluxDB 3 Enterprise license](/influxdb3/enterprise/admin/license/) * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) * [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/) * [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/) * [InfluxDB 3 Enterprise Internals](/influxdb3/enterprise/reference/internals/durability/) [backup](/influxdb3/enterprise/tags/backup/)[restore](/influxdb3/enterprise/tags/restore/)[administration](/influxdb3/enterprise/tags/administration/)[cluster](/influxdb3/enterprise/tags/cluster/)[object storage](/influxdb3/enterprise/tags/object-storage/) | Location | Description | | --- | --- | | Location | Description | | Cluster files | | | /_catalog_checkpoint | Catalog state checkpoint file | | /catalog/ | Catalog log files tracking catalog state changes | | /commercial_license | Commercial license file (if applicable) | | /trial_or_home_license | Trial or home license file (if applicable) | | /enterprise | Enterprise configuration file | | Node files | | | /wal/ | Write-ahead log files containing written data | | /snapshots/ | Snapshot files | | /dbs//
// | Parquet files organized by database , table , and time | | /table-snapshots//
/ | Table snapshot files (regenerated on restart, optional for backup) | | Compactor node additional files | | | /cs | Compaction summary files | | /cd | Compaction detail files | | /c | Generation detail and Parquet files | | Command | Description | | --- | --- | | Command | Description | | influxdb3 status backup | Show the status of a backup | | influxdb3 show backups | List existing backups | | influxdb3 delete backup | Delete a backup | | influxdb3 cancel backup | Cancel an in-progress backup | | Command | Description | | --- | --- | | Command | Description | | influxdb3 create restore | Start a restore from a backup | | influxdb3 status restore | Show the status of a restore | | influxdb3 show restores | List restores | | influxdb3 cancel restore | Cancel an in-progress restore | | Provider | License bound to | | --- | --- | | Provider | License bound to | | AWS S3 | Endpoint + bucket name + region + cluster ID | | Azure Blob Storage | Storage account name + container name + cluster ID | | Google Cloud Storage | GCS base URL + bucket name + cluster ID | | File storage | Exact file path + cluster ID | --- # Configure specialized cluster nodes Source: https://docs.influxdata.com/influxdb3/enterprise/admin/clustering/ > [!Note] > Thread allocation on this page applies to the Parquet storage engine. > If your cluster runs the upgraded storage engine (the default for new > clusters in InfluxDB 3 Enterprise 3.11+), see the[storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/)instead. Optimize performance for specific workloads in your InfluxDB 3 Enterprise cluster by configuring specialized nodes in distributed deployments. Assign specific modes and thread allocations to nodes to maximize cluster efficiency. * [Specialize nodes for specific workloads](#specialize-nodes-for-specific-workloads) * [Configure node modes](#configure-node-modes) * [Allocate threads by node type](#allocate-threads-by-node-type) * [Configure ingest nodes](#configure-ingest-nodes) * [Configure query nodes](#configure-query-nodes) * [Configure compactor nodes](#configure-compactor-nodes) * [Configure a process node](#configure-a-process-node) * [Multi-mode configurations](#multi-mode-configurations) * [Cluster architecture examples](#cluster-architecture-examples) * [Scale your cluster](#scale-your-cluster) * [Monitor performance](#monitor-performance) * [Troubleshoot node configurations](#troubleshoot-node-configurations) * [Best practices](#best-practices) * [Migrate to specialized nodes](#migrate-to-specialized-nodes) * [Manage configurations](#manage-configurations) ## Specialize nodes for specific workloads In an InfluxDB 3 Enterprise cluster, you can dedicate nodes to specific tasks: * **Ingest nodes**: Optimized for high-throughput data ingestion * **Query nodes**: Maximized for complex analytical queries * **Compactor nodes**: Dedicated to data compaction and optimization * **Process node**: Runs only the Processing Engine. We recommend a standalone process node, which runs only the Processing Engine. See [Configure a standalone process node](/influxdb3/enterprise/admin/processing-engine/). * **All-in-one nodes**: Balanced for mixed workloads (single-node deployments only) ## Configure node modes Pass the `--mode` parameter when starting the node to specify its capabilities: ```bash # Single mode influxdb3 serve --mode=ingest # Multiple modes influxdb3 serve --mode=ingest,query # All modes (default, for single-node Enterprise only) influxdb3 serve --mode=all ``` Available modes: * `all`: All capabilities enabled (single-node Enterprise deployments only) * `ingest`: Data ingestion and line protocol parsing * `query`: Query execution and data retrieval * `compact`: Background compaction and optimization * `process`: Runs the Processing Engine, which runs plugins for triggers. Requires [`--plugin-dir`](/influxdb3/enterprise/reference/config-options/#plugin-dir). Setting `--plugin-dir` adds `process` mode to a node, whatever its `--mode`. In a cluster, run `process` alone on a dedicated node, see [Configure a process node](#configure-a-process-node). > [!Note] > #### Nodes without query mode refuse data queries (Parquet engine) > > A node that doesn’t run `query` mode returns `405 Method Not Allowed` for data queries. > System table queries still work. > The upgraded storage engine already behaves this way. > [!Warning] > #### Don’t use all mode in a multi-node cluster > > Use `all` mode for **single-node** Enterprise deployments only. > Some cluster features such as replication and catalog refresh aren’t designed to work with `all`-mode nodes. > In a multi-node cluster, use explicit modes (`ingest`, `query`, `compact`, `process`). > With the Parquet engine, assign `compact` to exactly one node. > With the upgraded storage engine, one compact node at a time holds the compactor lease, and with [distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/) the other compact nodes can run compaction jobs. ## Allocate threads by node type > [!Important] > With the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters starting on 3.11+), ingest and compaction run on the > IO thread pool instead of the DataFusion thread pool. Follow the[storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/)instead of the guidance in this section. ### Critical concept: Thread pools Every node has two thread pools that must be properly configured: 1. **IO threads**: Parse line protocol, handle HTTP requests 2. **DataFusion threads**: Execute queries, create data snapshots (convert [WAL data](/influxdb3/enterprise/reference/internals/durability/#write-ahead-log-wal-persistence) to Parquet files), perform compaction > [!Note] > Even specialized nodes need both thread types. Ingest nodes use DataFusion threads > for creating data snapshots that convert [WAL data](/influxdb3/enterprise/reference/internals/durability/#write-ahead-log-wal-persistence) to Parquet files, and query nodes use IO threads for handling requests. ## Configure ingest nodes Ingest nodes handle high-volume data writes and require significant IO thread allocation for line protocol parsing. ### Example medium ingester (32 cores) ```bash influxdb3 \ --num-io-threads=12 \ serve \ --num-cores=32 \ --datafusion-num-threads=20 \ --exec-mem-pool-size=60% \ --mode=ingest \ --node-id=ingester-01 ``` **Configuration rationale:** * **12 IO threads**: Handle multiple concurrent writers (Telegraf agents, applications) * **20 DataFusion threads**: Required for data snapshot operations that convert buffered writes to Parquet files * **60% memory pool**: Balance between write buffers and data snapshot operations ### Monitor ingest performance Key metrics for ingest nodes: ```bash # Monitor IO thread utilization top -H -p $(pgrep influxdb3) | grep io_worker # Check write request counts by endpoint curl -s http://localhost:8181/metrics | grep 'http_requests_total.*write' # Check overall HTTP request metrics curl -s http://localhost:8181/metrics | grep 'http_requests_total' # Monitor WAL size du -sh /path/to/data/wal/ ``` > [!Important] > #### Scale IO threads with concurrent writers > > If you see only 2 CPU cores at 100% on a large ingester, increase`--num-io-threads`. > Each concurrent writer can utilize approximately one IO thread. ## Configure query nodes Query nodes execute complex analytical queries and need maximum DataFusion threads. ### Analytical query node (64 cores) ```bash influxdb3 \ --num-io-threads=4 \ serve \ --num-cores=64 \ --datafusion-num-threads=60 \ --exec-mem-pool-size=90% \ --file-cache-size=8GB \ --mode=query \ --node-id=query-01 \ --cluster-id=prod-cluster ``` **Configuration rationale:** * **4 IO threads**: Minimal, just for HTTP request handling * **60 DataFusion threads**: Maximum parallelism for query execution * **90% memory pool**: Maximize memory for complex aggregations * **8 GB Parquet cache**: Keep frequently accessed data in memory ### Real-time query node (32 cores) ```bash influxdb3 \ --num-io-threads=6 \ serve \ --num-cores=32 \ --datafusion-num-threads=26 \ --exec-mem-pool-size=80% \ --file-cache-size=4GB \ --mode=query \ --node-id=query-02 ``` ### Optimize query settings You can configure `datafusion` properties for additional tuning of query nodes: ```bash influxdb3 serve \ --datafusion-config "datafusion.execution.batch_size:16384,datafusion.execution.target_partitions:60" \ --mode=query ``` ## Configure compactor nodes Compactor nodes optimize stored data through background compaction processes. > [!Warning] > Only **one** compactor node can run per cluster. > Multiple compactors writing compacted data to the same location will cause data corruption. > Any node mode that includes compaction (`compact` or `all`) counts toward this limit. ### Dedicated compactor (32 cores) ```bash influxdb3 \ --num-io-threads=2 \ serve \ --num-cores=32 \ --datafusion-num-threads=30 \ --compaction-gen2-duration=24h \ --compaction-check-interval=5m \ --mode=compact \ --node-id=compactor-01 \ --cluster-id=prod-cluster # Note: --compaction-row-limit option is not yet released in v3.5.0 # Uncomment when available in a future release: # --compaction-row-limit=2000000 \ ``` **Configuration rationale:** * **2 IO threads**: Minimal, compaction is DataFusion-intensive * **30 DataFusion threads**: Maximum threads for sort/merge operations * **24h gen2 duration**: Time-based compaction strategy ### Tune compaction parameters You can adjust compaction strategies to balance performance and resource usage: ```bash # Configure compaction strategy --compaction-multipliers=4,8,16 \ --compaction-max-num-files-per-plan=100 ``` ### Distributed compaction On the upgraded storage engine, you can spread compaction work across every compact node in the cluster instead of running it only on the lease holder. For setup and tuning guidance, see [Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/). ## Configure a process node A *process node* is a node that runs the Processing Engine (`process` mode). It runs plugins for triggers. You can run the Processing Engine in one of two architectures: | Architecture | Process node `--mode` |Recommendation | |---------------------------------------------------------------|---------------------------------------------------------|---------------| |[Standalone process node](#standalone-process-node-recommended)| `process` |**Recommended**| | [Mixed process node](#mixed-process-node-not-recommended) |`process` and one or more of `ingest`, `query`, `compact`|Not recommended| > [!Important] > #### Use a standalone process node > > We recommend a standalone process node for every cluster. > A standalone process node runs only the Processing Engine, so plugins don’t compete with ingest, query, or compaction for CPU and memory. > You can also restart the process node, or limit its resources, without affecting the other nodes. * [Requirements for every process node](#requirements-for-every-process-node) * [Standalone process node (recommended)](#standalone-process-node-recommended) * [Mixed process node (not recommended)](#mixed-process-node-not-recommended) ### Requirements for every process node The following rules apply to every architecture: * **Set `--plugin-dir` on the process node**:`--mode` with `process` and no [`--plugin-dir`](/influxdb3/enterprise/reference/config-options/#plugin-dir) fails at startup. Put your plugin files in that directory. * **Don’t set `--plugin-dir` on other nodes**: Setting `--plugin-dir` or `INFLUXDB3_PLUGIN_DIR` adds `process` mode to a node, whatever its `--mode`. The official Docker image sets `INFLUXDB3_PLUGIN_DIR=/plugins`; on containers that shouldn’t run the Processing Engine, unset it with `-e INFLUXDB3_UNSET_VARS=INFLUXDB3_PLUGIN_DIR`. * **Don’t use `all` mode**: In a cluster, list the process node’s modes explicitly. * **Set `--internode-bind-addr` on the nodes that plugins reach**: A plugin’s `influxdb3_local.query()` call runs on a node with `query` mode. Its `influxdb3_local.write()` calls and log messages go to a node with `ingest` mode. If the process node doesn’t run one of those modes itself, it reaches another node over the internode protocol, and every node with that mode needs `--internode-bind-addr`. If a node binds to a wildcard address such as `0.0.0.0`, set [`--conn-info`](/influxdb3/enterprise/reference/config-options/#conn-info) to an address that the process node can reach. * **Create triggers on the process node and pin them to it**: Set `--node-spec nodes:` on every trigger. * **Send Processing Engine requests to the process node**: Send trigger creation, package installation, and request trigger calls (`/api/v3/engine/`) to the process node. Other nodes return `405 Current node mode does not use the processing engine`. ### Standalone process node (recommended) A standalone process node runs only in `process` mode. It doesn’t store writes or answer data queries.**Use this architecture for every cluster.** * Start the process node with `--mode=process` and `--plugin-dir`. * Start every ingest node and every query node with `--internode-bind-addr`. * Use InfluxDB 3 Enterprise 3.12.0 or later. On earlier versions, WAL triggers on a standalone process node never run. ```bash influxdb3 serve \ --plugin-dir=/path/to/plugins \ --mode=process \ --node-id=process0 \ --cluster-id=prod-cluster ``` For the complete setup, see [Configure a standalone process node](/influxdb3/enterprise/admin/processing-engine/). ### Mixed process node (not recommended) A mixed process node runs `process` mode together with one or more of `ingest`, `query`, or `compact`. > [!Warning] > #### Use a standalone process node instead > > On a mixed process node, plugins share CPU and memory with ingest, query, or compaction work, so a plugin that uses a lot of CPU or memory slows that work. > When you restart the node, you also interrupt its ingest, query, or compaction work. > Use a [standalone process node](#standalone-process-node-recommended) unless you can’t add a node to the cluster. If you run a mixed process node, follow the [requirements for every process node](#requirements-for-every-process-node), and: 1. Add `process` to the node’s `--mode` list and set `--plugin-dir`. 2. If the node doesn’t run `query`, restart every query node with `--internode-bind-addr`. 3. If the node doesn’t run `ingest`, restart every ingest node with `--internode-bind-addr`. 4. Create each trigger on the mixed process node, and set `--node-spec` to its node ID. ## Multi-mode configurations Some deployments benefit from nodes handling multiple responsibilities. ### Ingest + Query node (48 cores) ```bash influxdb3 \ --num-io-threads=12 \ serve \ --num-cores=48 \ --datafusion-num-threads=36 \ --exec-mem-pool-size=75% \ --mode=ingest,query \ --node-id=hybrid-01 ``` ### Query + Compact node (32 cores) ```bash influxdb3 \ --num-io-threads=4 \ serve \ --num-cores=32 \ --datafusion-num-threads=28 \ --mode=query,compact \ --node-id=qc-01 ``` ## Cluster architecture examples ### Small cluster (3 nodes) > [!Note] > Only one node per cluster can run compaction. > In this example, Node 1 handles ingest, query, and compaction while Nodes 2–3 handle ingest and query only. ``` # Node 1: Ingest, query, and compactor mode: ingest,query,compact cores: 32 io_threads: 8 ## datafusion_threads: 24 # Node 2: Ingest and query (no compaction) mode: ingest,query cores: 32 io_threads: 8 ## datafusion_threads: 24 # Node 3: Ingest and query (no compaction) mode: ingest,query cores: 32 io_threads: 8 datafusion_threads: 24 ``` ### Medium cluster (6 nodes) ``` # Nodes 1-2: Ingesters mode: ingest cores: 48 io_threads: 16 ## datafusion_threads: 32 # Nodes 3-4: Query nodes mode: query cores: 48 io_threads: 4 ## datafusion_threads: 44 # Node 5: Compactor (only one compactor per cluster) mode: compact cores: 32 io_threads: 4 ## datafusion_threads: 28 # Node 6: Process node mode: process cores: 32 io_threads: 4 datafusion_threads: 28 ``` ### Large cluster (10+ nodes) ``` # Nodes 1-4: High-throughput ingesters mode: ingest cores: 96 io_threads: 20 ## datafusion_threads: 76 # Nodes 5-8: Query nodes mode: query cores: 64 io_threads: 4 ## datafusion_threads: 60 # Node 9: Dedicated compactor (only one compactor per cluster) mode: compact cores: 32 io_threads: 2 ## datafusion_threads: 30 # Node 10: Process node mode: process cores: 32 io_threads: 6 datafusion_threads: 26 ``` ## Scale your cluster ### Vertical scaling limitations InfluxDB 3 Enterprise uses a shared-nothing architecture where ingest nodes handle all writes. To maximize ingest performance: * **Scale IO threads with concurrent writers**: Each concurrent writer can utilize approximately one IO thread for line protocol parsing * **Use high-core machines**: Line protocol parsing is CPU-intensive and benefits from more cores * **Deploy multiple ingest nodes**: Run several ingest nodes behind a load balancer to distribute write load * **Optimize batch sizes**: Configure clients to send larger batches to reduce per-request overhead ### Scale queries horizontally Query nodes can scale horizontally since they all access the same object store: ```bash # Add query nodes as needed for i in {1..10}; do influxdb3 \ --num-io-threads=4 \ serve \ --num-cores=32 \ --datafusion-num-threads=28 \ --mode=query \ --node-id=query-$i & done ``` ## Monitor performance ### Node-specific metrics Monitor specialized nodes differently based on their role: #### Ingest nodes ```sql -- Monitor write activity through parquet file creation SELECT table_name, count(*) as files_created, sum(row_count) as total_rows, sum(size_bytes) as total_bytes FROM system.parquet_files WHERE max_time > extract(epoch from now() - INTERVAL '5 minutes') * 1000000000 GROUP BY table_name; ``` > [!Note] > On the legacy Parquet storage engine, `system.parquet_files` lists only Gen1 > files, including compacted Gen1 files that haven’t been cleaned up. > It doesn’t list compacted generations. > For more information, see [Gen1 file cleanup](/influxdb3/enterprise/admin/gen1-file-cleanup/). #### Query nodes ```sql -- Monitor query performance SELECT count(*) as query_count, avg(execute_duration) as avg_execute_time, max(max_memory) as max_memory_bytes FROM system.queries WHERE issue_time > now() - INTERVAL '5 minutes' AND success = true; ``` #### Compactor nodes ```sql -- Monitor compaction progress SELECT event_type, event_status, count(*) as event_count, avg(event_duration) as avg_duration FROM system.compaction_events WHERE event_time > now() - INTERVAL '1 hour' GROUP BY event_type, event_status ORDER BY event_count DESC; ``` ### Monitor cluster-wide metrics ```bash # Check node health via HTTP endpoints for node in ingester-01:8181 query-01:8181 compactor-01:8181; do echo "Node: $node" curl -s "http://$node/health" done # Monitor metrics from each node for node in ingester-01:8181 query-01:8181 compactor-01:8181; do echo "=== Metrics from $node ===" curl -s "http://$node/metrics" | grep -E "(cpu_usage|memory_usage|http_requests_total)" done # Query system tables for cluster-wide monitoring curl -X POST "http://query-01:8181/api/v3/query_sql" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{ "q": "SELECT * FROM system.queries WHERE issue_time > now() - INTERVAL '\''5 minutes'\'' ORDER BY issue_time DESC LIMIT 10", "db": "sensors" }' ``` > [!Tip] > ### Extend monitoring with plugins > > Enhance your cluster monitoring capabilities using the InfluxDB 3 processing engine. The [InfluxDB 3 plugins library](https://github.com/influxdata/influxdb3_plugins) includes several monitoring and alerting plugins: > > * **System metrics collection**: Collect CPU, memory, disk, and network statistics > * **Threshold monitoring**: Monitor metrics with configurable thresholds and alerting > * **Multi-channel notifications**: Send alerts via Slack, Discord, SMS, WhatsApp, and webhooks > * **Anomaly detection**: Identify unusual patterns in your data > * **Deadman checks**: Detect missing data streams > > For complete plugin documentation and setup instructions, see [Process data in InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/process/). ### Monitor and respond to performance issues Use the [monitoring queries](#monitor-cluster-wide-metrics) to identify the following patterns and their solutions: #### High CPU with low throughput (Ingest nodes) **Detection query:** ```sql -- Check for high failed query rate indicating parsing issues SELECT count(*) as total_queries, sum(CASE WHEN success = true THEN 1 ELSE 0 END) as successful_queries, sum(CASE WHEN success = false THEN 1 ELSE 0 END) as failed_queries FROM system.queries WHERE issue_time > now() - INTERVAL '5 minutes'; ``` **Symptoms:** * Only 2 CPU cores at 100% on large machines * High write latency despite available resources * Failed queries due to parsing timeouts **Solution:** Increase IO threads (see [Ingest node issues](#ingest-node-issues)) #### Memory pressure alerts (Query nodes) **Detection query:** ```sql -- Monitor queries with high memory usage or failures SELECT avg(max_memory) as avg_memory_bytes, max(max_memory) as peak_memory_bytes, sum(CASE WHEN success = false THEN 1 ELSE 0 END) as failed_queries FROM system.queries WHERE issue_time > now() - INTERVAL '5 minutes' AND query_type = 'sql'; ``` **Symptoms:** * Queries failing with out-of-memory errors * High memory usage approaching pool limits * Slow query execution times **Solution:** Increase memory pool or optimize queries (see [Query node issues](#query-node-issues)) #### Compaction falling behind (Compactor nodes) **Detection query:** ```sql -- Check compaction event frequency and success rate SELECT event_type, count(*) as event_count, sum(CASE WHEN event_status = 'success' THEN 1 ELSE 0 END) as successful_events FROM system.compaction_events WHERE event_time > now() - INTERVAL '1 hour' GROUP BY event_type; ``` **Symptoms:** * Decreasing compaction event frequency * Growing number of small Parquet files * Increasing query times due to file fragmentation **Solution:** For nodes using the Parquet-backed storage engine, increase DataFusion threads on your single compactor node (see [Compactor node issues](#compactor-node-issues)). The upgraded storage engine does not use DataFusion for compaction—refer to the [storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/) for tuning guidance. ## Troubleshoot node configurations ### Ingest node issues **Problem**: Low throughput despite available CPU ```bash # Check: Are only 2 cores busy? top -H -p $(pgrep influxdb3) # Solution: Increase IO threads --num-io-threads=16 ``` **Problem**: Data snapshot creation affecting ingest ```bash # Check: DataFusion threads at 100% during data snapshots to Parquet # Solution: Reserve more DataFusion threads for snapshot operations --datafusion-num-threads=40 ``` ### Query node issues **Problem**: Slow queries despite resources ```bash # Check: Memory pressure free -h # Solution: Increase memory pool --exec-mem-pool-size=90% ``` **Problem**: Poor cache hit rates ```bash # Solution: Increase Parquet cache --file-cache-size=10GB ``` ### Compactor node issues **Problem**: Compaction falling behind ```bash # Check: Compaction queue length # Solution: Increase threads on the single compactor node (only one compactor is allowed per cluster) --datafusion-num-threads=30 ``` ## Best practices 1. **Start with monitoring**: Understand bottlenecks before specializing nodes 2. **Test mode combinations**: Some workloads benefit from multi-mode nodes 3. **Plan for failure**: Ensure redundancy in critical node types 4. **Document your topology**: Keep clear records of node configurations 5. **Regular rebalancing**: Adjust thread allocation as workloads evolve 6. **Capacity planning**: Monitor trends and scale proactively ## Migrate to specialized nodes ### From single-node to specialized cluster > [!Note] > `all` mode is only for single-node Enterprise deployments. When scaling a single `all` node cluster to a multi-node cluster: * Replace the `all` node with nodes that have explicit, specialized modes * Assign `compact` mode to exactly one node that uses the same node-id as the `all` node being replaced ```bash # Phase 1: Single-node deployment influxdb3 serve \ --node-id=node0 \ --cluster-id=my-cluster \ --mode=all \ --num-io-threads=8 # Phase 2: Scale to multi-node cluster # Stop the all-mode node and start specialized nodes. # The compact node MUST use the same --node-id as the replaced all-mode node. # Compact node: reuses the same node-id as the replaced all-mode node influxdb3 serve \ --node-id=node0 \ --cluster-id=my-cluster \ --mode=compact # Ingest and query node influxdb3 serve \ --node-id=node1 \ --cluster-id=my-cluster \ --mode=ingest,query \ --num-io-threads=8 # Phase 3: Full specialization (optional) # Dedicated ingest node influxdb3 serve \ --node-id=node1 \ --cluster-id=my-cluster \ --mode=ingest \ --num-io-threads=16 # Dedicated query node influxdb3 serve \ --node-id=node2 \ --cluster-id=my-cluster \ --mode=query \ --num-io-threads=4 ``` ## Manage configurations ### Configure using environment variables ```bash # Set environment variables for node type export INFLUXDB3_MODE=ingest export INFLUXDB3_NUM_IO_THREADS=20 export INFLUXDB3_DATAFUSION_NUM_THREADS=76 influxdb3 serve --node-id=$HOSTNAME --cluster-id=prod ``` #### Related * [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) * [Performance tuning](/influxdb3/enterprise/admin/performance-tuning/) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) * [Query system data](/influxdb3/enterprise/admin/query-system-data/) [clustering](/influxdb3/enterprise/tags/clustering/)[performance](/influxdb3/enterprise/tags/performance/)[tuning](/influxdb3/enterprise/tags/tuning/)[ingest](/influxdb3/enterprise/tags/ingest/)[threads](/influxdb3/enterprise/tags/threads/) | Architecture | Process node --mode | Recommendation | | --- | --- | --- | | Architecture | Process node --mode | Recommendation | | Standalone process node | process | Recommended | | Mixed process node | process and one or more of ingest , query , compact | Not recommended | --- # Manage databases Source: https://docs.influxdata.com/influxdb3/enterprise/admin/databases/ An InfluxDB 3 Enterprise database is a named location where time series data is stored. Each database can contain multiple tables. > [!Note] > **If coming from InfluxDB v1**, the concepts of databases and retention policies > have been combined into a single concept–database. Retention policies are no > longer part of the InfluxDB data model. > However, InfluxDB 3 Enterprise does > support InfluxQL, which requires databases and retention policies. > See [InfluxQL DBRP naming convention](/influxdb3/enterprise/admin/databases/create/#influxql-dbrp-naming-convention). > > **If coming from InfluxDB v2, InfluxDB Cloud (TSM), or InfluxDB Cloud Serverless**,*database* and *bucket* are synonymous. ## Retention periods A database **retention period** is the maximum age of data stored in the database. The age of data is determined by the timestamp associated with each point. When a point’s timestamp is beyond the retention period (relative to now), the point is marked for deletion and is removed from the database the next time the retention enforcement service runs. By default, data does not expire. When you [create a database](/influxdb3/enterprise/admin/databases/create/), you can optionally set a retention period. The minimum practical retention period is 1 hour (`1h`). Database retention periods can be [updated after creation](/influxdb3/enterprise/reference/cli/influxdb3/update/database/). For tables within a database, you can [set table-specific retention periods](/influxdb3/enterprise/admin/tables/create/#retention-period)that override the database retention period. For complete details about retention periods, including duration formats, precedence rules, and deletion behavior, see[Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/). ## Database, table, and column limits InfluxDB 3 Enterprise places the following limits on databases, tables, and columns: ### Database limit **Maximum number of databases**: 100 ### Table limit **Maximum number of tables across all databases**: 10000 InfluxDB 3 Enterprise limits the number of tables you can have across *all*databases to 10000 by default . You can configure the table limit using the[`--num-table-limit` configuration option](/influxdb3/enterprise/reference/config-options/#num-table-limit). InfluxDB doesn’t limit how many tables you can have in an individual database, as long as the total across all databases is below the limit. Having more tables affects your InfluxDB 3 Enterprise installation in the following ways: [](#may-improve-query-performance-view-more-info) **May improve query performance** *View more info* Schemas with many tables that contain[focused sets of tags and fields](/influxdb3/enterprise/write-data/best-practices/schema-design/#design-for-performance)can make it easier for the query engine to identify what Parquet files contain the queried data, resulting in better query performance. [](#more-puts-into-object-storage-view-more-info) **More PUTs into object storage** *View more info* When using cloud-based object storage as your data backend, the more tables you have, the more `PUT` requests there are into your object store as InfluxDB persists data to Parquet files. Each `PUT` request incurs a monetary cost and increases the operating cost of InfluxDB 3 Enterprise. [](#more-work-for-the-compactor-view-more-info) **More work for the compactor** *View more info* To optimize storage over time, InfluxDB 3 Enterprise has a compactor that routinely compacts Parquet files. With more tables and Parquet files to compact, the compactor may need to be scaled to keep up with demand, adding to the operating cost of InfluxDB 3 Enterprise. ### Column limit **Maximum number of columns per table**: 500 Each row must include a time column, with the remaining columns representing tags and fields. As a result, by default, a table can have one time column and up to 499*combined* field and tag columns. If you attempt to write to a table and exceed the column limit, the write request fails and InfluxDB returns an error. You can configure the maximum number of columns per table using the [`num-total-columns-per-table-limit` configuration option](/influxdb3/enterprise/reference/config-options/#num-total-columns-per-table-limit). Higher numbers of columns has the following side-effects: [](#may-adversely-affect-system-performance) May adversely affect system performance InfluxData identified 500 columns as the safe limit for maintaining system performance and stability. Exceeding this threshold can result in[wide schemas](/influxdb3/enterprise/write-data/best-practices/schema-design/#avoid-wide-schemas), which can negatively impact performance and resource use, depending on your queries, the shape of your schema, and data types in the schema. ## [Create a database](/influxdb3/enterprise/admin/databases/create/) Use the influxdb3 CLI, HTTP API, or InfluxDB 3 Explorer to create a new database in InfluxDB 3 Enterprise. ``` # influxdb3 CLI influxdb3 create database \ --retention-period 30d \ DATABASE_NAME # HTTP API curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "retention_period": "30d" }' ``` [Read more ](/influxdb3/enterprise/admin/databases/create/) ## [List databases](/influxdb3/enterprise/admin/databases/list/) Use the influxdb3 CLI, HTTP API, or InfluxDB 3 Explorer to list databases in InfluxDB 3 Enterprise. ```sh # influxdb3 CLI influxdb3 show databases # HTTP API curl --request GET "http://localhost:8181/api/v3/configure/database" \ --header "Authorization: Bearer AUTH_TOKEN" ``` [Read more ](/influxdb3/enterprise/admin/databases/list/) ## [Delete a database](/influxdb3/enterprise/admin/databases/delete/) Use the influxdb3 CLI, HTTP API, or InfluxDB 3 Explorer to delete a database from InfluxDB 3 Enterprise. ```sh # influxdb3 CLI influxdb3 delete database DATABASE_NAME # HTTP API curl --request DELETE "http://localhost:8181/api/v3/configure/database?db=DATABASE_NAME" \ --header "Authorization: Bearer AUTH_TOKEN" ``` [Read more ](/influxdb3/enterprise/admin/databases/delete/) ## [Enforce a schema](/influxdb3/enterprise/admin/databases/enforce-schema/) Use explicit schema mode to declare a database’s tables and columns up front and reject writes that reference anything undeclared. ```sh # Create a database that rejects undeclared tables and columns influxdb3 create database \ --schema-mode explicit \ --token AUTH_TOKEN \ DATABASE_NAME # HTTP API curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "schema_mode": "explicit" }' ``` [Read more ](/influxdb3/enterprise/admin/databases/enforce-schema/) #### Related * [InfluxDB schema design recommendations](/influxdb3/enterprise/write-data/best-practices/schema-design/) * [influxdb3 CLI](/influxdb3/enterprise/reference/cli/influxdb3/) * [Database API reference](/influxdb3/enterprise/api/v3/#tag/Database) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) * [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/) [databases](/influxdb3/enterprise/tags/databases/) --- # Create a database Source: https://docs.influxdata.com/influxdb3/enterprise/admin/databases/create/ Use the [`influxdb3 create database` command](/influxdb3/enterprise/reference/cli/influxdb3/create/database/), the [HTTP API](/influxdb3/enterprise/api/v3/), or [InfluxDB 3 Explorer](/influxdb3/explorer/) to create a database in InfluxDB 3 Enterprise. * [Create a database using the influxdb3 CLI](#create-a-database-using-the-influxdb3-cli) * [Create a database using the HTTP API](#create-a-database-using-the-http-api) * [Create a database using InfluxDB 3 Explorer](#create-a-database-using-influxdb-3-explorer) * [Retention period](#retention-period) * [Schema mode](#schema-mode) * [Database naming restrictions](#database-naming-restrictions) * [InfluxQL DBRP naming convention](#influxql-dbrp-naming-convention) * [Database limit](#database-limit) ## Create a database using the influxdb3 CLI Use the [`influxdb3 create database` command](/influxdb3/enterprise/reference/cli/influxdb3/create/database/)to create a database. Provide the following: * Database name *(see [Database naming restrictions](#database-naming-restrictions))* * InfluxDB 3 Enterprise [admin token](/influxdb3/enterprise/admin/tokens/admin) * *(Optional)* [Retention period](#retention-period). If omitted, data doesn’t expire. * *(Optional)* [Schema mode](#schema-mode). If omitted, the database uses `implicit` schema mode. ```sh # Create a database with a 30-day retention period influxdb3 create database --retention-period 30d DATABASE_NAME # Create a database with a 90-day retention period using authentication influxdb3 create database \ --retention-period 90d \ --token AUTH_TOKEN \ DATABASE_NAME # Create a database with infinite retention (default) influxdb3 create database DATABASE_NAME ``` ```sh # Create a database with explicit schema mode influxdb3 create database --schema-mode explicit DATABASE_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database to create * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ## Create a database using the HTTP API To create a database using the HTTP API, send a `POST` request to the `/api/v3/configure/database` endpoint: ``` POST http://localhost:8181/api/v3/configure/database ``` Include the following in your request: * **Headers**: * `Content-Type: application/json` * `Authorization: Bearer` with your [admin token](/influxdb3/enterprise/admin/tokens/) * **Request body** (JSON object): * `db` *(string, required)*: Database name * `retention_period` *(string, optional)*: [Retention period](#retention-period). If omitted, data doesn’t expire. * `schema_mode` *(string, optional)*: [Schema mode](#schema-mode): `implicit` or `explicit` . Defaults to `implicit`. ```bash # Create a database with a 30-day retention period curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "retention_period": "30d" }' # Create a database with a 90-day retention period curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "retention_period": "90d" }' # Create a database with infinite retention (default) curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME" }' ``` ```bash # Create a database with explicit schema mode curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "schema_mode": "explicit" }' ``` Replace the following: * `DATABASE_NAME`: the name of the database to create * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Response A successful request returns HTTP status `200` with the database configuration. ## Create a database using InfluxDB 3 Explorer > [!Caution] > Currently, you can’t set a retention period when creating a database in Explorer. > To create a database with a retention period, see one of the following: > > * [Create a database using the influxdb3 CLI](#create-a-database-using-the-influxdb3-cli) > * [Create a database using the HTTP API](#create-a-database-using-the-http-api) Use the [InfluxDB 3 Explorer](/influxdb3/explorer/) web interface to create a database in InfluxDB 3 Enterprise: 1. If you haven’t already, see how to [get started with Explorer and connect to your InfluxDB 3 Enterprise server](/influxdb3/explorer/get-started/). 2. In Explorer, click **Manage Databases** in the left navigation. 3. Click **+ Create New**. 4. Enter a database name. 5. Click **Create Database**. For more information, see [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/). * [Retention period](#retention-period) * [Schema mode](#schema-mode) * [Database naming restrictions](#database-naming-restrictions) * [InfluxQL DBRP naming convention](#influxql-dbrp-naming-convention) * [Database limit](#database-limit) ## Retention period By default, data does not expire. When creating a database, set a *retention period* to automatically delete expired data and optimize storage. ### Retention period syntax The retention period value is a time duration value made up of a numeric value plus a duration unit. For example, `30d` means 30 days. The retention period value cannot be negative or contain whitespace. #### Valid durations units include * **m**: minute * **h**: hour * **d**: day * **w**: week * **mo**: month * **y**: year > [!Warning] > #### Retention period constraints > > * **Minimum for data retention**: The practical minimum retention period is 1 hour (`1h`). > * **Zero-duration periods**: Setting a retention period to `0` (for example,`0d` or `0h`) is allowed but marks all data for immediate deletion at query time.*This differs from InfluxDB 1.x and 2.x where `0d` meant infinite retention.* > * **Infinite retention**: Use `none` to set an infinite retention period. For more information about retention periods, see [Data retention](/influxdb3/enterprise/reference/internals/data-retention/). > [!Note] > #### Database retention serves as default for tables > > The database retention period serves as the default retention period for all tables in > the database, unless a table has its own retention period defined. Table-level retention > periods override database retention periods. > > For more information, see [Retention period precedence](/influxdb3/enterprise/reference/internals/data-retention/#retention-period-precedence). ## Schema mode Every database has a *schema mode* that controls where table and column definitions come from: * **`implicit`** *(default)*: Tables and columns are created automatically from the line protocol you write. * **`explicit`**: Tables and columns must be declared with the[`influxdb3 create table` command](/influxdb3/enterprise/reference/cli/influxdb3/create/table/)or the `POST /api/v3/configure/table` API before you can write to them. A write that names an undeclared table or column is rejected. Schema mode is set when you create the database and can’t be changed afterward. If you need a different schema mode, create a new database and migrate your data. For more information, including how writes are rejected and how to evolve a declared schema, see[Enforce a schema](/influxdb3/enterprise/admin/databases/enforce-schema/). ## Database naming restrictions Database names must adhere to the following naming restrictions: * **Length**: Maximum 511 ASCII characters * **Allowed characters**: ASCII alphanumeric characters (a-z, A-Z, 0-9), underscore (`_`), and dash (`-`). Multibyte (non-ASCII) characters are not allowed. * **Forward slash**: At most one forward slash (`/`), which separates the database name from a v1 retention policy name (for example, `mydb/autogen`). The slash can’t be the first or last character. * **Prohibited characters**: Whitespace and characters other than those listed above * **Starting character**: Must start with an ASCII letter or number * **Case sensitivity**: Database names are case-sensitive ### Valid database name examples ```text mydb sensor_data prod-metrics logs/application webserver123 ``` ### Invalid database name examples ```text my database # Contains whitespace sensor.data # Contains period app@server # Contains special character _internal # Starts with underscore /mydb # Starts with a slash mydb/rp/extra # Contains more than one slash ``` For comprehensive information about naming restrictions for all InfluxDB identifiers, see [Naming restrictions and conventions](/influxdb3/enterprise/reference/naming-restrictions/). ## InfluxQL DBRP naming convention In InfluxDB 1.x, data is stored in [databases](/influxdb/v1/concepts/glossary/#database)and [retention policies](/influxdb/v1/concepts/glossary/#retention-policy-rp). In InfluxDB 3 Enterprise, databases and retention policies have been merged into*databases*, where databases have a retention period, but retention policies are no longer part of the data model. Because InfluxQL uses the 1.x data model, a database must be mapped to a v1 database and retention policy (DBRP) to be queryable with InfluxQL. **When naming a database that you want to query with InfluxQL**, use the following naming convention to automatically map v1 DBRP combinations to an InfluxDB 3 Enterprise database: ```text database_name/retention_policy_name ``` ### Database naming examples |v1 Database name|v1 Retention Policy name| New database name | |----------------|------------------------|-------------------------| | db | rp | db/rp | | telegraf | autogen | telegraf/autogen | | webmetrics | 1w-downsampled |webmetrics/1w-downsampled| ## Database limit **Default maximum number of databases**: 100 *For more information about InfluxDB 3 Enterprise database, table, and column limits, see [Database, table, and column limits](/influxdb3/enterprise/admin/databases/#database-table-and-column-limits).* #### Related * [influxdb3 create database](/influxdb3/enterprise/reference/cli/influxdb3/create/database/) * [Create database API](/influxdb3/enterprise/api/database/#operation/PostConfigureDatabase) * [Naming restrictions and conventions](/influxdb3/enterprise/reference/naming-restrictions/) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) * [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/) | v1 Database name | v1 Retention Policy name | New database name | | --- | --- | --- | | v1 Database name | v1 Retention Policy name | New database name | | db | rp | db/rp | | telegraf | autogen | telegraf/autogen | | webmetrics | 1w-downsampled | webmetrics/1w-downsampled | --- # Delete a database Source: https://docs.influxdata.com/influxdb3/enterprise/admin/databases/delete/ Use the [`influxdb3 delete database` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/database/), the [HTTP API](/influxdb3/enterprise/api/v3/), or [InfluxDB 3 Explorer](/influxdb3/explorer/) to delete a database from InfluxDB 3 Enterprise. > [!Caution] > #### Deleting a database cannot be undone > > Deleting a database is a destructive action. > Once a database is deleted, data stored in that database cannot be recovered. * [Delete a database using the influxdb3 CLI](#delete-a-database-using-the-influxdb3-cli) * [Delete a database using the HTTP API](#delete-a-database-using-the-http-api) * [Delete a database using InfluxDB 3 Explorer](#delete-a-database-using-influxdb-3-explorer) * [Delete data only (preserve schema and resources)](#delete-data-only-preserve-schema-and-resources) ## Delete a database using the influxdb3 CLI Provide the following: * Name of the database to delete * * InfluxDB 3 Enterprise [admin token](/influxdb3/enterprise/admin/tokens/admin) ```sh influxdb3 delete database DATABASE_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database to delete Enter `yes` to confirm that you want to delete the database. ## Delete a database using the HTTP API To delete a database using the HTTP API, send a `DELETE` request to the `/api/v3/configure/database` endpoint: ``` DELETE http://localhost:8181/api/v3/configure/database ``` Include the following in your request: * **Query parameters**: * `db`: Database name to delete * **Headers**: * `Authorization: Bearer` with your [admin token](/influxdb3/enterprise/admin/tokens/) ```bash curl --request DELETE "http://localhost:8181/api/v3/configure/database?db=DATABASE_NAME" \ --header "Authorization: Bearer AUTH_TOKEN" ``` Replace the following: * `DATABASE_NAME`: the name of the database to delete * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Response A successful deletion returns HTTP status `200` with no content body. ## Delete a database using InfluxDB 3 Explorer You can also delete databases using the [InfluxDB 3 Explorer](/influxdb3/explorer/) web interface: 1. If you haven’t already, see how to [get started with Explorer and connect to your InfluxDB 3 Enterprise server](/influxdb3/explorer/get-started/). 2. In Explorer, click **Databases** in the left navigation. 3. Find the database you want to delete in the list. 4. Click the **Delete** icon (trash can) next to the database name. 5. In the confirmation dialog, type the database name to confirm. 6. Click **Delete Database**. > [!Caution] > This action cannot be undone. All data in the database will be permanently deleted. For more information, see [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/). ## Delete data only (preserve schema and resources) InfluxDB 3 Enterprise supports deleting only the data in a database while preserving the database schema and associated resources. This is useful when you want to clear old data and re-write new data to the same structure without recreating resources. ### What is preserved When using the data-only deletion option, the following are preserved: * **Database schema**: Tables and column definitions * **Authentication tokens**: Database-scoped access tokens * **Processing engine configurations**: Triggers and plugin configurations * **Caches**: Last value caches (LVC) and distinct value caches (DVC) ### Delete data only using the CLI Use the [`--data-only`](/influxdb3/enterprise/reference/cli/influxdb3/delete/database/#options) flag to delete data while preserving the database schema and resources–for example: ```sh influxdb3 delete database --data-only DATABASE_NAME ``` Replace `DATABASE_NAME` with the name of your database. #### Delete data and remove tables To delete data and remove table schemas while preserving database-level resources (tokens, triggers, configurations), combine `--data-only` with [`--remove-tables`](/influxdb3/enterprise/reference/cli/influxdb3/delete/database/#options): ```sh influxdb3 delete database --data-only --remove-tables DATABASE_NAME ``` This preserves: * Authentication tokens * Processing engine triggers and configurations But removes: * All data * Table schemas * Table-level caches (LVC and DVC) ### Delete data only using the HTTP API To delete only data using the HTTP API, include the `data_only=true` query parameter: ```bash curl --request DELETE "http://localhost:8181/api/v3/configure/database?db=DATABASE_NAME&data_only=true" \ --header "Authorization: Bearer AUTH_TOKEN" ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) #### Delete data and remove tables To also remove table schemas, add the `remove_tables=true` parameter: ```bash curl --request DELETE "http://localhost:8181/api/v3/configure/database?db=DATABASE_NAME&data_only=true&remove_tables=true" \ --header "Authorization: Bearer AUTH_TOKEN" ``` #### Related * [influxdb3 delete database](/influxdb3/enterprise/reference/cli/influxdb3/delete/database/) * [Delete database API](/influxdb3/enterprise/api/database/#operation/DeleteConfigureDatabase) * [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/) --- # Enforce a schema Source: https://docs.influxdata.com/influxdb3/enterprise/admin/databases/enforce-schema/ By default, InfluxDB 3 Enterprise databases use *implicit* schema mode: a write that references a new table or column creates it.*Explicit* schema mode inverts that: tables and columns must be declared through the configuration API before you can write to them, and a write that references anything undeclared is rejected. Explicit schema mode is available in InfluxDB 3 Enterprise only. InfluxDB 3 Core rejects a request to create a database with `explicit`schema mode with HTTP status `400`(`explicit schema mode is only available in InfluxDB 3 Enterprise`). * [Create a database with explicit schema mode](#create-a-database-with-explicit-schema-mode) * [Declare tables and columns](#declare-tables-and-columns) * [Write to an explicit database](#write-to-an-explicit-database) * [Partial writes](#partial-writes) * [Evolve a declared schema](#evolve-a-declared-schema) * [Propagation across a cluster](#propagation-across-a-cluster) * [Limits](#limits) * [Upgrade considerations](#upgrade-considerations) ## Create a database with explicit schema mode [Schema mode](/influxdb3/enterprise/admin/databases/create/#schema-mode) is set when you create a database and can’t be changed afterward. ```sh influxdb3 create database \ --schema-mode explicit \ --token AUTH_TOKEN \ DATABASE_NAME ``` or ```bash curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "schema_mode": "explicit" }' ``` Replace the following: * `DATABASE_NAME`: the name of the database to create * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) `retention_period` still works alongside `schema_mode`. If you omit `schema_mode` or set it to `implicit`, the database uses the default, unenforced behavior. As a result, existing scripts and clients keep working unchanged. ## Declare tables and columns Use the same [`influxdb3 create table` command](/influxdb3/enterprise/reference/cli/influxdb3/create/table/)you’d use in an implicit database to declare a table’s tag and field columns: ```sh influxdb3 create table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --tags host,region \ --fields usage:float64,online:bool \ TABLE_NAME ``` or ```bash curl --request POST "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["host", "region"], "fields": [ {"name": "usage", "type": "float64"}, {"name": "online", "type": "bool"} ] }' ``` This is the same `POST /api/v3/configure/table` endpoint that declares tables in an implicit database. Explicit mode only changes what happens when a write later references a table or column that isn’t declared. A declared table always gets its `time` column automatically. > [!Important] > In InfluxDB 3 Enterprise 3.12 with the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/), > a query on a table that has no fields returns HTTP `500`. > In an explicit database, a write can’t add a field to a declared table. > Declare at least one field when you create the table. For field type options, see[Field data types](/influxdb3/enterprise/reference/cli/influxdb3/create/table/#field-data-types). ## Write to an explicit database Line protocol that matches the declared schema is accepted as usual: ```sh influxdb3 write --database DATABASE_NAME 'TABLE_NAME,host=a,region=west usage=1.0,online=true' ``` A write that names an undeclared table or column is rejected. An undeclared column error names the database, the table, the column, and the column type. An undeclared table error names the database and the table: ```text column 'rack' (iox::column_type::tag) is not defined in table 'TABLE_NAME' of database 'DATABASE_NAME', which uses explicit schemas; add the column with the /api/v3/configure/table API before writing to it ``` ```text table 'other_table' is not defined in database 'DATABASE_NAME', which uses explicit schemas; create the table with the /api/v3/configure/table API before writing to it ``` Both errors return HTTP status `400`. A few things to know about rejections: * **A declared column written at the wrong type** is rejected the same way it is in an implicit database, with the existing `InvalidColumnType`error. Explicit mode adds nothing here. The error has the following form: ``` invalid column type for column '', expected , got ``` * **A declared tag written as a field, or a field written as a tag,** is a wrong type on a declared column. It gets the same `invalid column type` error, not the undeclared-column error. * **Every write endpoint rejects the line with status `400`.**The v1 (`/write`), v2 (`/api/v2/write`), and v3 (`/api/v3/write_lp`) endpoints all enforce the declared schema. The endpoint changes only the error format and whether other lines in the batch are stored. See [Partial writes](#partial-writes). ## Partial writes A rejection is a line-protocol error like any other. As a result, the `accept_partial` parameter governs the rest of the batch on the `/api/v3/write_lp` endpoint: * With `accept_partial=false`, the whole request fails and nothing is written. * With `accept_partial=true`, the offending lines are reported and the rest of the batch is written. The `/api/v3/write_lp` endpoint defaults `accept_partial` to `true`. A client that sends a batch with one undeclared column and no`accept_partial` parameter gets a `400` response *and* has its other lines stored. The legacy `/write` and `/api/v2/write` endpoints fail the whole request when any line references an undeclared column, and they store nothing. Setting `accept_partial=true` doesn’t change this. ## Other ways data arrives The schema check also applies to data that doesn’t come through a line protocol write endpoint. | How the data arrives | Undeclared table or column | Other lines in the request | |-----------------------------------------------------------|---------------------------------------------------------------------------------------------|------------------------------------------------| | `/api/v3/write_lp`, default (`accept_partial=true`) | `400` `partial write of line protocol occurred`, and the body lists each rejected line | Written | | `/api/v3/write_lp`, `accept_partial=false` | `400` `line protocol parsing error` | Not written | | `/api/v2/write`, `/write` | `400`, and the server rejects the whole request | Not written | | Plugin, `influxdb3_local.write_sync()` | Raises an exception | Not applicable | | Plugin, `influxdb3_local.write()` |Fails after the plugin run, and the server logs an `ERROR` in `system.processing_engine_logs`| Not applicable | |Bulk import (`import upload` or `import from-object-store`)| `500` `Could not modify catalog`, and the server creates no import job |Not imported (the server rejects the whole file)| Every rejection message names the table or column and says to add it with the`/api/v3/configure/table` API. For a schedule trigger, a failed `write()` doesn’t stop the trigger. For details, see[Writes to explicit schema databases](/influxdb3/enterprise/plugins/python-api-reference/#writes-to-explicit-schema-databases). For bulk import, declare every column in the file before you import. See [Import data](/influxdb3/enterprise/admin/import-data/#import-into-an-explicit-schema-database). ### Declared columns written with the wrong type A line that writes a declared column with the wrong type is a type error, not an undeclared-column error, so declaring more columns doesn’t fix it. The following results come from requests with `accept_partial=false` and one line per request. Each request returns HTTP status `400`. A declared tag written as a field (`t,region=west host="x"`) returns the following message: ```text invalid column type for column 'host', expected iox::column_type::tag, got iox::column_type::field::string ``` A declared field written as a tag (`t,host=a,usage=5 …`) returns the following message: ```text invalid column type for column 'usage', expected iox::column_type::field::float, got iox::column_type::tag ``` A declared field written with the wrong type (`usage="str"` for a `float64`field) returns the following message: ```text invalid column type for column 'usage', expected iox::column_type::field::float, got iox::column_type::field::string ``` ## Evolve a declared schema Add tag and field columns to a declared table with the[`influxdb3 update table` command](/influxdb3/enterprise/reference/cli/influxdb3/update/table/)or a `PATCH` request to `/api/v3/configure/table`: ```sh influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --tags rack \ --fields temp:float64 \ TABLE_NAME ``` or ```bash curl --request PATCH "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["rack"], "fields": [{"name": "temp", "type": "float64"}] }' ``` The write that was rejected above is accepted once you declare the column. Schema evolution is **add-only**: you can declare new columns, but you can’t remove, rename, or retype an existing one. Declaring a column that already exists at its declared type is a no-op. Declaring it at a different type returns an error. > [!Important] > #### Use PATCH, not PUT, to add columns > > `PUT /api/v3/configure/table` updates only a table’s retention period. > The endpoint ignores a `tags` or `fields` array in the request body and still returns HTTP status `200`. > In an explicit database, the next write that references the undeclared column fails with a `400` error. For more information, see [Add columns to a table](/influxdb3/enterprise/admin/tables/update/). ## Propagation across a cluster In a multi-node cluster, nodes poll the object store for catalog updates at[`--catalog-sync-interval`](/influxdb3/enterprise/reference/cli/influxdb3/serve/)(default `1s`). Enforcement reads each node’s local view of the catalog. That view can lag the node that accepted a declaration by about one interval, and sometimes longer. A client that declares a column on one node and immediately writes it to a different node can have that write rejected. The reason is that the second node hasn’t yet seen the declaration. The rejection is a per-line error the client can retry. It self-corrects once the writing node’s catalog catches up. A node’s catalog advances synchronously with a declaration made on that node. That’s why a column declared on the same node you write to is never rejected this way. Add-only evolution is what keeps this safe. A lagging node’s view of a declared column is never ahead of the catalog, only behind. As a result, the lag can reject a write that should have been accepted. It can never accept a write that should have been rejected. ## Limits * **Explicit schema mode is fixed at creation.** No API changes it afterward. An implicit database can’t be made explicit. An explicit database can’t be relaxed to implicit. If you need a different mode, create a new database and migrate your data. * **Deleting and recreating a database resets the mode.** The mode belongs to the database that was created. As a result, recreating a database without `schema_mode` creates an implicit one. * **The `_internal` database is always implicit** and can’t be created, deleted, or patched through the API. * **Schema evolution is add-only.** Column removal, rename, and type change are out of scope. * **Value-level validation, per-column permissions, and retention rules**are separate concerns and aren’t part of schema mode. ## Upgrade considerations > [!Important] > #### Explicit schema mode requires your cluster to have already committed to 3.12 > > Creating an explicit database requires the cluster’s catalog to already > be at the catalog feature level that ships with 3.12. > The catalog commits that level automatically once every node in the > cluster is running 3.12. No explicit database has to exist for this to > happen. > Once every node has started on 3.12 and the cluster has committed to > that level, no node in the cluster can roll back to a 3.11.x binary, > whether or not you ever create an explicit database. > > By the time you can create an explicit database, your cluster has > therefore already lost 3.11.x rollback compatibility. > For more information about catalog version constraints during an > upgrade, see[Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/). #### Related * [Create a database](/influxdb3/enterprise/admin/databases/create/) * [Create a table](/influxdb3/enterprise/admin/tables/create/) * [Add columns to a table](/influxdb3/enterprise/admin/tables/update/) * [influxdb3 create database](/influxdb3/enterprise/reference/cli/influxdb3/create/database/) * [influxdb3 create table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) * [influxdb3 update table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) * [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/) [databases](/influxdb3/enterprise/tags/databases/)[tables](/influxdb3/enterprise/tags/tables/)[schema](/influxdb3/enterprise/tags/schema/)[write](/influxdb3/enterprise/tags/write/) | How the data arrives | Undeclared table or column | Other lines in the request | | --- | --- | --- | | How the data arrives | Undeclared table or column | Other lines in the request | | /api/v3/write_lp , default ( accept_partial=true ) | 400 partial write of line protocol occurred , and the body lists each rejected line | Written | | /api/v3/write_lp , accept_partial=false | 400 line protocol parsing error | Not written | | /api/v2/write , /write | 400 , and the server rejects the whole request | Not written | | Plugin, influxdb3_local.write_sync() | Raises an exception | Not applicable | | Plugin, influxdb3_local.write() | Fails after the plugin run, and the server logs an ERROR in system.processing_engine_logs | Not applicable | | Bulk import ( import upload or import from-object-store ) | 500 Could not modify catalog , and the server creates no import job | Not imported (the server rejects the whole file) | --- # List databases Source: https://docs.influxdata.com/influxdb3/enterprise/admin/databases/list/ Use the [`influxdb3 show databases` command](/influxdb3/enterprise/reference/cli/influxdb3/show/databases/), the [`/api/v3/configure/database`](/influxdb3/enterprise/api/v3/) HTTP API endpoint, or [InfluxDB 3 Explorer](/influxdb3/explorer/) to list databases in InfluxDB 3 Enterprise. * [List databases using the influxdb3 CLI](#list-databases-using-the-influxdb3-cli) * [List databases using the HTTP API](#list-databases-using-the-http-api) * [List databases using InfluxDB 3 Explorer](#list-databases-using-influxdb-3-explorer) ## List databases using the influxdb3 CLI Provide the following: * *(Optional)* [Output format](#output-formats) with the `--format` option * *(Optional)* [Show deleted databases](#list-deleted-databases) with the`--show-deleted` option * InfluxDB 3 Enterprise [admin token](/influxdb3/enterprise/admin/tokens/admin)with the `-t`, `--token` option ```sh influxdb3 show databases ``` ### Output formats The `influxdb3 show databases` command supports output formats: * `pretty` *(default)* * `json` * `jsonl` * `csv` * `parquet` *(must [output to a file](#output-to-a-parquet-file))* Use the `--format` flag to specify the output format: ```sh influxdb3 show databases --format json ``` #### Example output [](#view-example-pretty-formatted-output) View example pretty-formatted output ``` +---------------+ | iox::database | +---------------+ | home | | home_actions | | noaa | +---------------+ ``` [](#view-example-json-formatted-output) View example JSON-formatted output ```json [{"iox::database":"home"},{"iox::database":"home_actions"},{"iox::database":"noaa"}] ``` [](#view-example-json-line-formatted-output) View example JSON-line-formatted output ```jsonl {"iox::database":"home"} {"iox::database":"home_actions"} {"iox::database":"noaa"} ``` [](#view-example-csv-formatted-output) View example CSV-formatted output ```csv iox::database home home_actions noaa ``` #### Output to a Parquet file [Parquet](https://parquet.apache.org/) is a binary format. Use the `--output` option to specify the file where you want to save the Parquet data. ```sh influxdb3 show databases \ --format parquet \ --output databases.parquet ``` Alternatively, use the `influxdb3 query` command to query system tables: ```sh influxdb3 query \ --database _internal \ --format parquet \ --output databases.parquet \ "SELECT * FROM system.databases" ``` ### List deleted databases To list deleted databases, include the `--show-deleted` option with your`influxdb3 show databases` command: ```sh influxdb3 show databases --show-deleted ``` ## List databases using the HTTP API To list databases using the HTTP API, send a `GET` request to the `/api/v3/configure/database` endpoint. ``` GET http://localhost:8181/api/v3/configure/database?format=pretty ``` Include the `format` query parameter and specify one of the following formats: * `pretty` * `json` * `jsonl` * `csv` * `parquet` [Pretty output](#pretty-output)[JSON output](#json-output)[Parquet output](#parquet-output) Include the following in your request: * **Headers**: * `Authorization: Bearer` with your [token](/influxdb3/enterprise/admin/tokens/) * **Query Parameters**: * `format=pretty` ```bash curl --request GET "http://localhost:8181/api/v3/configure/database?format=pretty" \ --header "Authorization: Bearer AUTH_TOKEN" ``` The response body contains a table of database names: ```text +---------------------+ | iox::database | +---------------------+ | _internal | | home | | home_actions | | noaa | +---------------------+ ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your [token](/influxdb3/enterprise/admin/tokens/) * **Query Parameters**: * `format=json` ```bash curl --request GET "http://localhost:8181/api/v3/configure/database?format=json" \ --header "Authorization: Bearer AUTH_TOKEN" ``` The response body contains a JSON array of database objects whose keys are `iox::database`: ```json [ { "iox::database": "home" }, { "iox::database": "home_actions" }, { "iox::database": "noaa" } ] ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your [token](/influxdb3/enterprise/admin/tokens/) * **Query Parameters**: * `format=parquet` * An output destination for the Parquet file ```bash curl "http://localhost:8181/api/v3/configure/database?format=parquet" \ -o databases.parquet \ --header "Authorization: Bearer AUTH_TOKEN" ``` For Parquet responses, you must provide an output destination because the format is binary. The response contains the databases list. A successful request returns HTTP status `200`. ## List databases using InfluxDB 3 Explorer You can also view all databases using the [InfluxDB 3 Explorer](/influxdb3/explorer/) web interface: 1. If you haven’t already, see how to [get started with Explorer and connect to your InfluxDB 3 Enterprise server](/influxdb3/explorer/get-started/). 2. In Explorer, click **Databases** in the left navigation. 3. The Databases page displays a list of all databases with the following information: * Database name * Retention period (if configured) * Number of tables in the database * Creation date For more information, see [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/). #### Related * [influxdb3 show databases](/influxdb3/enterprise/reference/cli/influxdb3/show/databases/) * [List databases API](/influxdb3/enterprise/api/database/#operation/GetConfigureDatabase) * [Manage databases with InfluxDB 3 Explorer](/influxdb3/explorer/manage-databases/) --- # Delete data Source: https://docs.influxdata.com/influxdb3/enterprise/admin/delete-data/ Row-level deletion is an InfluxDB 3 Enterprise feature that requires the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters on InfluxDB 3 Enterprise 3.11+. On clusters that started on 3.10 or earlier, first run the[storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree)(`--upgrade-pacha-tree`). Use it to delete rows from a database by time range and an optional tag predicate. ## How row deletion works Row deletion is **asynchronous**. When you run `influxdb3 delete rows`, InfluxDB 3 Enterprise records a delete request and persists it to object storage. The [compactor](/influxdb3/enterprise/get-started/multi-server/#high-availability-with-a-dedicated-compactor)applies the request the next time it rewrites the affected run sets. By default, deletes don’t apply for up to **24 hours** after you submit the request. This delay is tunable with the `--row-delete-min-age` server flag. > [!Note] > Because deletes are applied during compaction, rows remain queryable until the > compactor processes the request. > Don’t expect rows to disappear immediately after the command returns. ## Delete rows Use the `influxdb3 delete rows` command to delete rows from a database. For the complete command syntax and flags, see the[`influxdb3 delete rows`](/influxdb3/enterprise/reference/cli/influxdb3/delete/rows/)CLI reference. ### Specify a time range A delete request must include an explicit time scope. Use one of the following: * `--min-time`: start of the range (**inclusive**) * `--max-time`: end of the range (**exclusive**) * `--all-time`: delete rows across all time ### Filter with a tag predicate You can narrow a delete to rows that match a tag predicate. Predicates support **tag equality only**: * Compare a tag to a string literal–for example, `region = "us-west"`. * Combine conditions with `AND` only. * `OR`, `NOT`, and `IN` operators, and field columns, are **not supported**. ### Required permissions Deleting rows from a database requires the `db::delete`[token permission](/influxdb3/enterprise/admin/tokens/). Deleting rows from the `_internal` database requires an admin token. ### Cancel a pending delete Use the `influxdb3 cancel row-delete` command to cancel a pending delete request before the compactor applies it. > [!Note] > InfluxDB 3 Enterprise allows at most **1000 pending delete requests** across the > cluster. ## Monitor row deletes Query the `system.row_deletes` system table to review the status of delete requests: ```sql SELECT * FROM system.row_deletes ``` InfluxDB 3 Enterprise also emits nine `influxdb3_compactor_row_delete_*` metrics that report row-deletion activity in the compactor. > [!Note] > A dedicated system-tables reference is planned as a follow-up. > Until then, this guide documents the `system.row_deletes` table. ## Known issues ### Ghost rows after a completed delete Rows in the un-compacted ingest tail can survive after a delete request reports`completed`. If you observe rows that should have been deleted, re-issue the delete after the affected data compacts, then verify row counts. ### system.row\_deletes can return HTTP 500 Querying `system.row_deletes` after an `--all-time` delete that has no tag predicate can return an HTTP `500` error. As a workaround, use the `GET /api/v3/row_delete_requests` API endpoint to review delete requests instead. #### Related * [Manage tokens](/influxdb3/enterprise/admin/tokens/) --- # Manage the Distinct Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/distinct-value-cache/ The InfluxDB 3 Enterprise Distinct Value Cache (DVC) lets you cache distinct values of one or more columns in a table, improving the performance of queries that return distinct tag and field values. The DVC is an in-memory cache that stores distinct values for specific columns in a table. When you create a DVC, you can specify what columns’ distinct values to cache, the maximum number of distinct value combinations to cache, and the maximum age of cached values. A DVC is associated with a table, which can have multiple DVCs. * [Create a Distinct Value Cache](#create-a-distinct-value-cache) * [Query a Distinct Value Cache](#query-a-distinct-value-cache) * [Show information about Distinct Value Caches](#show-information-about-distinct-value-caches) * [Delete a Distinct Value Cache](#delete-a-distinct-value-cache) * [Important things to know about the Distinct Value Cache](#important-things-to-know-about-the-distinct-value-cache) * [High cardinality limits](#high-cardinality-limits) Consider a dataset with the following schema: * wind\_data (table) * tags: * country * *multiple European countries* * county * *multiple European counties* * city * *multiple European cities* * fields: * wind\_speed (float) * wind\_direction (integer) If you cache distinct values for `country`, `county`, and `city`, the DVC looks similar to this: | country | county | city | |--------------|----------------|------------| | Austria | Salzburg | Salzburg | | Austria | Vienna | Vienna | | Belgium | Antwerp | Antwerp | | Belgium | West Flanders | Bruges | |Czech Republic| Liberec Region | Liberec | |Czech Republic| Prague | Prague | | Denmark | Capital Region | Copenhagen | | Denmark |Southern Denmark| Odense | | Estonia |Ida-Viru County |Kohtla-Järve| | Estonia |Ida-Viru County | Narva | | … | … | … | > [!Important] > #### Repeated values in DVC results > > Distinct values may appear multiple times in a column when querying the DVC, > but only when associated with distinct values in other columns. > If you query a single column in the DVC, no values are repeated in the results. > [!Note] > #### Null column values > > *Null* column values are still considered values and are cached in the DVC. > If you write data to a table and don’t provide a value for an existing column, > the column value is cached as *null* and treated as a distinct value. ## [Create a Distinct Value Cache](/influxdb3/enterprise/admin/distinct-value-cache/create/) Use the [`influxdb3 create distinct_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/create/distinct_cache/)to create a Distinct Value Cache. ```bash influxdb3 create distinct_cache \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ --table home \ --node-spec "nodes:node-01,node-02" \ --columns country,county,city \ --max-cardinality 10000 \ --max-age 24h \ windDistinctCache ``` ## [Query a Distinct Value Cache](/influxdb3/enterprise/admin/distinct-value-cache/query/) Use the [`distinct_cache()` SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#distinct_cache)in the `FROM` clause of an SQL `SELECT` statement to query data from the Distinct Value Cache. ```sql SELECT * FROM distinct_cache('table-name', 'cache-name') ``` > [!Important] > You must use SQL to query the DVC. > InfluxQL does not support the `distinct_cache()` function. ## [Show information about Distinct Value Caches](/influxdb3/enterprise/admin/distinct-value-cache/show/) Use the `influxdb3 show system table` command to query and output Distinct Value Cache information from the `distinct_caches` system table. ```bash influxdb3 show system \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ table distinct_caches ``` ## [Delete a Distinct Value Cache](/influxdb3/enterprise/admin/distinct-value-cache/delete/) Use the [`influxdb3 delete distinct_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/distinct_cache/)to delete a Distinct Value Cache. ```bash influxdb3 delete distinct_cache \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ --table wind_data \ windDistinctCache ``` ## Important things to know about the Distinct Value Cache DVCs are stored in memory; the larger the cache, the more memory your InfluxDB 3 node requires to maintain it. Consider the following: * [Cache data loading](#cache-data-loading) * [High cardinality limits](#high-cardinality-limits) ## Cache data loading On cache creation, InfluxDB 3 Enterprise loads historical data into the cache. On restart, the server automatically reloads cache data. In a multi-node cluster, only nodes included in the cache’s[node specification](/influxdb3/enterprise/admin/distinct-value-cache/create/)load historical data–at creation and on restart. Excluded nodes still serve the cache and add newly written data to it, but until new data arrives, queries served by those nodes return only values written after the cache was created (or after the node restarted). Only specify a node list to reduce the initial cache warm-up load, and only if you accept temporarily incomplete query results from excluded nodes. ### High cardinality limits “Cardinality” refers to the number of unique key column combinations in your cached data and essentially defines the maximum number of rows to store in your DVC. While the InfluxDB 3 storage engine is not limited by cardinality, it does affect the DVC. You can define a custom maximum cardinality limit for a DVC, but higher cardinality increases memory requirements for storing the DVC and can affect DVC query performance. #### Related * [distinct\_cache SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#distinct_cache) * [Manage Distinct Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/distinct-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) | country | county | city | | --- | --- | --- | | country | county | city | | Austria | Salzburg | Salzburg | | Austria | Vienna | Vienna | | Belgium | Antwerp | Antwerp | | Belgium | West Flanders | Bruges | | Czech Republic | Liberec Region | Liberec | | Czech Republic | Prague | Prague | | Denmark | Capital Region | Copenhagen | | Denmark | Southern Denmark | Odense | | Estonia | Ida-Viru County | Kohtla-Järve | | Estonia | Ida-Viru County | Narva | | … | … | … | --- # Create a Distinct Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/distinct-value-cache/create/ Use the [`influxdb3 create distinct_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/create/distinct_cache/)to create a Distinct Value Cache (DVC). Provide the following: * **Database** (`-d`, `--database`): *(Required)* The name of the database to associate the DVC with. You can also use the `INFLUXDB3_DATABASE_NAME`environment variable to specify the database. * **Token** (`--token`): *(Required)* Your InfluxDB 3 Enterprise admin authentication token. You can also use the `INFLUXDB3_AUTH_TOKEN` environment variable to specify the token. * **Table** (`-t`, `--table`): *(Required)* The name of the table to associate the DVC with. * **Node specification** (`-n`, `--node-spec`): Specify which nodes load historical data into the DVC. The default (`all`) loads historical data on all query- and processing-capable nodes. To reduce the initial cache warm-up load, use `nodes:[,...]` to limit the historical data load to specific nodes. All query-capable nodes serve the DVC and add newly written data to it, but excluded nodes return incomplete query results until new data arrives. Only specify a node list if you accept temporarily incomplete query results from excluded nodes. * **Columns** (`--columns`): *(Required)* Specify which columns to cache distinct values for. These are typically tag columns but can also be string fields. * **Maximum cardinality** (`--max-cardinality`): Specify the maximum number of distinct value combinations to store in the cache. The default maximum cardinality is `100000`. * **Maximum age** (`--max-age`): Specify the maximum age of distinct values to keep in the DVC in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)form. The default maximum age is `24 hours`. * **Cache name**: A unique name for the cache. If you don’t provide one, InfluxDB automatically generates a cache name for you. ```bash influxdb3 create distinct_cache \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ --node-spec "nodes:NODE_LIST" \ --columns COLUMNS \ --max-cardinality MAX_CARDINALITY \ --max-age MAX_AGE \ DVC_NAME ``` ## Use the HTTP API To use the HTTP API to create a Distinct Value Cache, send a `POST` request to the `/api/v3/configure/distinct_cache` endpoint. ``` POST /api/v3/configure/distinct_cache ``` ```bash curl -X POST "https://localhost:8181/api/v3/configure/distinct_cache" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "name": "DVC_NAME", "columns": ["COLUMNS"], "max_cardinality": MAX_CARDINALITY, "max_age": MAX_AGE }' ``` ### Example ``` curl -X POST "https://localhost:8181/api/v3/configure/distinct_cache" \ --header "Authorization: Bearer 00xoXX0xXXx0000XxxxXx0Xx0xx0" \ --json '{ "db": "example-db", "table": "wind_data", "name": "windDistinctCache", "columns": ["country", "county", "city"], "max_cardinality": 10000, "max_age": 86400 }' ``` **Response codes:** * `201` : Success. The distinct cache has been created. * `204` : Not created. A distinct cache with this configuration already exists. * `400` : Bad request. > [!Note] > #### API parameter differences > > * **Columns format**: The API uses a JSON array (`["country", "county", "city"]`) > instead of the CLI’s comma-delimited format (`country,county,city`). > * **Maximum age format**: The API uses seconds (`86400`) instead of the CLI’s[humantime format](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) (`24h`, `1 day`). Replace the following: * `DATABASE_NAME`: the name of the database to associate the DVC with * `AUTH_TOKEN`: your InfluxDB 3 Enterprise admin authentication token * `TABLE_NAME`: the name of the table to associate the DVC with * `NODE_LIST`: a comma-delimited list of node IDs to configure the DVC on–for example:`node-01,node-02`. * `COLUMNS`: a comma-delimited list of columns to cache distinct values for–for example:`country,county,city` * `MAX_CARDINALITY`: the maximum number of distinct value combinations to cache–for example: `10000` * `MAX_AGE`: the maximum age of distinct values to keep in the cache in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)form–for example: `6h`, `1 day`, `1 week` * `DVC_NAME`: a unique name for the DVC The cache imports the distinct values from the table and starts caching them. > [!Important] > #### DVC size and persistence > > The DVC is stored in memory, so it’s important to consider the size and > persistence of the cache. For more information, see[Important things to know about the Distinct Value Cache](/influxdb3/enterprise/admin/distinct-value-cache/#important-things-to-know-about-the-distinct-value-cache). #### Related * [influxdb3 create distinct\_cache](/influxdb3/enterprise/reference/cli/influxdb3/create/distinct_cache/) * [Manage Distinct Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/distinct-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) --- # Delete a Distinct Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/distinct-value-cache/delete/ Use the [`influxdb3 delete distinct_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/distinct_cache/)to delete a Distinct Value Cache (DVC). Provide the following: * **Database** (`-d`, `--database`): *(Required)* The name of the database that the DVC you want to delete is associated with. You can also use the`INFLUXDB3_DATABASE_NAME` environment variable to specify the database. * **Token** (`--token`): *(Required)* Your InfluxDB 3 Enterprise admin authentication token. You can also use the `INFLUXDB3_AUTH_TOKEN` environment variable to specify the token. * **Table** (`-t`, `--table`): *(Required)* The name of the table that the DVC you want to delete is associated with. * **Cache name**: The name of the DVC to delete. ```bash influxdb3 delete distinct_cache \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ DVC_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database that the DVC you want to delete is associated with * `AUTH_TOKEN`: your InfluxDB 3 Enterprise admin authentication token * `TABLE_NAME`: the name of the table associated with the DVC you want to delete * `DVC_NAME`: the name of the DVC to delete > [!Caution] > This is a destructive action that cannot be undone. Once deleted, any queries > against the deleted DVC will return an error. #### Related * [influxdb3 delete distinct\_cache](/influxdb3/enterprise/reference/cli/influxdb3/delete/distinct_cache/) * [Manage Distinct Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/distinct-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) --- # Query a Distinct Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/distinct-value-cache/query/ Use the [`distinct_cache()` SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#distinct_cache)in the `FROM` clause of an SQL `SELECT` statement to query data from the Distinct Value Cache (DVC). > [!Important] > You must use SQL to query the DVC. > InfluxQL does not support the `distinct_cache()` function. `distinct_cache()` supports the following arguments: * **table\_name**: *(Required)* The name of the table the DVC is associated with formatted as a string literal. * **cache\_name**: The name of the DVC to query formatted as a string literal. This argument is only required if there is more than one DVC associated with the specified table. ```sql SELECT * FROM distinct_cache('table_name', 'cache_name') ``` You can use other [SQL clauses](/influxdb3/enterprise/reference/sql/#statements-and-clauses)to modify query results. For example, you can use the `WHERE` clause to return the distinct tag values associated with another distinct tag value: ```sql SELECT city FROM distinct_cache('wind_data', 'windDistinctCache') WHERE country = 'Spain' ``` ## Use the HTTP API To use the HTTP API to query cached data, send a `GET` or `POST` request to the `/api/v3/query_sql` endpoint and include the [`distinct_cache()`](/influxdb3/enterprise/reference/sql/functions/cache/#distinct_cache) function in your query. ``` GET /api/v3/query_sql ``` ``` POST /api/v3/query_sql ``` ```bash curl -X POST "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "q": "SELECT * FROM distinct_cache('\''TABLE_NAME'\'', '\''CACHE_NAME'\'')", "format": "json" }' ``` ## Example with WHERE clause ```bash curl -X POST "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer 00xoXX0xXXx0000XxxxXx0Xx0xx0" \ --json '{ "db": "example-db", "q": "SELECT room, temp FROM last_cache('\''home'\'', '\''homeCache'\'') WHERE room = '\''Kitchen'\''", "format": "json" }' ``` #### Related * [distinct\_cache SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#distinct_cache) * [Manage Distinct Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/distinct-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) --- # Show information about Distinct Value Caches Source: https://docs.influxdata.com/influxdb3/enterprise/admin/distinct-value-cache/show/ Use the [`influxdb3 show system table` command](/influxdb3/enterprise/reference/cli/influxdb3/show/system/table/)to query and output Distinct Value Cache information from the `distinct_caches`system table. ```bash influxdb3 show system \ --database DATABASE_NAME \ --token AUTH_TOKEN \ table distinct_caches ``` This returns a table similar to the following: | table | name |column\_ids| column\_names |max\_cardinality|max\_age\_seconds| |----------|-----------------|-----------|-----------------------|----------------|-----------------| |wind\_data| wind\_distinct | [0, 1, 2] |[country, county, city]| 100000 | 86400 | | weather |weather\_distinct| [0] | [location] | 100 | 604800 | | bitcoin | bitcoin\_dis | [0, 1] | [code, crypto] | 5000 | 86400 | | home | home\_distinct | [0, 1] | [room, wall] | 12000 | 15770000 | ## Query specific columns from the distinct\_caches system table Use the `--select` option to query specific columns from the `distinct_caches`system table. Provide a comma-delimited list of columns to return: ```bash influxdb3 show system \ --database DATABASE_NAME \ --token AUTH_TOKEN \ table distinct_caches \ --select name,column_names,max_age_seconds ``` ## Sort distinct\_caches system table output Use the `--order-by` option to sort data from the `distinct_caches` system table by specific columns. Provide a comma-delimited list of columns to sort by: ```bash influxdb3 show system \ --database DATABASE_NAME \ --token AUTH_TOKEN \ table distinct_caches \ --order-by max_cardinality,max_age_seconds ``` > [!Note] > Results are sorted in ascending order based on the provided columns. In the examples above, replace the following: * `DATABASE_NAME`: the name of the database to query system data from * `AUTH_TOKEN`: your InfluxDB 3 Enterprise admin authentication token ## Use the HTTP API To use the HTTP API to query and output cache information from the system table, send a `GET` or `POST` request to the `/api/v3/query_sql` endpoint. ``` GET /api/v3/query_sql ``` ``` POST /api/v3/query_sql ``` ### Query all caches ```bash curl -X POST "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "q": "SELECT * FROM system.distinct_caches", "format": "json" }' ``` ## Query specific cache details ```bash curl -X POST "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "q": "SELECT * FROM system.distinct_caches WHERE name = '\''CACHE_NAME'\''", "format": "json" }' ``` #### Related * [Manage Distinct Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/distinct-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) | table | name | column_ids | column_names | max_cardinality | max_age_seconds | | --- | --- | --- | --- | --- | --- | | table | name | column_ids | column_names | max_cardinality | max_age_seconds | | wind_data | wind_distinct | [0, 1, 2] | [country, county, city] | 100000 | 86400 | | weather | weather_distinct | [0] | [location] | 100 | 604800 | | bitcoin | bitcoin_dis | [0, 1] | [code, crypto] | 5000 | 86400 | | home | home_distinct | [0, 1] | [room, wall] | 12000 | 15770000 | --- # Distributed compaction Source: https://docs.influxdata.com/influxdb3/enterprise/admin/distributed-compaction/ Distributed compaction lets a cluster run compaction jobs on every[compact node](/influxdb3/enterprise/admin/clustering/#configure-node-modes), not only the node that currently holds the compactor lease. > [!Important] > #### Beta feature > > Distributed compaction is a beta feature of the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). > It isn’t available on clusters still running the Parquet engine. > The default configuration (`--compactor-dispatch-target local`) keeps the > v3.11 behavior, so upgrading to 3.12 doesn’t change compaction placement > unless you opt in. * [How distributed compaction works](#how-distributed-compaction-works) * [Choose a dispatch target](#choose-a-dispatch-target) * [Size compact nodes for the fleet](#size-compact-nodes-for-the-fleet) * [Failure handling](#failure-handling) * [Monitor distributed compaction](#monitor-distributed-compaction) * [Example: a fleet of compact nodes](#example-a-fleet-of-compact-nodes) ## How distributed compaction works At any moment, exactly one compact node holds the compactor lease and acts as the **primary**. The primary plans compaction work, dispatches it, and publishes the compactor checkpoint. It never hands scheduling state to another node. Every other node running in `compact` or `all` mode is a potential**worker**. A worker only takes work if it advertises an internode address with`--internode-bind-addr` (see [conn-info](/influxdb3/enterprise/reference/config-options/#conn-info)) so the primary can connect to it. A compact node without an internode address stays idle as a worker, even though it still competes for the compactor lease. Where a job actually executes is a configuration choice, not a fixed role: the primary can run jobs itself, hand them to workers, or both, depending on `--compactor-dispatch-target`. If the primary is replaced, for example because the node restarts or loses the lease, a standby compact node takes over, reloads the compactor checkpoint, and continues scheduling. Uncommitted work is replanned; nothing dispatched is lost. ## Choose a dispatch target [`--compactor-dispatch-target`](/influxdb3/enterprise/reference/config-options/#compactor-dispatch-target)(environment variable `INFLUXDB3_COMPACTOR_DISPATCH_TARGET`) selects where compaction jobs run: | Value | Behavior | |-------------------|---------------------------------------------------------------------------------------------------------------------------------| |`local` *(default)*| Every job runs on the lease holder itself. This matches InfluxDB 3 Enterprise 3.11 and earlier behavior. | | `remote` |Jobs run only on other compact nodes. With no worker connected, jobs wait and the dispatcher retries as workers become available.| | `all` | The lease holder and every other compact node are eligible. | Set the same `--compactor-dispatch-target` value on every compact node. Only the primary’s value is used; a worker’s own value has no effect while it is a worker, and it still runs the jobs dispatched to it. A mismatch shows up when the lease moves: the new primary applies its own value, so compaction placement changes. For example, if a node set to `local`takes the lease, every other compact node goes idle. `remote` keeps compaction execution off the primary, so an execution failure that would otherwise kill the process (such as an out-of-memory condition) can’t take the scheduler down with it. Consider `remote` or `all` when compaction jobs are large enough to risk destabilizing the node that also plans and dispatches them. ## Size compact nodes for the fleet Run compact nodes of the same size when you use `remote` or `all`. The primary plans and hands out work against the capacity of the fleet, not against its own node, and it assumes every worker reports the same execution limits: * [`--compactor-input-size-budget`](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor) * [`--compactor-max-concurrent-merges`](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor) If compact nodes advertise different values for either flag, the primary plans for the **smallest** value across the fleet and logs a warning naming the fields that differ. An unequal fleet still runs correctly, but it gets the throughput of its weakest node. ## Failure handling Distributed compaction is built to lose workers and the primary without losing work: * **A worker dies or stops responding.** The primary detects the loss through a missed heartbeat or a failed dispatch acknowledgment (`--compactor-dispatch-ack-timeout`, default `5s`), fails that worker’s in-flight jobs retryably, and redispatches them to another eligible worker. Any files the dead worker already wrote become orphans for[orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/)to reclaim. * **The primary dies.** A standby compact node takes the lease, reloads the compactor checkpoint, and replans uncommitted work. The scheduling gap is roughly one lease term. * **A worker rejects a dispatch.** A worker at capacity or draining rejects the dispatch without costing a retry attempt: the primary requeues the job for the next pass. ## Monitor distributed compaction Every compact node exports Prometheus metrics under the`influxdb3_compactor_` prefix at `/metrics`. A metric only reports a nonzero value on the node playing the role it describes: lease metrics move on every node, dispatch and fleet-capacity metrics move on the current primary, and execution metrics move on whichever node runs the work. > [!Note] > Prometheus exposition appends a `_total` suffix to counters. > For example, `influxdb3_compactor_dispatches` (a counter) is scraped as`influxdb3_compactor_dispatches_total`. > Gauges, such as `influxdb3_compactor_is_primary`, keep their exact name. | Metric | Type | Labels | Reports | |----------------------------------------------|-------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `influxdb3_compactor_is_primary` | Gauge | |`1` on the node that holds the compactor lease, `0` everywhere else. Alert if no node reports `1`, or if more than one node reports `1` for longer than about one lease term.| | `influxdb3_compactor_lease_version` | Gauge | | The current lease term. Steps on every term change. | | `influxdb3_compactor_fleet_workers` | Gauge | | Workers the primary currently counts for planning. `0` with `--compactor-dispatch-target remote` means compaction is stalled for want of a worker. | | `influxdb3_compactor_worker_discovered` | Gauge |`node_id`| `1` for each compaction-eligible peer the primary finds in the catalog with connection info. | | `influxdb3_compactor_worker_connected` | Gauge |`node_id`| `1` for each worker with a live claim and heartbeat. `0` while `worker_discovered` is `1` means the primary can’t reach that worker. | | `influxdb3_compactor_worker_failures` |Counter|`node_id`| Unplanned worker deaths, per worker. | | `influxdb3_compactor_dispatches` |Counter|`node_id`| Jobs handed to a worker, per receiving node (redispatches included). All work landing on the primary’s own node in a multi-node cluster means no remote worker is in use. | | `influxdb3_compactor_dispatch_ack_timeouts` |Counter|`node_id`| Dispatches a worker didn’t acknowledge within `--compactor-dispatch-ack-timeout`. Each timeout disconnects that worker. | |`influxdb3_compactor_unassigned_output_groups`| Gauge | | Jobs held because no worker is ready or has room for them. A sustained value above zero with `remote` means no worker is connected. | | `influxdb3_compactor_dispatch_held_no_fit` |Counter| | Times the primary held a job because no eligible worker had room. Read alongside `unassigned_output_groups` to tell a saturated fleet from an absent one. | For the complete metric set, including per-worker capacity, file-ID grant, and work-host metrics, see the compactor observability reference shipped with InfluxDB 3 Enterprise. ## Example: a fleet of compact nodes The following example starts two compact nodes that share compaction work. Replace the following: * `CLUSTER_ID`: your cluster identifier * `S3_BUCKET`: your object store bucket name * `COMPACT_NODE_1_ADDR`,`COMPACT_NODE_2_ADDR`: the internode gRPC addresses of each compact node ```bash # Compact node 1 influxdb3 serve \ --node-id compact01 \ --cluster-id CLUSTER_ID \ --mode compact \ --object-store s3 \ --bucket S3_BUCKET \ --compactor-dispatch-target all \ --compactor-input-size-budget 12GB \ --compactor-max-concurrent-merges 8 \ --internode-bind-addr COMPACT_NODE_1_ADDR # Compact node 2 influxdb3 serve \ --node-id compact02 \ --cluster-id CLUSTER_ID \ --mode compact \ --object-store s3 \ --bucket S3_BUCKET \ --compactor-dispatch-target all \ --compactor-input-size-budget 12GB \ --compactor-max-concurrent-merges 8 \ --internode-bind-addr COMPACT_NODE_2_ADDR ``` Whichever node acquires the compactor lease first becomes the primary. Because both nodes set `--compactor-dispatch-target all`, the primary plans work for itself and the other node alike. #### Related * [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) * [Orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/) * [Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) * [influxdb3 debug](/influxdb3/enterprise/reference/cli/influxdb3/debug/) [compaction](/influxdb3/enterprise/tags/compaction/)[clustering](/influxdb3/enterprise/tags/clustering/)[performance](/influxdb3/enterprise/tags/performance/)[beta](/influxdb3/enterprise/tags/beta/) | Value | Behavior | | --- | --- | | Value | Behavior | | local (default) | Every job runs on the lease holder itself. This matches InfluxDB 3 Enterprise 3.11 and earlier behavior. | | remote | Jobs run only on other compact nodes. With no worker connected, jobs wait and the dispatcher retries as workers become available. | | all | The lease holder and every other compact node are eligible. | | Metric | Type | Labels | Reports | | --- | --- | --- | --- | | Metric | Type | Labels | Reports | | influxdb3_compactor_is_primary | Gauge | | 1 on the node that holds the compactor lease, 0 everywhere else. Alert if no node reports 1 , or if more than one node reports 1 for longer than about one lease term. | | influxdb3_compactor_lease_version | Gauge | | The current lease term. Steps on every term change. | | influxdb3_compactor_fleet_workers | Gauge | | Workers the primary currently counts for planning. 0 with --compactor-dispatch-target remote means compaction is stalled for want of a worker. | | influxdb3_compactor_worker_discovered | Gauge | node_id | 1 for each compaction-eligible peer the primary finds in the catalog with connection info. | | influxdb3_compactor_worker_connected | Gauge | node_id | 1 for each worker with a live claim and heartbeat. 0 while worker_discovered is 1 means the primary can’t reach that worker. | | influxdb3_compactor_worker_failures | Counter | node_id | Unplanned worker deaths, per worker. | | influxdb3_compactor_dispatches | Counter | node_id | Jobs handed to a worker, per receiving node (redispatches included). All work landing on the primary’s own node in a multi-node cluster means no remote worker is in use. | | influxdb3_compactor_dispatch_ack_timeouts | Counter | node_id | Dispatches a worker didn’t acknowledge within --compactor-dispatch-ack-timeout . Each timeout disconnects that worker. | | influxdb3_compactor_unassigned_output_groups | Gauge | | Jobs held because no worker is ready or has room for them. A sustained value above zero with remote means no worker is connected. | | influxdb3_compactor_dispatch_held_no_fit | Counter | | Times the primary held a job because no eligible worker had room. Read alongside unassigned_output_groups to tell a saturated fleet from an absent one. | --- # Use the integrated Explorer UI Source: https://docs.influxdata.com/influxdb3/enterprise/admin/explorer-ui/ * InfluxDB 3 Enterprise v3.11+ InfluxDB 3 Enterprise embeds InfluxDB 3 Explorer 1.11 and serves it directly from the server process. You don’t need to run a separate[standalone Explorer](/influxdb3/explorer/) deployment to get a query and dashboarding UI. * [Enable the Explorer UI](#enable-the-explorer-ui) * [Quick start without authentication](#quick-start-without-authentication) * [Quick start with user authentication](#quick-start-with-user-authentication) * [Default connections and query routing](#default-connections-and-query-routing) * [Configure SSO for the Explorer UI](#configure-sso-for-the-explorer-ui) * [Sessions](#sessions) * [Explorer application data](#explorer-application-data) * [Migrate data from the 3.11 UI](#migrate-data-from-the-311-ui) ## Enable the Explorer UI Include `webui` in `--mode` and set[`--webui-session-secret`](/influxdb3/enterprise/reference/config-options/#webui-session-secret): ```bash influxdb3 serve --mode all,webui --webui-session-secret WEBUI_SESSION_SECRET ``` Replace `WEBUI_SESSION_SECRET`with a unique secret. Every node in the cluster that runs `webui` mode must share the same session secret so a browser session stays valid across nodes. `--mode` accepts `webui` alongside any other mode (for example, `all,webui`, `query,webui`, or `webui` by itself). A node that doesn’t include `webui` in `--mode` doesn’t serve the UI. The server serves Explorer at the root path of its regular HTTP address and port–for example, [http://localhost:8181/](http://localhost:8181/). Explorer doesn’t use a separate port. Explorer runs without a plugin directory. To use the plugin features in Explorer, create a directory and pass it to[`--plugin-dir`](/influxdb3/enterprise/reference/config-options/#plugin-dir). > [!Important] > #### Control who can reach Explorer > > What someone can do after reaching Explorer depends on whether[user authentication](/influxdb3/enterprise/admin/security/manage-users/)is enabled. > > * **Without user authentication**, anyone who can reach Explorer can use the > InfluxDB connection configured in it, with that token’s permissions. > Treat reaching Explorer the same as holding the token: use tokens scoped to > the task, and put an authenticating reverse proxy with TLS in front of any > remote access. > * **With user authentication** (v3.12+), users sign in before they reach the > UI, and their[role](/influxdb3/enterprise/reference/internals/rbac/) determines what they > can do. Use TLS for remote access. > > Either way, bind the server to an interface you intend to expose. > To control which interface the server listens on, see[`--http-bind`](/influxdb3/enterprise/reference/config-options/#http-bind). > When browsers reach Explorer over HTTPS, also set[`--webui-cookie-secure`](/influxdb3/enterprise/reference/config-options/#webui-cookie-secure)so session cookies are never sent over HTTP. ## Quick start without authentication To try Explorer without setting up user authentication, start the server with `--without-auth`: ```bash influxdb3 serve --mode all,webui \ --webui-session-secret local-dev-only \ --without-auth ``` Explorer configures a working default connection automatically, so you can open the UI and start querying immediately. ## Quick start with user authentication When you start with[`--user-auth-type`](/influxdb3/enterprise/reference/config-options/#user-auth-type)set and the cluster has no users and no operator token yet, Explorer shows a setup page (“Set up InfluxDB 3 Explorer”) instead of the normal UI. If the cluster already has an operator token (for example, an existing deployment that used token authentication), the setup page isn’t offered and `influxdb3 manage init-admin` returns `409 operator token already configured`. Create the first admin user with the operator token instead: ```bash influxdb3 create user --username USERNAME --role Admin --token OPERATOR_TOKEN ``` * If password sign-in is available (`--jwt-key-id` and `--jwt-private-key`are set), the setup page lets you create the first admin user directly in the browser and shows the operator token once. Store it securely: it can’t be retrieved again. * If only OAuth sign-in is configured (`--user-auth-type oauth` without JWT keys), the setup page can’t take a password, so it directs you to create the first admin from the CLI instead: ``` influxdb3 manage init-admin --oauth-id OAUTH_SUBJECT ``` Then sign in through your identity provider. See[Bootstrap the initial admin](/influxdb3/enterprise/admin/security/manage-users/#bootstrap-the-initial-admin)for the equivalent CLI-only workflow. After you sign in, Explorer configures the default connection for you. For the setup page, signing in, and managing users in the Explorer UI, see the InfluxDB 3 Explorer documentation: * [Set up the first user](/influxdb3/explorer/authentication/set-up-first-user/) * [Authentication](/influxdb3/explorer/authentication/) * [Manage users with InfluxDB 3 Explorer](/influxdb3/explorer/manage-users/) ## Default connections and query routing Explorer always needs a default connection (a server URL and API token) to query data. How that connection gets configured depends on how you start the server: * **`--without-auth`**: Explorer configures a working default connection automatically, without a token. * **User authentication enabled** (`--user-auth-type basic` and/or `oauth`): Explorer configures the default connection for you after you sign in. * **`apiv3_` token authentication only** (the default: no `--without-auth`and no `--user-auth-type`): Explorer has no way to obtain a token automatically, so you provide a server URL and[token](/influxdb3/enterprise/admin/tokens/)yourself the first time you connect. The node that serves the Explorer UI doesn’t have to be the node that serves queries or writes: * Every node that serves the UI forwards Explorer’s default-connection requests over the internode protocol to a running node in the cluster, round-robin: a query-capable node for reads, and an ingest-capable node for writes. Only nodes that advertise an internode address (`--internode-bind-addr` or[`--conn-info`](/influxdb3/enterprise/reference/config-options/#conn-info)) are used, and the serving node counts as a candidate if it advertises one. * If no such node is available, a serving node that itself runs `query` or`ingest` mode (for example, a single `all,webui` node) handles the request locally. * Sign-in and user-management requests are always handled by the node that serves the UI. ## Configure SSO for the Explorer UI For browser-based single sign-on through the Explorer UI (as opposed to CLI OAuth login), set[`--webui-public-uri`](/influxdb3/enterprise/reference/config-options/#webui-public-uri)together with `--oauth-client-id`, `--oauth-issuer`, and `--oauth-audience`: ```bash influxdb3 serve --mode all,webui \ --webui-session-secret WEBUI_SESSION_SECRET \ --user-auth-type oauth \ --oauth-issuer https://my-idp.example.com/ \ --oauth-audience my-audience \ --oauth-client-id OAUTH_CLIENT_ID \ --webui-public-uri WEBUI_PUBLIC_URI ``` Replace `WEBUI_SESSION_SECRET`with your session secret,`WEBUI_PUBLIC_URI`with the browser-reachable base URL of the Explorer UI (for example,`https://explorer.example.com`), and`OAUTH_CLIENT_ID`with your identity provider’s OAuth client ID. InfluxDB 3 Enterprise derives the OAuth callback URL as`WEBUI_PUBLIC_URI/auth/callback`. Register that exact URL with your identity provider as an allowed redirect URI. See[Authenticate with OAuth/OIDC](/influxdb3/enterprise/admin/security/manage-users/#optional-authenticate-with-oauthoidc)for the other OAuth flags. ## Sessions Explorer stores sessions on the server and gives the browser a cookie that identifies the session.`--webui-session-secret` signs that cookie so the server can reject cookies that were altered or forged. The secret doesn’t encrypt the cookie. Keep the secret private: anyone who has it can create cookies the server accepts. Sessions are sliding and last 120 days: * A session that has 60 days or less remaining is renewed for another 120 days on its next authenticated request, so an actively used session doesn’t expire. * A session with no authenticated requests expires 120 days after it was created or last renewed. Changing `--webui-session-secret` invalidates every existing session and signs all users out. To manage the session secret: * **Generate the secret once and reuse it.**To generate a secret, run `openssl rand -base64 24`. * **Keep the secret out of your shell history and process list.**Set the secret through the `INFLUXDB3_WEBUI_SESSION_SECRET` environment variable instead of the command line when you can. * **Rotate the secret if it may have been exposed.**Rotating signs out every user. ## Explorer application data The integrated Explorer keeps its application state in a SQLite database that the server synchronizes to object storage for each cluster. You don’t mount a volume to persist it, which is the main operational difference from the[Explorer Docker container](/influxdb3/explorer/install/docker/#persist-data-across-restarts). ## Migrate data from the 3.11 UI If a browser has data from before you turned on user authentication (for example, from the integrated Explorer in InfluxDB 3.11, or from using Explorer on this deployment without authentication), Explorer offers to migrate that browser’s dashboards, saved queries, query history, and connections to your signed-in account the first time you sign in from that browser. You can accept or decline the offer. Either way, InfluxDB 3 Enterprise keeps the original data. Declining doesn’t delete anything, and you won’t be asked again from that browser. If you accept, connection credentials (API tokens and object-storage credentials) are cleared from the migrated connections, so you need to re-enter them. #### Related * [InfluxDB 3 Explorer documentation](/influxdb3/explorer/) * [Authentication in InfluxDB 3 Explorer](/influxdb3/explorer/authentication/) * [Manage users with InfluxDB 3 Explorer](/influxdb3/explorer/manage-users/) * [Manage users and authentication](/influxdb3/enterprise/admin/security/manage-users/) * [Role-based access control (RBAC)](/influxdb3/enterprise/reference/internals/rbac/) * [Manage tokens](/influxdb3/enterprise/admin/tokens/) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) --- # Manage file indexes Source: https://docs.influxdata.com/influxdb3/enterprise/admin/file-index/ InfluxDB 3 Enterprise lets you customize how your data is indexed to help optimize query performance for your specific workload, especially workloads that include single-series queries. Indexes help the InfluxDB query engine quickly identify the physical location of files that contain the queried data. By default, InfluxDB indexes on the primary key—`time` and tag columns. However, if your schema includes tags that you don’t specifically use when querying, you can define a custom indexing strategy to only index on `time` and columns important to your query workload. For example, if your schema includes the following columns: * country * state\_province * county * city * postal\_code And in your query workload, you only query based on country, state or province, and city, you can create a custom file indexing strategy that only indexes on`time` and these specific columns. This makes your index more efficient and improves the performance of your single-series queries. > [!Note] > File indexes can use any string column, including both tags and fields. * [Indexing life cycle](#indexing-life-cycle) * [Create a custom file index](#create-a-custom-file-index) * [Delete a custom file index](#delete-a-custom-file-index) ## Indexing life cycle InfluxDB 3 Enterprise builds indexes as it compacts data. Compaction is the process that organizes and optimizes Parquet files in storage and occurs in multiple phases or generations. Generation 1 (gen1) data is un-compacted and is not indexed. Generation 2 (gen2) data and beyond is all indexed. > [!Note] > In clusters that use the legacy Parquet storage engine, compaction doesn’t > automatically delete Gen1 Parquet files after it compacts their data into > later generations. > The Gen1 files remain in object storage and continue to consume storage space. > To delete them, see [Gen1 file cleanup](/influxdb3/enterprise/admin/gen1-file-cleanup/). ## [Create a custom file index](/influxdb3/enterprise/admin/file-index/create/) Use the [`influxdb3 create file_index` command](/influxdb3/enterprise/reference/cli/influxdb3/create/file_index/) to create a custom file indexing strategy for a database or a table. ```bash influxdb3 create file_index \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ --table wind_data \ country,city ``` ## [Delete a custom file index](/influxdb3/enterprise/admin/file-index/delete/) Use the [`influxdb3 delete file_index` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/file_index/) to delete a custom file indexing strategy from a database or a table and revert to the default indexing strategy. ```bash influxdb3 delete file_index \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ --table wind_data ``` [indexing](/influxdb3/enterprise/tags/indexing/) --- # Create a custom file index Source: https://docs.influxdata.com/influxdb3/enterprise/admin/file-index/create/ Use the [`influxdb3 create file_index` command](/influxdb3/enterprise/reference/cli/influxdb3/create/file_index/)to create a custom file indexing strategy for a database or table. Provide the following: * **Token** (`--token`): *(Required)* Your [admin token](/influxdb3/enterprise/admin/tokens/). You can also use the `INFLUXDB3_AUTH_TOKEN` environment variable to specify the token. * **Database** (`-d`, `--database`): *(Required)* The name of the database to apply the index to. You can also use the `INFLUXDB3_DATABASE_NAME`environment variable to specify the database. * **Table** (`-t`, `--table`): The name of the table to apply the index to. If no table is specified, the indexing strategy applies to all tables in the specified database. * **Columns**: *(Required)* A comma-separated list of string columns to index on. These are typically tag columns but can also be string fields. ```bash influxdb3 create file_index \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ COLUMNS ``` Replace the following placeholders with your values: * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) * `DATABASE_NAME`: the name of the database to create the file index in * `TABLE_NAME`: the name of the table to create the file index in * `COLUMNS`: a comma-separated list of columns to index on–for example: `host,application` #### Related * [influxdb3 create file\_index](/influxdb3/enterprise/reference/cli/influxdb3/create/file_index/) [indexing](/influxdb3/enterprise/tags/indexing/) --- # Delete a custom file index Source: https://docs.influxdata.com/influxdb3/enterprise/admin/file-index/delete/ Use the [`influxdb3 delete file_index` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/file_index/)to delete a custom file indexing strategy from a database or a table and revert to the default indexing strategy. Provide the following: * **Token** (`--token`): *(Required)* Your [admin token](/influxdb3/enterprise/admin/tokens/). You can also use the `INFLUXDB3_AUTH_TOKEN` environment variable to specify the token. * **Database** (`-d`, `--database`): *(Required)* The name of the database to apply remove the custom index from. You can also use the `INFLUXDB3_DATABASE_NAME`environment variable to specify the database. * **Table** (`-t`, `--table`): The name of the table to remove the custom index from. If no table is specified, the custom indexing strategy is removed from all tables in the specified database. ```bash influxdb3 delete file_index \ --host http://localhost:8585 \ --database DATABASE_NAME \ --table TABLE_NAME \ ``` Replace the following placeholders with your values: * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) * `DATABASE_NAME`: the name of the database to remove the custom file index from * `TABLE_NAME`: the name of the table to remove the custom file index from #### Related * [influxdb3 delete file\_index](/influxdb3/enterprise/reference/cli/influxdb3/delete/file_index/) [indexing](/influxdb3/enterprise/tags/indexing/) --- # Gen1 file cleanup Source: https://docs.influxdata.com/influxdb3/enterprise/admin/gen1-file-cleanup/ In InfluxDB 3 Enterprise clusters that use the legacy Parquet storage engine, compaction doesn’t delete Gen1 Parquet files after it merges their data into later generations. The Gen1 files remain in object storage until you delete them with the Gen1 cleanup endpoint.[`--compaction-cleanup-wait`](/influxdb3/enterprise/reference/config-options/#compaction-cleanup-wait)doesn’t apply to Gen1 files, and restarting a node doesn’t delete them. Queries read the compacted generations, so cleanup reclaims storage without changing query results. The Gen1 cleanup endpoint is available in InfluxDB 3 Enterprise 3.9.0 and later. Clusters that use the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)don’t need it. * [Run Gen1 cleanup](#run-gen1-cleanup) * [Which files cleanup deletes](#which-files-cleanup-deletes) * [Inspect Gen1 and compacted files](#inspect-gen1-and-compacted-files) ## Run Gen1 cleanup Send a `POST` request to the `/api/v3/configure/gen1_cleanup` endpoint of each ingest node. Each request cleans up the files that node persisted, across all databases and tables. A node that only runs the compactor doesn’t perform cleanup. ```bash curl --request POST \ "INGEST_NODE_URL/api/v3/configure/gen1_cleanup?min_age=24h" \ --header "Authorization: Bearer AUTH_TOKEN" ``` Replace the following: * `INGEST_NODE_URL`: the URL of an ingest node–for example, `http://localhost:8181` * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) The endpoint accepts the following optional query parameters: | Parameter |Default| Description | |-------------|-------|-------------------------------------------------------------------------------------------------------| | `min_age` | `24h` |Skip snapshots persisted more recently than this duration. This isn’t a retention period for your data.| |`batch_size` | `500` | Maximum number of snapshots to process in one request. | |`concurrency`| `10` | Maximum number of concurrent file deletions. | The endpoint responds with `202 Accepted` and either `Gen1 cleanup started` or`Gen1 cleanup already running`. The response confirms that cleanup started, not that it finished. Cleanup runs in the background, one run at a time per node. When it finishes, it logs `Gen1 cleanup finished` with the number of files and snapshots it deleted, or `Gen1 cleanup: no eligible files found`. To work through a large backlog, send the request again after each run finishes. To keep Gen1 storage from growing, send the request on a schedule. ## Which files cleanup deletes Cleanup deletes only Gen1 files whose data the compactor has already merged into later generations. It reads the node’s snapshots to find these files and deletes a snapshot after it deletes all of the snapshot’s files. Cleanup keeps the following: * Gen1 files the compactor hasn’t compacted yet. * Files from the most recent snapshot the compactor has processed, and from any newer snapshots. In a cluster that has taken only one snapshot, cleanup deletes nothing until the node takes a second snapshot and the compactor compacts it. * Snapshots persisted more recently than `min_age`. > [!Warning] > Don’t delete Gen1 files or snapshots from object storage yourself. > Cleanup uses the snapshots to find the files it can safely delete. ## Inspect Gen1 and compacted files The `system.parquet_files` table lists the Gen1 files a node tracks, including compacted Gen1 files that cleanup hasn’t deleted yet. It doesn’t list compacted generations. To list the compacted files for a table, query the `system.compacted_data`table. Queries to this table must filter on `table_name`. ```bash influxdb3 query \ --database DATABASE_NAME \ --token AUTH_TOKEN \ "SELECT generation_level, count(*) AS files, sum(parquet_size_bytes) AS bytes, sum(parquet_row_count) AS rows FROM system.compacted_data WHERE table_name = 'TABLE_NAME' GROUP BY generation_level ORDER BY generation_level" ``` Replace the following: * `DATABASE_NAME`: the name of the database that contains the table * `TABLE_NAME`: the name of the table * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) #### Related * [Manage file indexes](/influxdb3/enterprise/admin/file-index/) * [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) [compaction](/influxdb3/enterprise/tags/compaction/)[storage](/influxdb3/enterprise/tags/storage/)[maintenance](/influxdb3/enterprise/tags/maintenance/) | Parameter | Default | Description | | --- | --- | --- | | Parameter | Default | Description | | min_age | 24h | Skip snapshots persisted more recently than this duration. This isn’t a retention period for your data. | | batch_size | 500 | Maximum number of snapshots to process in one request. | | concurrency | 10 | Maximum number of concurrent file deletions. | --- # Identify InfluxDB 3 Enterprise version Source: https://docs.influxdata.com/influxdb3/enterprise/admin/identify-version/ Identifying which InfluxDB product and version you’re using is essential for accessing the correct documentation, troubleshooting issues, and planning upgrades. > [!Note] > #### Identify your InfluxDB version > > If you are unsure which InfluxDB product you are using, use our interactive version detector to help identify it: > > [Detect my InfluxDB version](#) > *(Interactive feature in HTML: Opens version detector modal)* ## Quick detection methods ### InfluxDB 3 version detection Check the version using the `influxdb3` command: ```bash influxdb3 --version ``` Send a `GET` request to the `/ping` endpoint to examine HTTP response headers–for example: ```bash curl -i http://localhost:8181/ping ``` Use `GET`. A `HEAD` request to `/ping` returns `404`. The response includes version information in the **headers** and **body**: * **Headers**: * `x-influxdb-version`: Version number (for example, 3.12.2) * `x-influxdb-build`:`Enterprise` * **JSON body**: Contains `version`, `revision`, and `process_id` Example response: ``` HTTP/1.1 200 OK x-influxdb-build: Enterprise x-influxdb-version: 3.12.2 {"version":"3.12.2","revision":"abc123def","process_id":"..."} ``` > [!Note] > The [`/ping` endpoint](/influxdb3/enterprise/api/server-information/#operation/GetPing) requires authentication by default in InfluxDB 3 Enterprise. > Include an [admin token](/influxdb3/enterprise/admin/tokens/admin/) or a[system token](/influxdb3/enterprise/admin/tokens/resource/create/#create-a-system-token)(with `system:ping:read` permission) using [`Bearer` authentication](/influxdb3/enterprise/api/authentication/#section/Authentication/BearerAuthentication)in the `Authorization` header. > [!Note] > #### SQL version() function > > The SQL `version()` function returns the[DataFusion](https://datafusion.apache.org/) query engine version, not the > InfluxDB product version. Use the methods above to identify your InfluxDB > version. ## Understanding InfluxDB products For a complete comparison of InfluxDB versions and deployment options, see the [InfluxDB platform overview](/platform/). ## Troubleshooting ### Can’t access your InfluxDB instance? If you can’t directly access your InfluxDB instance: 1. Use the [interactive version detector](#identify-your-influxdb-version) above or [Ask InfluxData AI](#) 2. Answer questions about your setup: * License type (paid or free) * Hosting model (cloud or self-hosted) * Server age * Query language preferences 3. Get product recommendations based on your answers ### Still not sure? If you’re still uncertain, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/) * [Administer InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/) * [influxdb3 CLI](/influxdb3/enterprise/reference/cli/influxdb3/) --- # Import data Source: https://docs.influxdata.com/influxdb3/enterprise/admin/import-data/ Bulk import is an InfluxDB 3 Enterprise feature that requires the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters on InfluxDB 3 Enterprise 3.11+. On clusters that started on 3.10 or earlier, first run the[storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree)(`--upgrade-pacha-tree`). Use bulk import to load your existing Parquet files into a database and table. InfluxDB 3 Enterprise writes the imported data to your object storage. The target database and table must already exist before you import your data into them. Bring your Parquet files in with either of the following commands: * [`influxdb3 import upload`](/influxdb3/enterprise/reference/cli/influxdb3/import/upload/): Upload files from a local path, or from an object store URL that your machine can reach. File bytes stream through the `influxdb3` client. * [`influxdb3 import from-object-store`](/influxdb3/enterprise/reference/cli/influxdb3/import/from-object-store/): Import files that already sit in InfluxDB 3 Enterprise’s own object store, or in an S3 bucket. InfluxDB 3 Enterprise reads the files itself, using its own object store credentials, without streaming file bytes through the client. ## How bulk import works Bulk import reads your generic Parquet files, maps their columns to InfluxDB 3 Enterprise types, and writes the resulting rows into an existing table. Each file becomes a separate import job. InfluxDB 3 Enterprise stores the imported data in your object storage and compacts it automatically. Rows become queryable after the compactor processes them, not immediately after the upload completes. > [!Note] > Because imported data is queryable only after compaction, expect a delay before > imported rows appear in query results after an import command returns. ## Permissions The token you use for an import command needs the following permissions: | Command | Required action | Example permission | |------------------------------------|-----------------------------------|---------------------------| | `influxdb3 import upload` | `write` on the target database | `db:DATABASE_NAME:write` | |`influxdb3 import from-object-store`| `write` on the target database | `db:DATABASE_NAME:write` | | `influxdb3 import list` |`describe` on at least one database|`db:DATABASE_NAME:describe`| An import doesn’t create the database, so the token doesn’t need the `create`action. If the token lacks the `write` action, the import fails with HTTP status `403`and the following message: ```text Not authorized to import into database ``` You get this error even when the database doesn’t exist. `influxdb3 import list` returns only the import jobs for databases your token can describe. If your token can’t describe any database, the command returns an HTTP 403 error. For more on creating tokens with these permissions, see[Create a resource token](/influxdb3/enterprise/admin/tokens/resource/create/). ## Upload Parquet files from your machine Use the `influxdb3 import upload` command to upload one or more Parquet files. For the complete command syntax and flags, see the[`influxdb3 import upload`](/influxdb3/enterprise/reference/cli/influxdb3/import/upload/)CLI reference. You can pass either of the following as the import source: * A single Parquet file. * A directory, which InfluxDB 3 Enterprise processes recursively for `*.parquet`files. InfluxDB 3 Enterprise creates one import job per file. * An object store URL, such as `s3://bucket/prefix`.`influxdb3` reads credentials from the environment; use `--source-opt` to set or override store options. ## Import Parquet files from object storage Use the `influxdb3 import from-object-store` command to import every Parquet file found recursively under a prefix directly on the server, without streaming file bytes through the `influxdb3` client. For the complete command syntax and flags, see the[`influxdb3 import from-object-store`](/influxdb3/enterprise/reference/cli/influxdb3/import/from-object-store/)CLI reference. The source you pass is either of the following: * A prefix in InfluxDB 3 Enterprise’s own object store, for example`some/prefix`. * An `s3://bucket/prefix` URL for an external S3 bucket. InfluxDB 3 Enterprise rejects any other URL scheme with an HTTP 400 error. > [!Note] > You can’t use your object store’s root, or the reserved`uploaded_imports/` prefix, as an import source. > Choose a dedicated prefix for your source files. ### How the server reads the source InfluxDB 3 Enterprise uses its own configured object store credentials to read the source. You don’t send credentials in the command or the request. * When the source is a prefix in InfluxDB 3 Enterprise’s own object store, staging each file is a copy within that same store. * When the source is an external S3 bucket, InfluxDB 3 Enterprise’s own object store must also be S3 (or an S3-compatible endpoint set with`--aws-endpoint`). If it isn’t, InfluxDB 3 Enterprise has no credentials to reach another bucket and rejects the request with an HTTP 400 error. InfluxDB 3 Enterprise’s credentials must also be able to read the source bucket. This is typically automatic within one AWS account, and otherwise possible only when a bucket policy on the source bucket grants that access. For an external S3 bucket, InfluxDB 3 Enterprise first tries a true S3 server-side copy: a single request that moves the file directly between buckets without streaming its bytes through InfluxDB 3 Enterprise. Server-side copy applies only when all of the following hold: * Both the source and destination object stores are S3 (or an S3-compatible endpoint set with `--aws-endpoint`). * InfluxDB 3 Enterprise’s credentials can read the source bucket and write the destination bucket. * The source file is smaller than 5 GiB. When any of these doesn’t hold, InfluxDB 3 Enterprise falls back automatically to streaming the file through itself (never through the `influxdb3`client) and logs a warning. Because InfluxDB 3 Enterprise reads directly from your object store instead of your machine re-uploading the bytes, importing from a prefix or bucket that the server can reach directly avoids the round trip through your machine that`influxdb3 import upload` requires. Control the server-side copy attempt with the[`--import-attempt-server-side-copy`](/influxdb3/enterprise/reference/config-options/#import-attempt-server-side-copy)server option (default `true`). Set it to `false` to always use the streamed fallback. ## Review import jobs Use the `influxdb3 import list` command to review import jobs. For details, see the[`influxdb3 import list`](/influxdb3/enterprise/reference/cli/influxdb3/import/list/)CLI reference. ## Map columns to InfluxDB types Use `--column` flags to map Parquet columns to InfluxDB 3 Enterprise types when running `influxdb3 import upload` or `influxdb3 import from-object-store`. The following types are supported: | Type | Description | |--------|-----------------------------| | `i64` | Signed 64-bit integer field | | `u64` |Unsigned 64-bit integer field| | `f64` | 64-bit float field | | `bool` | Boolean field | |`string`| String field | | `time` | Timestamp | | `tag` | Tag | Any Parquet column that you don’t map with a `--column` flag is imported as a field, typed from its Parquet type. This is true even when the target table already declares the column as a tag. Map every string tag column, even when the table already declares it. Without the mapping, the import fails with the following error: ```text invalid column type for column 'host', expected iox::column_type::tag, got iox::column_type::field::string ``` For example, to import `host` as a tag, add `--column host=tag`. ## Import into an explicit schema database In a database that uses[`explicit` schema mode](/influxdb3/enterprise/admin/databases/enforce-schema/), every column in the Parquet file must already be declared on the table. If the file has an undeclared column, the import fails and InfluxDB 3 Enterprise creates no import job. This applies to both `influxdb3 import upload` and`influxdb3 import from-object-store`. Declare the columns first with[`influxdb3 update table`](/influxdb3/enterprise/reference/cli/influxdb3/update/table/)or a `PATCH` request to `/api/v3/configure/table`. For more information, see[Add columns to a table](/influxdb3/enterprise/admin/tables/update/). In InfluxDB 3 Enterprise 3.12, the rejection returns HTTP status `500` with an error message like the following: ```text Could not modify catalog: column 'usage' (iox::column_type::field::float) is not defined in table 'cpu' of database 'DATABASE_NAME', which uses explicit schemas; add the column with the /api/v3/configure/table API before writing to it ``` ## Troubleshoot imports ### The table doesn’t exist Importing into a table that doesn’t exist returns HTTP status `404` with the following message, in any schema mode: ```text Table TABLE_NAME does not exist ``` Create the table first. ### Invalid column type for a tag column The import fails with the following error: ```text invalid column type for column 'host', expected iox::column_type::tag, got iox::column_type::field::string ``` This error means that you didn’t map a string tag column. Add `--column COLUMN_NAME=tag` for each tag column. See [Map columns to InfluxDB types](#map-columns-to-influxdb-types). ### Unable to infer data type for a column The import fails with HTTP status `400` and the following error: ```text Unable to infer data type for column 'host' based on input parquet file ``` This error means that a string column needs a mapping. Add a `--column` flag for it. ### Import list shows 0 for the minimum and maximum timestamps `influxdb3 import list` can show `min_timestamp_ns` and `max_timestamp_ns` as`0`. InfluxDB 3 Enterprise takes the range from the time column’s Parquet statistics. When the file has no statistics that the server can read, the fields are `0`. A `0` doesn’t mean the import failed, and the imported data isn’t affected. #### Related * [influxdb3 import](/influxdb3/enterprise/reference/cli/influxdb3/import/) | Command | Required action | Example permission | | --- | --- | --- | | Command | Required action | Example permission | | influxdb3 import upload | write on the target database | db:DATABASE_NAME:write | | influxdb3 import from-object-store | write on the target database | db:DATABASE_NAME:write | | influxdb3 import list | describe on at least one database | db:DATABASE_NAME:describe | | Type | Description | | --- | --- | | Type | Description | | i64 | Signed 64-bit integer field | | u64 | Unsigned 64-bit integer field | | f64 | 64-bit float field | | bool | Boolean field | | string | String field | | time | Timestamp | | tag | Tag | --- # Manage the Last Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/last-value-cache/ The InfluxDB 3 Enterprise Last Value Cache (LVC) lets you cache the most recent values for specific fields in a table, improving the performance of queries that return the most recent value of a field for specific series or the last N values of a field. The LVC is an in-memory cache that stores the last N number of values for specific fields of series in a table. When you create an LVC, you can specify what fields to cache, what tags to use to identify each series, and the number of values to cache for each unique series. An LVC is associated with a table, which can have multiple LVCs. * [Create a Last Value Cache](#create-a-last-value-cache) * [Query a Last Value Cache](#query-a-last-value-cache) * [Show information about Last Value Caches](#show-information-about-last-value-caches) * [Delete a Last Value Cache](#delete-a-last-value-cache) * [Important things to know about the Last Value Cache](#important-things-to-know-about-the-last-value-cache) * [High cardinality key columns](#high-cardinality-key-columns) * [Value count](#value-count) * [Defining value columns](#defining-value-columns) Consider a dataset with the following schema (similar to the[home sensor sample dataset](/influxdb3/enterprise/reference/sample-data/#home-sensor-data)): * home (table) * tags: * room * kitchen * living room * wall * north * east * south * fields: * co (integer) * temp (float) * hum (float) If you cache the last value for each field per room and wall, the LVC looks similar to this: | room |wall |co |hum |temp| time | |-----------|-----|---|----|----|--------------------| | Kitchen |east |26 |36.5|22.7|2022-01-01T20:00:00Z| |Living Room|north|17 |36.4|22.2|2022-01-01T20:00:00Z| |Living Room|south|16 |36.3|22.1|2022-01-01T20:00:00Z| If you cache the last four values of each field per room and wall, the LVC looks similar to the following: | room |wall |co |hum |temp| time | |-----------|-----|---|----|----|--------------------| | Kitchen |east |26 |36.5|22.7|2022-01-01T20:00:00Z| | Kitchen |east | 9 |36.0|22.7|2022-01-01T17:00:00Z| | Kitchen |east | 3 |36.2|22.7|2022-01-01T15:00:00Z| | Kitchen |east | 0 |36.1|22.7|2022-01-01T10:00:00Z| |Living Room|north|17 |36.4|22.2|2022-01-01T20:00:00Z| |Living Room|north| 5 |35.9|22.6|2022-01-01T17:00:00Z| |Living Room|north| 1 |36.1|22.3|2022-01-01T15:00:00Z| |Living Room|north| 0 |36.0|21.8|2022-01-01T10:00:00Z| |Living Room|south|16 |36.3|22.1|2022-01-01T20:00:00Z| |Living Room|south| 4 |35.8|22.5|2022-01-01T17:00:00Z| |Living Room|south| 0 |36.0|22.3|2022-01-01T15:00:00Z| |Living Room|south| 0 |35.9|21.8|2022-01-01T10:00:00Z| > [!Note] > #### Null column values > > *Null* column values are still considered values and are cached in the LVC. > If you write data to a table and don’t provide a value for an existing column, > the column value is cached as *null*. ## [Create a Last Value Cache](/influxdb3/enterprise/admin/last-value-cache/create/) Use the [`influxdb3 create last_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/create/last_cache/)to create a Last Value Cache. ```bash influxdb3 create last_cache \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ --table home \ --node-spec "nodes:node-01,node-02" \ --key-columns room,wall \ --value-columns temp,hum,co \ --count 5 \ --ttl 30mins \ homeLastCache ``` ## [Query a Last Value Cache](/influxdb3/enterprise/admin/last-value-cache/query/) Use the [`last_cache()` SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#last_cache)in the `FROM` clause of an SQL `SELECT` statement to query data from the Last Value Cache. ```sql SELECT * FROM last_cache('table-name', 'cache-name') ``` > [!Important] > You must use SQL to query the LVC. > InfluxQL does not support the `last_cache()` function. ## [Show information about Last Value Caches](/influxdb3/enterprise/admin/last-value-cache/show/) Use the `influxdb3 show system table` command to query and output Last Value Cache information from the `last_caches` system table. ```bash influxdb3 show system \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ table last_caches ``` ## [Delete a Last Value Cache](/influxdb3/enterprise/admin/last-value-cache/delete/) Use the [`influxdb3 delete last_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/last_cache/)to delete a Last Value Cache. ```bash influxdb3 delete last_cache \ --database example-db \ --token 00xoXX0xXXx0000XxxxXx0Xx0xx0 \ --table home \ homeLastCache ``` ## Important things to know about the Last Value Cache LVCs are stored in memory; the larger the cache, the more memory your InfluxDB 3 node requires to maintain it. Consider the following: * [Cache data loading](#cache-data-loading) * [High cardinality key columns](#high-cardinality-key-columns) * [Value count](#value-count) ## Cache data loading On cache creation, InfluxDB 3 Enterprise loads historical data into the cache. On restart, the server automatically reloads cache data. In a multi-node cluster, only nodes included in the cache’s[node specification](/influxdb3/enterprise/admin/last-value-cache/create/)load historical data–at creation and on restart. Excluded nodes still serve the cache and add newly written data to it, but until new data arrives, queries served by those nodes return only values written after the cache was created (or after the node restarted). Only specify a node list to reduce the initial cache warm-up load, and only if you accept temporarily incomplete query results from excluded nodes. ### High cardinality key columns “Cardinality” refers to the number of unique key column combinations in your cached data. While the InfluxDB 3 storage engine is not limited by cardinality, it does affect the LVC. Higher cardinality increases memory requirements for storing the LVC and can affect LVC query performance. We recommend the following: * Only use tags important to your query workload as key columns in the LVC. Caching unnecessary tags or fields as key columns results in higher cardinality without any benefit. * Avoid including high-cardinality key columns in your LVC. * Don’t include multiple high-cardinality key columns in your LVC. To estimate total key column cardinality in an LVC, use the following equation: ```txt num_uniq_col_val_N [× num_uniq_col_val_N …] = key_column_cardinality ``` ### Value count By increasing the number of values to store in the LVC, you increase the number of rows stored in the cache and the amount of memory required to store them. Be judicious with the number of values to store. This count is per unique key column combination. If you include two tags as key columns, one with three unique values and the other with 10, you could have up to 30 unique key column combinations. If you want to keep the last 10 values, you could potentially have 300+ rows in the cache. To get an idea of the number of rows required to cache the specified number of values, use the following equation: ```txt key_column_cardinality × count = number_of_rows ``` ### Defining value columns When creating an LVC, if you include the `--value-columns` options to specify which fields to cache as value columns, any new fields added in the future will not be added to the cache. However, if you omit the `--value-columns` option, all columns other than those specified as `--key-columns` are cached as value columns, including columns that are added later. #### Related * [last\_cache SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#last_cache) * [Manage Last Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/last-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) | room | wall | co | hum | temp | time | | --- | --- | --- | --- | --- | --- | | room | wall | co | hum | temp | time | | Kitchen | east | 26 | 36.5 | 22.7 | 2022-01-01T20:00:00Z | | Living Room | north | 17 | 36.4 | 22.2 | 2022-01-01T20:00:00Z | | Living Room | south | 16 | 36.3 | 22.1 | 2022-01-01T20:00:00Z | | room | wall | co | hum | temp | time | | --- | --- | --- | --- | --- | --- | | room | wall | co | hum | temp | time | | Kitchen | east | 26 | 36.5 | 22.7 | 2022-01-01T20:00:00Z | | Kitchen | east | 9 | 36.0 | 22.7 | 2022-01-01T17:00:00Z | | Kitchen | east | 3 | 36.2 | 22.7 | 2022-01-01T15:00:00Z | | Kitchen | east | 0 | 36.1 | 22.7 | 2022-01-01T10:00:00Z | | Living Room | north | 17 | 36.4 | 22.2 | 2022-01-01T20:00:00Z | | Living Room | north | 5 | 35.9 | 22.6 | 2022-01-01T17:00:00Z | | Living Room | north | 1 | 36.1 | 22.3 | 2022-01-01T15:00:00Z | | Living Room | north | 0 | 36.0 | 21.8 | 2022-01-01T10:00:00Z | | Living Room | south | 16 | 36.3 | 22.1 | 2022-01-01T20:00:00Z | | Living Room | south | 4 | 35.8 | 22.5 | 2022-01-01T17:00:00Z | | Living Room | south | 0 | 36.0 | 22.3 | 2022-01-01T15:00:00Z | | Living Room | south | 0 | 35.9 | 21.8 | 2022-01-01T10:00:00Z | --- # Create a Last Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/last-value-cache/create/ Use the [`influxdb3 create last_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/create/last_cache/)to create a Last Value Cache (LVC). Provide the following: * **Database** (`-d`, `--database`): *(Required)* The name of the database to associate the LVC with. You can also use the `INFLUXDB3_DATABASE_NAME`environment variable to specify the database. * **Token** (`--token`): *(Required)* Your InfluxDB 3 Enterprise admin authentication token. You can also use the `INFLUXDB3_AUTH_TOKEN` environment variable to specify the token. * **Table** (`-t`, `--table`): *(Required)* The name of the table to associate the LVC with. * **Node specification** (`-n`, `--node-spec`): Specify which nodes load historical data into the LVC. The default (`all`) loads historical data on all query- and processing-capable nodes. To reduce the initial cache warm-up load, use `nodes:[,...]` to limit the historical data load to specific nodes. All query-capable nodes serve the LVC and add newly written data to it, but excluded nodes return incomplete query results until new data arrives. Only specify a node list if you accept temporarily incomplete query results from excluded nodes. * **Key columns** (`--key-columns`): Specify which columns to include in the primary key of the cache. Rows in the LVC are uniquely identified by their timestamp and key columns, so include all the columns you need to identify each row. These are typically tags, but you can use any columns with the following types: * String * Integer * Unsigned integer * Boolean * **Value columns** (`--value-columns`): Specify which columns to cache as value columns. These are typically fields but can also be tags. By default, `time` and columns other than those specified as `--key-columns` are cached as value columns. * **Count** (`--count`): The number of values to cache per unique key column combination. The supported range is `[1-10]`. The default count is `1`. * **Time-to-Live (TTL)** (`--ttl`): The time-to-live for cached values in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)form. The default TTL is four hours. * **Cache name**: A unique name for the cache. If you don’t provide one, InfluxDB automatically generates a cache name for you. ```bash influxdb3 create last_cache \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ --node-spec "nodes:NODE_LIST" \ --key-columns KEY_COLUMNS \ --value-columns VALUE_COLUMNS \ --count COUNT \ --ttl TTL\ LVC_NAME ``` ## Use the HTTP API To use the HTTP API to create a Last Value Cache, send a `POST` request to the `/api/v3/configure/last_cache` endpoint. ``` POST /api/v3/configure/last_cache ``` ```bash curl -X POST "https://localhost:8181/api/v3/configure/last_cache" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "name": "LVC_NAME", "key_columns": ["KEY_COLUMNS"], "value_columns": ["VALUE_COLUMNS"], "count": COUNT, "ttl": TTL }' ``` ### Example ``` curl -X POST "https://localhost:8181/api/v3/configure/last_cache" \ --header "Authorization: Bearer 00xoXX0xXXx0000XxxxXx0Xx0xx0" \ --json '{ "db": "example-db", "table": "home", "name": "homeLastCache", "key_columns": ["room", "wall"], "value_columns": ["temp", "hum", "co"], "count": 5, "ttl": 14400 }' ``` **Response codes:** * `201` : Success. Last cache created. * `400` : Bad request. * `401` : Unauthorized. * `404` : Cache not found. * `409` : Cache already exists. > [!Note] > #### API parameter differences > > Column format: The API uses JSON arrays ([“room”, “wall”]) instead of the CLI’s comma-delimited format (room,wall). > TTL format: The API uses seconds (14400) instead of the CLI’s humantime format (4h, 4 hours). Replace the following: * `DATABASE_NAME`: the name of the database to associate the LVC with * `AUTH_TOKEN`: your InfluxDB 3 Enterprise admin authentication token * `TABLE_NAME`: the name of the table to associate the LVC with * `NODE_LIST`: a comma-delimited list of node IDs to configure the LVC on–for example:`node-01,node-02`. * `KEY_COLUMNS`: a comma-delimited list of columns to use to unique identify each series–for example: `room,wall` * `VALUE_COLUMNS`: a comma-delimited list of columns to cache as value columns–for example: `temp,hum,co` * `COUNT`: the number of last values to cache per series–for example: `5` * `TTL`: the TTL of cached values in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)form–for example: `10s`, `1min 30sec`, `3 hours` * `LVC_NAME`: a unique name for the LVC The cache imports the distinct values from the table and starts caching them. > [!Important] > #### LVC size and persistence > > The LVC is stored in memory, so it’s important to consider the size and persistence > of the cache. For more information, see[Important things to know about the Last Value Cache.](/influxdb3/enterprise/admin/last-value-cache/#important-things-to-know-about-the-last-value-cache) #### Related * [influxdb3 create last\_cache](/influxdb3/enterprise/reference/cli/influxdb3/create/last_cache/) * [Manage Last Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/last-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) --- # Delete a Last Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/last-value-cache/delete/ Use the [`influxdb3 delete last_cache` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/last_cache/)to delete a Last Value Cache (LVC). Provide the following: * **Database** (`-d`, `--database`): *(Required)* The name of the database that the LVC you want to delete is associated with. You can also use the`INFLUXDB3_DATABASE_NAME` environment variable to specify the database. * **Token** (`--token`): *(Required)* Your InfluxDB 3 Enterprise admin authentication token. You can also use the `INFLUXDB3_AUTH_TOKEN` environment variable to specify the database. * **Table** (`-t`, `--table`): *(Required)* The name of the table that the LVC you want to delete is associated with. * **Cache name**: The name of the LVC to delete. ```bash influxdb3 delete last_cache \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ LVC_NAME ``` ## Use the HTTP API To use the HTTP API to delete a Last Value Cache, send a `DELETE` request to the `/api/v3/configure/last_cache` endpoint with query parameters. ``` DELETE /api/v3/configure/last_cache ``` ```bash curl -X DELETE "https://localhost:8181/api/v3/configure/last_cache?db=DATABASE_NAME&table=TABLE_NAME&name=LVC_NAME" \ --header "Authorization: Bearer AUTH_TOKEN" ``` ## Example ```bash curl -X DELETE "https://localhost:8181/api/v3/configure/last_cache?db=example-db&table=home&name=homeLastCache" \ --header "Authorization: Bearer 00xoXX0xXXx0000XxxxXx0Xx0xx0" ``` **Response codes:** * `200` : Success. The last cache has been deleted. * `400` : Bad request. * `401` : Unauthorized. * `404` : Cache not found. Replace the following: * `DATABASE_NAME`: the name of the database that the LVC you want to delete is associated with * `AUTH_TOKEN`: your InfluxDB 3 Enterprise admin authentication token * `TABLE_NAME`: the name of the table that the LVC you want to delete is associated with * `LVC`: the name of the LVC to delete > [!Caution] > This is a destructive action that cannot be undone. Once deleted, any queries > against the deleted LVC will return an error. #### Related * [influxdb3 delete last\_cache](/influxdb3/enterprise/reference/cli/influxdb3/delete/last_cache/) * [Manage Last Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/last-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) --- # Query a Last Value Cache Source: https://docs.influxdata.com/influxdb3/enterprise/admin/last-value-cache/query/ Use the [`last_cache()` SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#last_cache)in the `FROM` clause of an SQL `SELECT` statement to query data from the Last Value Cache (LVC). > [!Important] > You must use SQL to query the LVC. > InfluxQL does not support the `last_cache()` function. `last_cache()` supports the following arguments: * **table\_name**: *(Required)* The name of the table the LVC is associated with formatted as a string literal. * **cache\_name**: The name of the LVC to query formatted as a string literal. This argument is only required if there is more than one LVC associated with the specified table. ```sql SELECT * FROM last_cache('table_name', 'cache_name') ``` You can use other [SQL clauses](/influxdb3/enterprise/reference/sql/#statements-and-clauses)to modify query results. For example, you can use the `WHERE` clause to return the last value for a specific tag set: ```sql SELECT room, temp FROM last_cache('home', 'homeCache') WHERE room = 'Kitchen' ``` #### Related * [last\_cache SQL function](/influxdb3/enterprise/reference/sql/functions/cache/#last_cache) * [Manage Last Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/last-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) --- # Show information about Last Value Caches Source: https://docs.influxdata.com/influxdb3/enterprise/admin/last-value-cache/show/ Use the [`influxdb3 show system table` command](/influxdb3/enterprise/reference/cli/influxdb3/show/system/table/)to query and output Last Value Cache information from the `last_caches` system table. ```bash influxdb3 show system \ --database DATABASE_NAME \ --token AUTH_TOKEN \ table last_caches ``` This returns a table similar to the following: | table | name |key\_column\_ids|key\_column\_names|value\_column\_ids| value\_column\_names |count| ttl | |-------|-------------|----------------|------------------|------------------|----------------------------------------------------|-----|-----| |weather|weather\_last| [0] | [location] | [2, 3, 4, 5, 1] |[precip, temp\_avg, temp\_max, temp\_min, wind\_avg]| 1 |86400| |bitcoin|bitcoin\_last| [0, 1] | [code, crypto] | [4] | [price] | 1 |14400| |numbers|numbers\_last| [] | [] | [0, 1] | [a, b] | 5 |14400| | home | home\_last | [0] | [room] | [1, 2, 3] | [temp, hum, co] | 5 | 60 | ## Query specific columns from the last\_caches system table Use the `--select` option to query specific columns from the `last_caches`system table. Provide a comma-delimited list of columns to return: ```bash influxdb3 show system \ --database DATABASE_NAME \ --token AUTH_TOKEN \ table last_caches \ --select name,key_column_names,value_column_names ``` ## Sort last\_caches system table output Use the `--order-by` option to sort data from the `last_caches` system table by specific columns. Provide a comma-delimited list of columns to sort by: ```bash influxdb3 show system \ --database DATABASE_NAME \ --token AUTH_TOKEN \ table last_caches \ --order-by table,ttl ``` > [!Note] > Results are sorted in ascending order based on the provided columns. In the examples above, replace the following: * `DATABASE_NAME`: the name of the database to query system data from * `AUTH_TOKEN`: your InfluxDB 3 Enterprise admin authentication token ## Use the HTTP API To use the HTTP API to query and output cache information from the system table, send a `GET` or `POST` request to the `/api/v3/query_sql` endpoint. ``` GET /api/v3/query_sql ``` ``` POST /api/v3/query_sql ``` ### Query all last value caches ```bash curl -X POST "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "q": "SELECT * FROM system.last_caches", "format": "json" }' ``` ## Query specific cache details ```bash curl -X POST "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "DATABASE_NAME", "q": "SELECT * FROM system.last_caches WHERE name = '\''CACHE_NAME'\''", "format": "json" }' ``` #### Related * [Manage Last Value Caches with InfluxDB 3 Explorer](/influxdb3/explorer/manage-caches/last-value-caches/) [cache](/influxdb3/enterprise/tags/cache/) | table | name | key_column_ids | key_column_names | value_column_ids | value_column_names | count | ttl | | --- | --- | --- | --- | --- | --- | --- | --- | | table | name | key_column_ids | key_column_names | value_column_ids | value_column_names | count | ttl | | weather | weather_last | [0] | [location] | [2, 3, 4, 5, 1] | [precip, temp_avg, temp_max, temp_min, wind_avg] | 1 | 86400 | | bitcoin | bitcoin_last | [0, 1] | [code, crypto] | [4] | [price] | 1 | 14400 | | numbers | numbers_last | [] | [] | [0, 1] | [a, b] | 5 | 14400 | | home | home_last | [0] | [room] | [1, 2, 3] | [temp, hum, co] | 5 | 60 | --- # Manage your InfluxDB 3 Enterprise license Source: https://docs.influxdata.com/influxdb3/enterprise/admin/license/ InfluxDB 3 Enterprise licenses authorize the use of the InfluxDB 3 Enterprise software and apply to a single cluster. Licenses are primarily based on the number of CPUs InfluxDB can use, but there are other limitations depending on the license type. > [!Note] > #### License storage portability in v3.10 and later > > Starting in **v3.10**, InfluxDB 3 Enterprise licenses are no longer tied to a specific cluster’s storage location. > You can copy a valid license file to another object store when moving a > deployment. > Licenses are still required, with per-cluster CPU core limits and license expiration continuing to apply. > This change applies to Commercial, Trial, and At-Home licenses and isn’t > backwards compatible with versions earlier than v3.10. * [License feature comparison](#license-feature-comparison) * [CPU limit](#cpu-limit) * [CPU accounting](#cpu-accounting) * [Upgrade from InfluxDB 3 Core](#upgrade-from-influxdb-3-core) * [Activate a license](#activate-a-license) * [Activate a trial or at-home license](#activate-a-trial-or-at-home-license) * [Activate a commercial license](#activate-a-commercial-license) * [Move a valid license to another object store](#move-a-valid-license-to-another-object-store) * [Change your license type](#change-your-license-type) * [Renew a license](#renew-a-license) * [Expiration behavior](#expiration-behavior) ## License feature comparison The following InfluxDB 3 Enterprise license types are available: * **Trial**: 30-day trial license with full access to InfluxDB 3 Enterprise capabilities. * **At-Home**: For at-home hobbyist use with limited access to InfluxDB 3 Enterprise capabilities. * **Commercial**: Commercial license with full access to InfluxDB 3 Enterprise capabilities. | Features | Trial |At-Home| Commercial | |--------------|-------|-------|--------------| |CPU Core Limit| 256 | 2 |*Per contract*| | Expiration |30 days|*Never*|*Per contract*| | Multi-node | | | | |Commercial use| | | | All other InfluxDB 3 Enterprise features are available to all licenses. ## CPU limit Each InfluxDB 3 Enterprise license limits the number of CPUs InfluxDB can use. The CPU limit is per cluster, not per machine. A cluster may consist of multiple nodes that share the available CPU limit. For example, you can purchase a 32-CPU Commercial license and set up an InfluxDB 3 Enterprise cluster with the following: * 3 × writer nodes, each with 4 CPUs (12 total) * 1 × compactor node with 8 CPUs * 3 × query nodes, each with 4 CPUs (12 total) With the InfluxDB 3 Enterprise Commercial license, CPU cores are purchased in batches of 8, 16, 32, 64, or 128 cores. ### CPU accounting CPU cores are determined by whatever the operating system of the host machine reports as its core count. InfluxDB 3 Enterprise does not differentiate between physical and virtual CPU cores. > [!Note] > If using Linux, InfluxDB uses whatever cgroup CPU accounting is active–for > example: `cpuset` or `cpu.shares`. ## Upgrade from InfluxDB 3 Core To upgrade from InfluxDB 3 Core to InfluxDB 3 Enterprise, see[Upgrade from Core](/influxdb3/enterprise/admin/upgrade-from-core/). ## Activate a license Each InfluxDB 3 Enterprise license must be activated when you start the server, but the process of activating the license depends on the license type: * [Activate a trial or at-home license](#activate-a-trial-or-at-home-license) * [Activate a commercial license](#activate-a-commercial-license) ### Activate a trial or at-home license 1. Use the [`influxdb3 serve` command](/influxdb3/enterprise/reference/cli/influxdb3/serve/) to start the server. If the server doesn’t find a license file or email address, the server prompts you to enter your email address. If you’re [activating an InfluxDB trial or home license with Docker](?t=Docker+Compose#start-the-server-with-your-license-email) or [with DEB/RPM installs](?t=DEB%2FRPM+%28TOML%29#start-the-server-with-your-license-email), include options to [skip the email prompt](#skip-the-email-prompt). 2. The server prompts you to select a license type. Select `trial` or `home`. 3. In the verification email from InfluxDB 3 Enterprise, click the button to verify your email address. After you verify your email address, InfluxDB 3 Enterprise auto-generates a license (associated with your cluster and email address) and stores the license file in your object store. The license file is a JWT file that contains the license information. > [!Important] > #### Activate a trial or home license with Docker or Linux packaging > > If you’re starting a new InfluxDB 3 Enterprise server in a Docker container or > installed via DEB or RPM, you must use one of the methods to[skip the email prompt](#skip-the-email-prompt) and specify a license type. > This ensures that the server can generate the license file after you > verify your email address. See the following examples: > > * [Docker Compose example](?t=Docker+Compose#start-the-server-with-your-license-email) > * [DEB and RPM TOML configuration](?t=DEB%2FRPM+%28TOML%29#start-the-server-with-your-license-email) #### Skip the email prompt To skip the email prompt when starting the server, you can provide your email address using one of the following methods: * **CLI option:** Use the [`--license-email`](/influxdb3/enterprise/reference/cli/influxdb3/serve/) option with the `influxdb3 serve` command * **Environment variable:** Set the `INFLUXDB3_LICENSE_EMAIL` environment variable * **TOML config (DEB/RPM-only):** Set the [`license-email`](/influxdb3/enterprise/reference/config-options/#license-email) option in the [`/etc/influxdb3/influxdb3-enterprise.conf` file](/influxdb3/enterprise/install/#toml-configuration-linux) for a DEB or RPM install > [!Note] > #### Deprecated Enterprise license environment variables > > In InfluxDB 3 Enterprise 3.11+, use `INFLUXDB3_LICENSE_EMAIL`,`INFLUXDB3_LICENSE_FILE`, and `INFLUXDB3_LICENSE_TYPE`. > The legacy `INFLUXDB3_ENTERPRISE_*` names remain supported as deprecated > aliases, but the server logs a deprecation warning at startup. If the server finds a valid license file in your object store, it ignores the license email option. See examples to [start the server with your license email](#start-the-server-with-your-license-email). #### Select a license type without a prompt For a noninteractive startup, specify either `trial` or `home` in addition to the license email: * **CLI option:** Use `--license-type trial` or `--license-type home` with the`influxdb3 serve` command. * **Environment variable:** Set the `INFLUXDB3_LICENSE_TYPE`environment variable. * **TOML config (DEB/RPM-only):** Set `license-type` in`/etc/influxdb3/influxdb3-enterprise.conf`. #### Use an existing trial or at-home license When you activate a trial or at-home license, InfluxDB registers your email address with the license server. To use your existing license–for example, if you deleted your license file–provide your email address using one of the following methods: * **CLI option:** Use the [`--license-email`](/influxdb3/enterprise/reference/cli/influxdb3/serve/) option with the `influxdb3 serve` command * **Environment variable:** Set the `INFLUXDB3_LICENSE_EMAIL` environment variable * **TOML config (DEB/RPM-only):** Set the [`license-email`](/influxdb3/enterprise/reference/config-options/#license-email) option in the [`/etc/influxdb3/influxdb3-enterprise.conf` file](/influxdb3/enterprise/install/#toml-configuration-linux) for a DEB or RPM install InfluxDB validates your email address with the license server and uses your existing license if it’s still valid. ### Activate a commercial license 1. [Contact InfluxData Sales](https://influxdata.com/contact-sales/) to obtain an InfluxDB 3 Enterprise Commercial license. Provide the following: * Cluster UUID * Object Store Info This information is provided in the output of the InfluxDB 3 Enterprise server if you try to start the server without a valid license. InfluxData will provide you with a commercial license file. The license file is a JWT file that contains the license information. 2. When starting the InfluxDB 3 Enterprise server, provide the license file path using one of the following methods: * **CLI option:** Use the [`--license-file`](/influxdb3/enterprise/reference/config-options/#license-file) option with the `influxdb3 serve` command * **Environment variable:** Set the `INFLUXDB3_LICENSE_FILE` environment variable. * **TOML config (DEB/RPM-only):** Set the [`license-file`](/influxdb3/enterprise/reference/config-options/#license-file) option in the [`/etc/influxdb3/influxdb3-enterprise.conf` file](/influxdb3/enterprise/install/#toml-configuration-linux) for a DEB or RPM install ### License detection InfluxDB 3 Enterprise checks for a license in the following order: 1. `--license-file` CLI option or `license-file` TOML option 2. `INFLUXDB3_LICENSE_FILE` environment variable 3. Default commercial license path in the object store 4. Default trial/home license path in the object store 5. `--license-email` CLI option or `license-email` TOML option 6. `INFLUXDB3_LICENSE_EMAIL` environment variable 7. If no license is found, the server won’t start #### Default license file location When using the default license path (items 3-4 above), InfluxDB 3 Enterprise looks for the license file in your [object store directory](/influxdb3/enterprise/get-started/setup/#object-store-examples): ```text //commercial_license //trial_or_home_license ``` ### Move a valid license to another object store In v3.10 and later, you can copy a valid license file to the object store for another InfluxDB 3 Enterprise deployment. Copy the file to the default license path that the new server checks: ```text //commercial_license //trial_or_home_license ``` When the new server finds a valid license file in this location, it uses the file before checking license email options. Copying a license file doesn’t renew an expired license or change its CPU limit. Treat license files as secrets. Don’t commit a license file or include its contents in logs, support requests, or other shared artifacts. ### Start the server with your license email The following examples show how to provide your license email for different InfluxDB 3 Enterprise installation methods: [Quick install (CLI)](#quick-install-cli)[Quick install (env var)](#quick-install-env-var)[Docker Compose](#docker-compose)[DEB/RPM (TOML)](#debrpm-toml) ```bash influxdb3 serve \ --cluster-id CLUSTER_ID \ --node-id NODE_ID \ --license-type LICENSE_TYPE \ --license-email EMAIL_ADDRESS \ # ... ``` ```bash export INFLUXDB3_LICENSE_EMAIL=EMAIL_ADDRESS export INFLUXDB3_LICENSE_TYPE=LICENSE_TYPE influxdb3 serve \ --cluster-id CLUSTER_ID \ --node-id NODE_ID \ # ... ``` ```yaml # compose.yaml name: data-crunching-stack services: influxdb3-enterprise: container_name: influxdb3-enterprise image: influxdb:3-enterprise ports: - 8181:8181 command: - influxdb3 - serve - --node-id=NODE_ID - --cluster-id=CLUSTER_ID - --license-type=LICENSE_TYPE - --object-store=file - --data-dir=/var/lib/influxdb3/data - --plugin-dir=/var/lib/influxdb3/plugins environment: - INFLUXDB3_LICENSE_EMAIL=EMAIL_ADDRESS - INFLUXDB3_LICENSE_TYPE=LICENSE_TYPE volumes: - type: bind # Path to store data on your host system source: ~/.influxdb3/data # Path to store data in the container target: /var/lib/influxdb3/data - type: bind # Path to store plugins on your host system source: ~/.influxdb3/plugins # Path to store plugins in the container target: /var/lib/influxdb3/plugins ``` Replace the following: * `EMAIL_ADDRESS`: Your email address for license activation * `LICENSE_TYPE`: `trial` or `home` * `NODE_ID`: Your existing node identifier from Core * `CLUSTER_ID`: A new cluster identifier for Enterprise * `~/.influxdb3/data`: The same data directory you used with Core * `~/.influxdb3/plugins`: The same plugins directory you used with Core 1. Edit `/etc/influxdb3/influxdb3-enterprise.conf` to add your license email: ``` license-email="EMAIL_ADDRESS" license-type="LICENSE_TYPE" ``` 2. To start the server, run the following command: ``` # systemd (modern systems; see logs with 'journalctl --unit influxdb3-enterprise') systemctl start influxdb3-enterprise # SysV init (legacy systems; logs to /var/lib/influxdb3/influxdb3-enterprise.log) /etc/init.d/influxdb3-enterprise start ``` ### Start the server with your license file The following examples show how to provide your license file for different InfluxDB 3 Enterprise installation methods: [Quick install (CLI)](#quick-install-cli-2)[Quick install (env var)](#quick-install-env-var-2)[DEB/RPM (TOML)](#debrpm-toml-2) ```bash influxdb3 serve \ --cluster-id cluster01 \ --node-id node01 \ --license-file /path/to/license-file.jwt \ # ... ``` ```bash export INFLUXDB3_LICENSE_FILE=/path/to/license-file.jwt influxdb3 serve \ --cluster-id cluster01 \ --node-id node01 \ # ... ``` 1. Edit `/etc/influxdb3/influxdb3-enterprise.conf` to add your license file path: ``` license-file="/etc/influxdb3/license-file.jwt" ``` 2. Ensure the license file has strict permissions that allow the database to read the file: ``` chown root:influxdb3 /etc/influxdb3/influxdb3-enterprise.conf chmod 0640 /etc/influxdb3/influxdb3-enterprise.conf ``` 3. To start the server, run the following command: ``` # systemd (modern systems; see logs with 'journalctl --unit influxdb3-enterprise') systemctl start influxdb3-enterprise # SysV init (legacy systems; logs to /var/lib/influxdb3/influxdb3-enterprise.log) /etc/init.d/influxdb3-enterprise start ``` For more information about `influxdb3 serve` options, see the[CLI reference](/influxdb3/enterprise/reference/cli/influxdb3/serve/). ## Change your license type If you need to change your InfluxDB 3 Enterprise license type, see the appropriate section below based on your current and target license types. ### Identify your current license Check your current license type and status: ```bash influxdb3 show license --host http://localhost:8181 ``` ### Switch between Trial and At-Home licenses To switch between Trial and At-Home license types (or reset a Trial license),[contact InfluxData support](https://support.influxdata.com/s/contactsupport). Include the following information in your request: * The email address associated with your current license * Your current license type (Trial or At-Home) * The license type you want to switch to ### Change to a Commercial license To upgrade to a Commercial license from any other license type: 1. [Contact InfluxData Sales](https://influxdata.com/contact-sales/) to obtain a Commercial license file. Provide: * Your cluster ID * Your object store information 2. **Stop the server.** 3. **Start the server with the commercial license file.** The commercial license takes precedence over any existing trial or at-home license in your object store. [Quick install (Linux/macOS)](#quick-install-linuxmacos)[Docker](#docker)[DEB/RPM (systemd)](#debrpm-systemd) Start the server with the `--license-file` option: ```bash influxdb3 serve \ --license-file /path/to/commercial-license.jwt \ # ... other options ``` Or set the environment variable: ```bash export INFLUXDB3_LICENSE_FILE=/path/to/commercial-license.jwt influxdb3 serve # ... other options ``` Verify the new license: ```bash influxdb3 show license --host http://localhost:8181 ``` Mount the license file and set the environment variable: ```bash docker run -d \ -v /path/to/commercial-license.jwt:/license.jwt:ro \ -e INFLUXDB3_LICENSE_FILE=/license.jwt \ # ... other options influxdb:3-enterprise ``` Verify the new license: ```bash influxdb3 show license --host http://localhost:8181 ``` 1. **Copy the license file** to the server: ``` sudo cp /path/to/commercial-license.jwt /etc/influxdb3/license.jwt sudo chown influxdb3:influxdb3 /etc/influxdb3/license.jwt sudo chmod 600 /etc/influxdb3/license.jwt ``` 2. **Update the configuration** to use the license file: Edit `/etc/influxdb3/influxdb3-enterprise.conf`: ``` license-file = "/etc/influxdb3/license.jwt" ``` 3. **Start the server:** ``` sudo systemctl start influxdb3-enterprise ``` 4. **Verify the new license:** ``` influxdb3 show license --host http://localhost:8181 ``` ## Renew a license To renew an InfluxDB 3 Enterprise Commercial license, contact[InfluxData Sales](https://influxdata.com/contact-sales/). ## Expiration behavior When your InfluxDB 3 Enterprise license expires, the following occurs: * Write requests continue to be accepted and processed. * Compactions continue to optimize persisted data. * Query requests return an error. * If the InfluxDB 3 Enterprise server stops, it will not restart without a valid, non-expired license. #### Related * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) * [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/) * [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/) * [Install InfluxDB 3 Core](/influxdb3/core/install/) | Features | Trial | At-Home | Commercial | | --- | --- | --- | --- | | Features | Trial | At-Home | Commercial | | CPU Core Limit | 256 | 2 | Per contract | | Expiration | 30 days | Never | Per contract | | Multi-node | | | | | Commercial use | | | | --- # Capture workload data Source: https://docs.influxdata.com/influxdb3/enterprise/admin/load-capture/ * InfluxDB 3 Enterprise v3.10.0+ Load capture records an anonymized profile of write requests, query requests, or both from a running InfluxDB 3 Enterprise cluster. InfluxData uses the profile to reproduce your workload and troubleshoot write or query performance while limiting exposure to the identifiers and values that load capture anonymizes. A capture anonymizes database names, table names, tag keys, tag values, field names, string and binary field values, and string query literals before writing them to object storage. You can preview, download, and inspect a capture locally, then share it with InfluxData support through your support channel. > [!Important] > #### Load capture requires the upgraded storage engine and a query-mode node > > Load capture requires the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/), > the default for new clusters. > Clusters that started on 3.10 or earlier can migrate with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > Send `loadcap` requests to a node with an explicit `--mode` setting that > includes `query`—for example, `--mode query` or`--mode ingest --mode query --mode compact`. > > A node using the default `--mode all` configuration returns`404 loadcap not available`, even though it includes query mode. > Load capture is also unavailable on ingest-, compact-, or process-only nodes > and on InfluxDB 3 Core. > For more information about node modes, see[Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/)and the [`--mode` option](/influxdb3/enterprise/reference/config-options/#mode). ## Capture types When you start a capture, choose one of three types: * **`query`**: records SQL, InfluxQL, and Flight SQL query text sent to the query node. * **`write`**: records anonymized write-ahead log (WAL) files and a snapshot of the catalog. * **`both`**: records query and write artifacts in one profile. A `write` or `both` capture requires write traffic reaching an ingest node in the same cluster so the query node observes replicated WAL data. A `query` capture records queries sent to the query node and does not require an ingest node. Constraints: * Only one capture can run on a node at a time. Starting another while one is active returns `409 Conflict`. * Capture durations use `s`, `m`, or `h` units—for example, `30s`, `5m`, or`1h`. The maximum duration is one hour. * The catalog snapshot in a `write` or `both` capture is taken when the capture*starts*. Create and populate the schema you want to capture before you start the capture. ## Before you begin * Identify a node in your cluster running query mode. * Load capture requires an[admin token](/influxdb3/enterprise/admin/tokens/admin/). Set connection environment variables to target the query node: ```bash export INFLUXDB3_HOST_URL="https://query-node.example.com:8181" export INFLUXDB3_AUTH_TOKEN="YOUR_ADMIN_TOKEN" ``` ## Capture a workload profile A capture records only the traffic that arrives while the capture is running. Queries executed before the capture starts and requests sent after the capture duration expires are not included in the profile. Start the capture first, and then run your workload. Complete the following steps in order: 1. [Start a capture](#start-a-capture) 2. [Send traffic during the capture window](#send-traffic-during-the-capture-window) 3. [Wait for the capture to complete](#wait-for-the-capture-to-complete) 4. [Preview the capture](#preview-the-capture) ### Start a capture Start a capture and specify the type and duration: ```bash influxdb3 loadcap start --type both --duration 5m ``` The command returns the profile ID: ```json { "profile_id": "550e8400-e29b-41d4-a716-446655440000" } ``` Store the profile ID to reference the capture in later commands: ```bash export PROFILE_ID="550e8400-e29b-41d4-a716-446655440000" ``` The capture is now running for the requested duration. ### Send traffic during the capture window With the capture running, generate the workload you want to record: * For a `query` or `both` capture, send SQL, InfluxQL, or Flight SQL queries to the query node. Only queries that the query node executes while the capture is running are recorded—queries that completed before the capture started do not appear in the profile. * For a `write` or `both` capture, send write requests to an ingest node in the same cluster so the query node observes replicated WAL data. ### Wait for the capture to complete The capture stops automatically when the requested duration expires. List capture profiles and their status: ```bash influxdb3 loadcap list ``` A profile reports `running`, `complete`, or `aborted` status, along with its type, requested duration, and creation time. Wait for the profile to report `complete` before you preview or download it. ### Preview the capture Preview a profile without downloading it: ```bash influxdb3 loadcap preview --profile-id "$PROFILE_ID" ``` The preview reports the file count, total size, WAL file count, query count, and—for `write` and `both` captures—an anonymized catalog summary. ## Inspect before you share A capture preserves the shape and timing of your workload. It does not hide every operational characteristic. Timestamps, numeric and boolean values, schema shape, file counts, query cadence, query time ranges, and regular expression selectivity are preserved by design. ### Download and list the archive Download the profile as a gzipped tar archive: ```bash influxdb3 loadcap download \ --profile-id "$PROFILE_ID" \ --output "$PROFILE_ID.tar.gz" ``` If you omit `--output`, the CLI writes `{profile_id}.tar.gz`. List the archive contents: ```bash tar -tzf "$PROFILE_ID.tar.gz" ``` A `both` profile archive typically contains an anonymized `catalog.json`, a zstd-compressed `queries.json.zst`, and one or more anonymized `.pt` WAL files: ```text catalog.json queries.json.zst 0-00000029.pt 0-00000030.pt ... ``` A `query`-only profile omits `catalog.json` and `.pt` files. A `write`-only profile omits `queries.json.zst`. ### Extract and inspect Extract the archive into a temporary directory: ```bash mkdir -p loadcap-inspect tar -xzf "$PROFILE_ID.tar.gz" -C loadcap-inspect ``` For a `write` or `both` capture, inspect the anonymized catalog: ```bash jq . loadcap-inspect/catalog.json ``` Inspect the anonymized queries: ```bash zstd -dc loadcap-inspect/queries.json.zst | jq . ``` Each query entry contains anonymized query text, the query language, an anonymized database identifier, and the original timestamp: ```json { "sql": "SELECT \"0-14\", \"0-T\" FROM \"0-gwKPhKqqB5\" WHERE \"0-14\" = 'MHLLIfxq48Mt0'", "query_language": "sql", "database": "0-RdYkt0Bktle", "timestamp": "2026-04-27T12:34:56.789Z" } ``` The `.pt` files are binary WAL files for InfluxData tooling. Verify their names and sizes: ```bash ls -lh loadcap-inspect/*.pt ``` ## What load capture anonymizes Each capture uses a random key that is unique to that capture. The same original name or value maps to the same anonymized value within one capture, but a different capture produces different values. The key is not included in the profile. Load capture anonymizes the following: * Database, table, tag, field family, and field names, replaced with deterministic identifiers such as `1-0Dcc2PQ9o7zG`. * Tag values, string field values, and binary field values in WAL files. * Identifiers and string literals in SQL, InfluxQL, and Flight SQL queries. Load capture preserves the following: * Timestamps, and numeric and boolean values in both captured write data and query literals. * Query structure, operators, function names, and time-range literals. * The InfluxQL `time` identifier and regular expression structure (anchors, character classes, and repetition), so query selectivity stays close to production. * Node IDs, modes, core counts, and node state in the catalog snapshot. Queries that the anonymizer cannot fully parse are replaced with a placeholder such as `` or ``. Statement types without full coverage—including `INSERT`, `UPDATE`, and`DELETE`—are replaced wholesale. A query capture therefore does not represent write-shaped statement traffic. ## Delete a capture Delete a profile when you no longer need it: ```bash influxdb3 loadcap delete --profile-id "$PROFILE_ID" ``` ## Share with InfluxData After you inspect a capture and confirm its contents, provide the archive to InfluxData support through your existing support channel. InfluxData uses the profile to reproduce and analyze your workload. For the load capture HTTP API, see the[Load capture API reference](/influxdb3/enterprise/api/load-capture/). #### Related * [Load capture](/influxdb3/enterprise/api/load-capture/) * [Manage admin tokens](/influxdb3/enterprise/admin/tokens/admin/) * [Performance tuning](/influxdb3/enterprise/admin/performance-tuning/) * [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) --- # Use the InfluxDB 3 MCP server Source: https://docs.influxdata.com/influxdb3/enterprise/admin/mcp-server/ InfluxDB provides two Model Context Protocol (MCP) servers for integrating with AI assistants: * [Manage your InfluxDB instance with the database MCP server](#manage-your-influxdb-instance-with-the-database-mcp-server) * [Query documentation from your IDE](#query-documentation-from-your-ide) ## Manage your InfluxDB instance with the database MCP server The **InfluxDB database MCP server** lets you interact with InfluxDB 3 Enterprise using natural language with large language model (LLM) agents. It enables database management, token handling, and SQL query generation in plain English—no coding required. This section walks you through configuring your LLM agent to run and use the InfluxDB database MCP server to interact with your InfluxDB 3 Enterprise server . ### Prerequisites * Node.js v18+ *(if using `npx` to run the MCP server)* * Docker *(if using Docker to run the MCP server)* * A running and reachable InfluxDB 3 Enterprise server . * A valid InfluxDB 3 Enterprise [admin or resource token](/influxdb3/enterprise/admin/tokens/#manage-admin-tokens) with appropriate resource-specific permissions * *(Optional)* An LLM assistant like Claude Desktop, ChatGPT Desktop, etc. ### Configure the database MCP server Use environment variables to configure the InfluxDB 3 MCP server and connect it to your InfluxDB 3 Enterprise server . Set the following environment variables when you start the MCP server: #### Required InfluxDB connection variables * **INFLUX\_DB\_PRODUCT\_TYPE**: `enterprise` * **INFLUX\_DB\_INSTANCE\_URL**: Your InfluxDB 3 Enterprise URL–for example: ``` http://localhost:8181 ``` If using Docker to run the InfluxDB MCP server and you want to manage an InfluxDB instance running on `localhost`, use the internal Docker network to connect to your local InfluxDB 3 Enterprise server–for example: ``` http://host.docker.internal:8181/ ``` * **INFLUX\_DB\_TOKEN**: Your InfluxDB 3 Enterprise [admin token](/influxdb3/enterprise/admin/tokens/admin) or [resource token](/influxdb3/enterprise/admin/tokens/resource). If using a resource token, your LLM agent can only perform the operations allowed by the token permissions. ### Configure your LLM agent to run the database MCP server To run the MCP server, use either Node.js and `npm` or Docker. Some LLM agents, like [Claude Desktop](https://claude.ai/download), start, run, and connect to the MCP server for you. The following instructions show how to configure**Claude Desktop** to use the InfluxDB database MCP server. [Node.js](#nodejs)[Docker](#docker) 1. Clone the [influxdata/influxdb3\_mcp\_server repository](https://github.com/influxdata/influxdb3_mcp_server)from GitHub. 2. Navigate to the `influxdb3_mcp_server` project directory: ``` cd influxdb3_mcp_server/ ``` 3. Install dependencies: ``` npm install ``` 4. Build the MCP server: ``` npm run build ``` This builds the files necessary to run the MCP server and stores them in `./build`. The `./build/index.js` file starts the MCP server. #### Configure your LLM Agent to use the Node.js-based MCP server In **Claude Desktop**, go to **Settings** \> **Developer** and edit your configuration. Enter the following JSON configuration: ```json { "mcpServers": { "influxdb": { "command": "node", "args": ["/path/to/influxdb3_mcp_server/build/index.js"], "env": { "INFLUX_DB_PRODUCT_TYPE": "enterprise", "INFLUX_DB_INSTANCE_URL": "http://localhost:8181", "INFLUX_DB_TOKEN": "AUTH_TOKEN" } } } } ``` Replace the following: * `path/to`: The absolute path to your `influxdb3_mcp_server` project directory. * `AUTH_TOKEN`: Your InfluxDB 3 Enterprise token. The permissions granted by the token determine what operations your LLM agents can perform. #### Configure your LLM Agent to use the Docker-based MCP server In **Claude Desktop**, go to **Settings** \> **Developer** and edit your configuration. Enter the following JSON configuration: * [Connect to a remote InfluxDB server](#connect-to-a-remote-influxdb-server) * [Connect to a local InfluxDB server](#connect-to-a-local-influxdb-server) In the examples below, replace the following: * `path/to`: The absolute path to your `influxdb3_mcp_server` project directory. * `AUTH_TOKEN`: Your InfluxDB 3 Enterprise token. The permissions granted by the token determine what operations your LLM agents can perform. ##### Connect to a remote InfluxDB server ```json { "mcpServers": { "influxdb": { "command": "docker", "args": [ "run", "--rm", "--interactive", "--env", "INFLUX_DB_PRODUCT_TYPE", "--env", "INFLUX_DB_INSTANCE_URL", "--env", "INFLUX_DB_TOKEN", "influxdata/influxdb3-mcp-server" ], "env": { "INFLUX_DB_PRODUCT_TYPE": "enterprise", "INFLUX_DB_INSTANCE_URL": "http://localhost:8181", "INFLUX_DB_TOKEN": "AUTH_TOKEN" } } } } ``` ##### Connect to a local InfluxDB server ```json { "mcpServers": { "influxdb": { "command": "docker", "args": [ "run", "--rm", "--interactive", "--add-host=host.docker.internal:host-gateway", "--env", "INFLUX_DB_PRODUCT_TYPE", "--env", "INFLUX_DB_INSTANCE_URL", "--env", "INFLUX_DB_TOKEN", "influxdata/influxdb3-mcp-server" ], "env": { "INFLUX_DB_PRODUCT_TYPE": "enterprise", "INFLUX_DB_INSTANCE_URL": "http://host.docker.internal:8181", "INFLUX_DB_TOKEN": "AUTH_TOKEN" } } } } ``` ### Supported features Once connected, you can use your LLM agent to perform tasks on your InfluxDB 3 Enterprise server , including: * Create, update, and delete databases * List tables and inspect schemas * Create and manage tokens * Query data without writing SQL or InfluxQL * Check server health and connection status #### Examples of supported prompts > > > “List all tables in the `production` database.” > > > > “Create a read-only token for the `metrics` database.” > > > > “Analyze last week’s sensor data for anomalies.” > > > > “Create a new database called `iot_sensors` with a 30-day retention policy.” > > > > “Show me the schema for the `sensor_data` table.” > > ## Query documentation from your IDE The **InfluxDB documentation MCP server** lets AI tools and agents search InfluxDB InfluxDB 3 Enterprise documentation directly from your development environment. Use it to find answers, code examples, and configuration details without leaving your IDE. ### Why use the documentation MCP server? When you connect the documentation MCP server to your AI coding assistant, the assistant can search InfluxDB and related tool documentation to answer your questions with accurate, up-to-date information. Instead of switching to a browser or guessing at syntax, you can ask questions in your IDE and get responses grounded in official documentation. **Common use cases:** * Get help writing queries, client library code, or CLI commands * Look up configuration options and environment variables * Find code examples for specific tasks * Troubleshoot errors with documentation-backed answers ### Install the documentation MCP server The documentation MCP server is a hosted service—you don’t need to install or run anything locally. Add the server URL to your AI assistant’s MCP configuration. > [!Note] > On first use, you’ll be prompted to sign in with a Google or GitHub account. > This authentication is used only for rate limiting—no personal data is collected. **MCP server URL:** ```text https://influxdb-docs.mcp.kapa.ai ``` The server uses SSE (Server-Sent Events) transport. #### Configure your AI assistant to use the documentation MCP server The following instructions show how to configure popular AI assistants to use the InfluxDB documentation MCP server. [Claude Desktop](#claude-desktop)[ChatGPT Desktop](#chatgpt-desktop)[GitHub Copilot (VS Code)](#github-copilot-vs-code)[Cursor](#cursor)[OpenCode](#opencode) In **Claude Desktop**, go to **Settings** \> **Developer** and edit your configuration. Add the following JSON configuration: ```json { "mcpServers": { "influxdb-docs": { "url": "https://influxdb-docs.mcp.kapa.ai" } } } ``` Save the file and restart Claude Desktop for the changes to take effect. In **ChatGPT Desktop**, go to **Settings** \> **Integrations** \> **Enable MCP** and add a new server. Add the following JSON configuration: ```json { "mcpServers": { "influxdb-docs": { "url": "https://influxdb-docs.mcp.kapa.ai", "transport": "sse" } } } ``` Save the configuration and restart ChatGPT Desktop. In **VS Code**, configure GitHub Copilot to use the MCP server: 1. Create or edit `.vscode/mcp.json` in your workspace or project directory 2. Add the following configuration: ```json { "servers": { "influxdb-docs": { "type": "http", "url": "https://influxdb-docs.mcp.kapa.ai" } } } ``` 1. Restart or reload VS Code 2. Open the Command Palette (`Ctrl+Shift+P` or `Cmd+Shift+P`) 3. Run **MCP: List Servers** to verify the server is registered The InfluxDB documentation MCP server will now be available through GitHub Copilot Chat. In **Cursor**, add the MCP server configuration to your MCP settings file. 1. Open **Settings** and navigate to **MCP Servers** 2. Click **Add MCP Server** or edit the configuration file directly 3. Add the following configuration to `.cursor/mcp.json` (project-level) or `~/.cursor/mcp.json` (global): ```json { "mcpServers": { "influxdb-docs": { "url": "https://influxdb-docs.mcp.kapa.ai", "transport": "streamableHttp" } } } ``` Save the file and restart Cursor. In **OpenCode**, configure the MCP server in your configuration file: 1. Create or edit `opencode.json` (or `opencode.jsonc`) in your workspace 2. Add the following configuration: ```json { "$schema": "https://opencode.ai/config.json", "mcp": { "influxdb-docs": { "type": "remote", "url": "https://influxdb-docs.mcp.kapa.ai", "enabled": true } } } ``` 1. Start OpenCode and use the `/init` command to verify the MCP server is accessible The InfluxDB documentation search tools will be available in your OpenCode sessions. ### Authentication and rate limits When you connect to the documentation MCP server for the first time, a sign-in window opens where you can choose to authenticate with a **Google** or **GitHub** account. The hosted MCP server uses your account only to generate a stable, opaque user ID for rate limiting—no personal data is collected: * **Google**: Requests only the `openid` scope. Does not request `email` or `profile`scopes—your name, email address, and other personal data are not collected. * **GitHub**: Requests no OAuth scopes. With no scopes requested, GitHub grants read-only access to public profile information only. The server does not access repositories, organizations, email addresses, or other GitHub data. The anonymous user ID enforces per-user rate limits to prevent abuse: * **40 requests** per user per hour * **200 requests** per user per day > [!Tip] > If you sign in with Google, the consent screen may display > “Associate you with your personal info on Google.” > This is Google’s generic wording for the `openid` scope—it means the app can recognize > that the same Google account is signing in again. > It does not grant access to your email, name, contacts, or other data. ### Search documentation with the MCP tool The documentation MCP server exposes a semantic search tool: ```text search_influxdb_knowledge_sources ``` This tool lets AI agents perform semantic retrieval over InfluxDB documentation and related knowledge sources. **What the tool does:** * Searches all InfluxDB documentation for a given query * Returns the most relevant chunks in descending order of relevance * Each chunk is a self-contained snippet from a single documentation page **Response format:** Each result includes: * `source_url`: URL of the original documentation page * `content`: The chunk content in Markdown MCP tool search results showing InfluxDB documentation ### Use the documentation MCP server After you install the documentation MCP server, your AI assistant can search InfluxDB documentation to help you with tasks. Ask questions naturally—the assistant uses the MCP server to find relevant documentation and provide accurate answers. #### Example prompts > > > “How do I write data to InfluxDB using Python?” > > > > “What’s the syntax for a SQL query with a WHERE clause in InfluxDB?” > > > > “Show me how to configure Telegraf to collect CPU metrics.” > > > > “What environment variables does the InfluxDB CLI use?” > > > > “How do I create a database token with read-only permissions?” > > #### Related * [InfluxDB 3 MCP Server GitHub Repository](https://github.com/influxdata/influxdb3_mcp_server) [MCP](/influxdb3/enterprise/tags/mcp/)[LLM](/influxdb3/enterprise/tags/llm/)[AI](/influxdb3/enterprise/tags/ai/) --- # Manage the node lifecycle Source: https://docs.influxdata.com/influxdb3/enterprise/admin/node-lifecycle/ Every InfluxDB 3 Enterprise server process registers itself as a *node* in the catalog—the metadata store that tracks databases, tables, and nodes. The catalog is the source of truth for a node’s identity and state, and it persists in object storage—independently of the process, its container, or its host. Understanding how a node moves through its states helps you restart, upgrade, scale, and decommission InfluxDB 3 Enterprise safely. * [Node states](#node-states) * [Lifecycle overview](#lifecycle-overview) * [Register a node](#register-a-node) * [Stop a node](#stop-a-node) * [Remove a node](#remove-a-node) * [Re-register a node](#re-register-a-node) * [Restart compared to removal](#restart-compared-to-removal) * [Deploy with an orchestrator](#deploy-with-an-orchestrator) * [Verify node state](#verify-node-state) * [Troubleshoot node lifecycle issues](#troubleshoot-node-lifecycle-issues) ## Node states The catalog records one of the following states for each node: | State | Description | |----------|----------------------------------------------------------------------| |`running` | The node started and registered itself in the catalog | |`stopping`|A graceful stop was requested, but the node hasn’t acknowledged it yet| |`stopped` | The node acknowledged its final snapshot and completed shutdown | |`removing`| The node is marked for permanent removal from the cluster | Node identity has two parts: * **Node ID**: The name you assign with[`--node-id`](/influxdb3/enterprise/reference/config-options/#node-id). It identifies the node across restarts and is the value you pass to node management commands. * **Instance ID**: A UUID that InfluxDB 3 Enterprise generates the first time a node ID registers. A node that restarts with the same node ID reuses its existing instance ID. ## Lifecycle overview stateDiagram-v2 [\*] --\> running: influxdb3 serve running --\> stopped: SIGTERM or SIGINT running --\> stopping: influxdb3 stop node stopping --\> stopped: node acknowledges final snapshot stopped --\> running: influxdb3 serve (same node ID) stopped --\> removing: influxdb3 remove node removing --\> [\*]: catalog entry and files purged A node reaches `stopped` by one of two paths: * **Process shutdown** (`SIGTERM`, `SIGINT`, or Ctrl-c) moves the node directly from `running` to `stopped`. * **[`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)**moves the node from `running` to `stopping`, and then to `stopped` after the node acknowledges its final snapshot. Only the second path records a final snapshot sequence, and only the second path frees the node’s licensed cores for other nodes. ## Register a node When you start a server with[`influxdb3 serve`](/influxdb3/enterprise/reference/cli/influxdb3/serve/), the node registers itself in the catalog and enters the `running` state: ```bash influxdb3 serve \ --node-id NODE_ID \ --cluster-id CLUSTER_ID \ --object-store s3 \ --bucket BUCKET_NAME ``` Replace the following: * `NODE_ID`: A unique identifier for this node * `CLUSTER_ID`: The identifier shared by all nodes in the cluster * `BUCKET_NAME`: The object storage bucket for the cluster Registration is how a node claims its node ID. If the node ID already exists in the catalog, InfluxDB 3 Enterprise applies the[re-registration rules](#re-register-a-node) before accepting the node. ## Stop a node How you stop a node depends on whether the node stays in the cluster. ### Stop a node for a restart To restart a node in place—for a rolling upgrade, a configuration change, or a node reschedule—signal the process (`SIGTERM`, `SIGINT`, or Ctrl-c). The node stops accepting writes, flushes its write-ahead log (WAL) buffer to object storage, waits for an in-progress snapshot to finish, and marks itself`stopped` in the catalog. Because the final flush writes buffered data to the WAL in object storage, acknowledged writes survive the shutdown, and WAL replay restores them when the node restarts with the same node ID. ### Stop a node before removing it To take a node out of the cluster permanently, use[`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)against the **live** node: ```bash influxdb3 stop node --node-id NODE_ID ``` The stop proceeds in two phases: 1. InfluxDB 3 Enterprise marks the node `stopping` in the catalog. 2. The node completes its stop cascade, draining its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail)—the writes buffered since the last snapshot. 3. The node acknowledges the stop, reads as `stopped`, and its licensed cores are freed for other nodes. By default, the command waits for the node to reach `stopped`(up to `--timeout`, default `5m`). Use `--no-wait` to return as soon as the cluster accepts the request. > [!Important] > #### Run stop node against the live node—don’t kill it first > > `stop node` is how a node drains its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail) and > records a final snapshot. > If you kill the process first and run `stop node` afterward, the dead process > can’t drain anything, and[removing the node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)can permanently delete the stranded writes. > If a node already stopped ungracefully, follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/). > [!Note] > #### A flush isn’t a snapshot > > A bare `SIGTERM` flushes the WAL buffer, but it doesn’t force a new snapshot. > The writes are durable and replay on restart, but they remain part of the WAL > tail. > That distinction matters only before[removing a node](#remove-a-node), because removal purges the node’s WAL > files. ### Stop a node that isn’t running Stopping a node that has already stopped returns HTTP `400 Bad Request`: ``` tried to stop a node (NODE_ID) that is already stopped ``` The error is safe to ignore—it reports that the node reached the state you asked for. Automation that stops nodes should treat this response as success rather than retrying. ### Stop a node whose process is already gone `--host` doesn’t have to point at the node you’re stopping. To stop a node whose process is dead, send the request to a **running** node in the same cluster: ```bash influxdb3 stop node \ --node-id NODE_ID \ --host http://RUNNING_NODE:8181 \ --token ADMIN_TOKEN ``` This clears a stale `running` entry that would otherwise keep the cluster expecting a node that never comes back. Remember that a dead process can’t drain its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail), so treat the node as crashed and follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) before you remove it. ## Remove a node Removal is permanent. After a node reaches `stopped`, remove it with[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/): ```bash influxdb3 remove node --node-id NODE_ID ``` InfluxDB 3 Enterprise marks the node `removing`, drains its data up to its final snapshot, and then purges the node’s catalog entry and its object-store file paths. > [!Warning] > #### Removal deletes the node’s files > > Removal permanently deletes the node’s object-store file paths, including its > WAL files. > Any acknowledged writes not covered by the node’s final snapshot are deleted > with them. > Complete a graceful [`stop node`](#stop-a-node-before-removing-it) first. Repeating a remove request for a node that’s already `removing` succeeds without changing state (HTTP `200 OK`). ### Node removal on the Parquet engine On a cluster that still runs the Parquet engine, `remove node` marks the node`removing`, but the Parquet compactor doesn’t complete the removal. The node stays `removing`, its node ID can’t register again, and its files stay in object storage. Only the compactor of the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)completes removals. * **If you plan to upgrade the storage engine**, remove the ingest nodes you’ve retired before you start the[upgrade](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet). Remove a node only after the compactor has absorbed its Parquet files. The upgraded compactor completes the removals after it first starts with`--upgrade-pacha-tree`. For the upgrade, use the latest 3.11.x or 3.12.x release. Starting with v3.11.8 and v3.12.2, the upgrade drops a removed ingest node from its roster only after the node’s Parquet files are converted or absorbed. A removed node with files the compactor hasn’t absorbed holds the upgrade at`upgrading`, and the compactor logs`ingest node removed from catalog with Parquet files not yet converted; upgrade cannot complete`. In earlier releases, any removed ingest node holds the upgrade at `upgrading`. * **If you don’t plan to upgrade**, leave the nodes you’ve retired `stopped`instead of removing them. A `stopped` node can register again under the same node ID; a `removing`node can’t. Removal on the Parquet engine doesn’t delete the node’s Parquet files. After the upgrade,[`influxdb3 manage cleanup-parquet`](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/)deletes Parquet data only for nodes still in the catalog, so a removed node’s Parquet files stay in object storage. ### Removal completes on the compactor’s schedule `removing` isn’t instantaneous, and you can’t force it to finish. The compactor drives removal: it waits until its per-node compaction floor passes the node’s recorded final snapshot sequence, and only then deletes the node’s object-store prefixes and purges the catalog entry. Nothing is deleted before it’s absorbed. A node can therefore sit in `removing` for hours if the compactor is backed up, under-resourced, or restarting—this is the removal working as designed, waiting on compaction. Removing several nodes at once multiplies the backlog it has to absorb. If nodes aren’t clearing, check the compactor before the nodes: ```bash # Compare each node's compaction floor against its snapshots influxdb3 query \ --database _internal \ --token AUTH_TOKEN \ "SELECT node_id, node_prefix, last_snapshot_sequence, \ last_compacted_wal_sequence_number FROM system.pt_compaction_nodes" ``` Nodes disappear from `show nodes` as each removal completes. > [!Warning] > #### Removal keeps the cluster expecting the node until it completes > > While a node is `removing`, its node ID can’t be reclaimed—`removing` is > terminal. > A replacement that reuses the ID (for example, a StatefulSet pod recreated > with the same ordinal name) starts but is refused registration. > Keep your replica count pinned below the removing ordinals until removal > finishes. ### When removal is refused InfluxDB 3 Enterprise refuses removal (HTTP `409 Conflict`) in the following cases: | Condition | Message | Resolution | |-----------------------------|--------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------| | Node isn’t `stopped` | `node 'NODE_ID' is not fully stopped (current state: STATE)` | [Stop the node gracefully](#stop-a-node-before-removing-it) and wait for `stopped` | | Node runs in `compact` mode | `node 'NODE_ID' has compact mode and cannot be removed` | Compactor nodes can’t be removed—see [Remove a compactor node](#remove-a-compactor-node) | |Node belongs to a query group| `cannot remove node 'NODE_ID' because it is a member of query group ...` | Remove the node from the query group first | | Unsnapshotted WAL remains |`node 'NODE_ID' has unsnapshotted WAL (wal file N, snapshotted through M)`|Restart the node, stop it gracefully, then remove it—see [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/)| > [!Note] > #### The unsnapshotted WAL safeguard requires the upgraded storage engine > > The unsnapshotted WAL check applies only to clusters that fully adopted the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters in InfluxDB 3 Enterprise 3.11+). > Clusters on the Parquet engine, or still mid-upgrade, aren’t guarded—on those > clusters, a graceful stop before removal is your only protection against > losing the[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail). > > Clusters that started on 3.10 or earlier keep the Parquet engine until you > restart them with `--upgrade-pacha-tree`. > If you started a storage engine upgrade, confirm it finished—query`system.upgrade_parquet_node` and check that every node reports `completed`. > See [Query system data](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables). ### Remove a compactor node A node running in `compact` mode holds the cluster’s single-writer compaction lease and can’t be removed. To retire the host running your compactor, start a replacement compactor node that reuses the same node ID, as described in[Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/#from-single-node-to-specialized-cluster). ## Re-register a node Whether a node ID can be claimed again depends on the current state of the node in the catalog: | Current state | Can register again? | |-----------------------|------------------------------------------------------------| | `stopped` | Yes—any instance can take over the node ID | |`running` or `stopping`|Only the same instance ID (an idempotent retry or a restart)| | `removing` | No—`removing` is terminal for the node ID | Because a node restarting with the same node ID reuses its existing instance ID, an ordinary restart always satisfies these rules—even if the node still reads as `running` after an ungraceful stop. > [!Warning] > #### Don’t reuse the node ID of a removed node > > After removal completes, InfluxDB 3 Enterprise purges the node’s catalog entry > and its object-store file paths. > While the node is `removing`, registration with that node ID is rejected. > Assign a new node ID to a replacement node instead of racing the removal. ## Restart compared to removal Choosing the wrong operation is the most common way to lose data during routine maintenance. | Goal | Operation | |-----------------------------------------------------------|-------------------------------------------------------------------| |Rolling upgrade, config change, pod reschedule, host reboot|Restart the process with the **same** node ID—don’t remove the node| | Permanently scale down or decommission hardware | `stop node`, verify `stopped`, then `remove node` | | Replace a failed host, keeping its data | Start a node with the same node ID and object store configuration | Restarting a node is not a removal: the node keeps its catalog entry, its instance ID, and its object-store files. Removal is the only operation that purges them. For upgrade-specific sequencing and catalog version constraints, see[Troubleshooting cluster upgrades](/influxdb3/enterprise/admin/upgrade/#troubleshooting-cluster-upgrades). ## Deploy with an orchestrator Helm, Kubernetes, and Ansible deployments drive the node lifecycle on your behalf. The following guidance keeps orchestrated restarts on the graceful path. ### Kubernetes and Helm **Give each node a stable node ID.**Use a StatefulSet so pod names are stable and ordinal-based, and derive`--node-id` from the pod name. The official[InfluxDB 3 Enterprise Helm chart](https://github.com/influxdata/helm-charts/tree/master/charts/influxdb3-enterprise)does this already—it runs a StatefulSet per[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes) and sets`--node-id=$(POD_NAME)` from `metadata.name`. A Deployment generates a new random pod name on every rollout, which registers a new node in the catalog on each restart and leaves the old entries behind. **Set a termination grace period that fits your WAL.**Kubernetes sends `SIGTERM`, waits `terminationGracePeriodSeconds`(default `30`), and then sends `SIGKILL`. A node that’s still flushing when `SIGKILL` arrives stops ungracefully. Set `terminationGracePeriodSeconds` well above your observed shutdown time: ```yaml spec: template: spec: terminationGracePeriodSeconds: 300 ``` > [!Important] > #### The Helm chart doesn’t set a grace period > > The InfluxDB 3 Enterprise Helm chart doesn’t set`terminationGracePeriodSeconds`, so pods inherit the Kubernetes default of > 30 seconds. > For nodes with a large WAL, raise it in your `values.yaml` overrides and > confirm the shutdown completes in the pod logs. **Don’t remove nodes during a rolling update.**A `helm upgrade` or `kubectl rollout restart` terminates and recreates pods with the same names. Each node re-registers with its existing node ID, which is exactly what you want. Never put[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)in a `preStop` hook—it turns every rollout into a permanent decommission. **Sequence rollouts across node modes yourself.**The Helm chart runs a separate StatefulSet per[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes), but the image tag is a single chart-wide value, so one `helm upgrade` rolls every mode at once. Kubernetes doesn’t order rollouts across StatefulSets, so a plain upgrade doesn’t follow the[recommended node upgrade order](/influxdb3/enterprise/admin/upgrade/#recommended-node-upgrade-order). Freeze the modes you aren’t upgrading yet with`updateStrategy.rollingUpdate.partition` and release them one at a time—see the**Helm** tab in[Perform a rolling upgrade](/influxdb3/enterprise/admin/upgrade/#perform-a-rolling-upgrade). **Protect nodes from concurrent drains.**A node drain (`kubectl drain`, cluster autoscaler, or a managed node-pool upgrade) evicts pods the same way a rollout does, but nothing stops several nodes from draining at once. Enable the chart’s pod disruption budget so voluntary disruptions take one node at a time: ```yaml ingester: podDisruptionBudget: enabled: true maxUnavailable: 1 ``` Set it per[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes)(`ingester`, `querier`, `compactor`, and `processingEngine`). The chart creates a budget for a mode only when that mode runs more than one replica, so single-replica modes still drain without protection—make sure their termination grace period is long enough to finish the final flush. **Scale down deliberately.**Reducing a StatefulSet’s replica count stops the pods but leaves the nodes in the catalog as `stopped`. Removing them is a separate, deliberate step: ```bash # 1. Confirm the node reached the stopped state influxdb3 show nodes # 2. Remove the stopped node from the cluster influxdb3 remove node --node-id NODE_ID ``` **Disabling a[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes) leaves its nodes in the catalog.**Setting a mode’s `enabled: false` (for example, `compactor.enabled=false`) deletes that StatefulSet, but the nodes it ran keep their catalog entries. Stop and remove them deliberately, in that order, the same way you would after scaling down. A [compactor node can’t be removed](#remove-a-compactor-node) at all, so disable the compactor only when you’re retiring the cluster or replacing it with a node that reuses the same node ID. **Uninstalling doesn’t clean the catalog.**`helm uninstall` removes Kubernetes resources, but the catalog and object storage persist, and the nodes remain as catalog entries. Reinstalling with the same node IDs and object store reattaches those nodes. ### Ansible and systemd **Let systemd send `SIGTERM`.**The systemd default `KillSignal` is `SIGTERM`, which starts a graceful shutdown—don’t override it with `SIGKILL`. **Raise `TimeoutStopSec`.**If the node doesn’t exit within `TimeoutStopSec`, systemd escalates to`SIGKILL`. Set it above your observed shutdown time: ```ini [Service] KillSignal=SIGTERM TimeoutStopSec=300 ``` **Roll one node at a time.**Use `serial: 1` in your playbook and confirm each node returns to `running`before proceeding. Group your inventory by[node mode](/influxdb3/enterprise/admin/clustering/#configure-node-modes) and order the plays to match the[recommended node upgrade order](/influxdb3/enterprise/admin/upgrade/#recommended-node-upgrade-order)—for a complete playbook, see the **Ansible** tab in[Perform a rolling upgrade](/influxdb3/enterprise/admin/upgrade/#perform-a-rolling-upgrade). **Don’t put `remove node` in a playbook.**A restart keeps the node’s catalog entry and object-store files; removal purges them. Reserve[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)for deliberate decommissioning, never for routine configuration or version rollouts. **Never use `kill -9`.**Ad hoc `kill -9`, `docker kill`, and force-stopped containers all skip the final flush. ## Verify node state Use [`influxdb3 show nodes`](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/)to check the state of every node in the cluster: ```bash influxdb3 show nodes ``` The output includes a `state` column for each node—abbreviated here to the lifecycle-relevant columns: ``` +---------+--------+------------+---------+ | node_id | mode | core_count | state | +---------+--------+------------+---------+ | node-1 | ingest | 1 | running | | node-2 | ingest | 1 | stopped | +---------+--------+------------+---------+ ``` Other nodes observe a state change after their catalog sync interval (default 1 second, set with `--catalog-sync-interval`), so allow for that delay when scripting checks. You can also query the `system.nodes` table in the `_internal` database: ```bash influxdb3 query \ --database _internal \ --token AUTH_TOKEN \ "SELECT node_id, mode, state, updated_at FROM system.nodes" ``` Replace `AUTH_TOKEN`with a token that has permission to query the `_internal` database. ## Troubleshoot node lifecycle issues ### A node is stuck in stopping The node was marked `stopping` but never acknowledged the stop—usually because the process died mid-cascade. Follow [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/): restart the node with the same node ID, stop it gracefully, and then remove it if you still intend to. ### A node still reads as running after it crashed A node that dies without a graceful shutdown never updates its catalog entry, so it keeps its last recorded state. Restarting the node with the same node ID re-registers it—the[re-registration rules](#re-register-a-node) allow it because the instance ID matches. ### Removal fails with a 409 conflict See [When removal is refused](#when-removal-is-refused) for each condition and its resolution. Prefer restarting and gracefully stopping the node over[`--force-finalize`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/#force-removal-of-a-node-that-did-not-shut-down-cleanly). Forcing removal discards the unsnapshotted writes the safeguard was protecting, and it starts an object-store cleanup for a node that never recorded a final snapshot. Restart the node, stop it gracefully, and then remove it. ### A node stays in removing On the Parquet engine, removal doesn’t complete until you upgrade the storage engine. See [Node removal on the Parquet engine](#removal-on-the-parquet-engine-completes-after-the-storage-engine-upgrade). On the upgraded storage engine, removal waits on the compactor, so a healthy cluster with a compaction backlog clears `removing` nodes slowly rather than never. See [Removal completes on the compactor’s schedule](#removal-completes-on-the-compactors-schedule)for how to measure the remaining gap. Check whether the compactor is making progress at all before assuming the removal is stuck: * Confirm the compactor node is `running` and isn’t restarting or being OOM-killed. * Watch the compaction floors in `system.pt_compaction_nodes` advance between queries. Don’t scale a replacement node into a removing node’s ID while you wait—the registration is refused until removal finishes. ### Nodes multiply in the catalog after each restart Each restart registered a new node ID. Check that your deployment assigns a stable node ID—see[Kubernetes and Helm](#kubernetes-and-helm). Stop and remove the stale entries after confirming which nodes are current. ### Writes fail during a rolling upgrade This is usually a catalog version constraint rather than a lifecycle problem. See [Troubleshooting cluster upgrades](/influxdb3/enterprise/admin/upgrade/#troubleshooting-cluster-upgrades). #### Related * [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) * [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) * [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/) * [Deploy InfluxDB 3 Enterprise on Kubernetes](/influxdb3/enterprise/install/kubernetes/) * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) * [influxdb3 remove node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/) * [influxdb3 show nodes](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/) * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) [clustering](/influxdb3/enterprise/tags/clustering/)[nodes](/influxdb3/enterprise/tags/nodes/)[lifecycle](/influxdb3/enterprise/tags/lifecycle/)[kubernetes](/influxdb3/enterprise/tags/kubernetes/)[wal](/influxdb3/enterprise/tags/wal/) | State | Description | | --- | --- | | State | Description | | running | The node started and registered itself in the catalog | | stopping | A graceful stop was requested, but the node hasn’t acknowledged it yet | | stopped | The node acknowledged its final snapshot and completed shutdown | | removing | The node is marked for permanent removal from the cluster | | Condition | Message | Resolution | | --- | --- | --- | | Condition | Message | Resolution | | Node isn’t stopped | node 'NODE_ID' is not fully stopped (current state: STATE) | Stop the node gracefully and wait for stopped | | Node runs in compact mode | node 'NODE_ID' has compact mode and cannot be removed | Compactor nodes can’t be removed—see Remove a compactor node | | Node belongs to a query group | cannot remove node 'NODE_ID' because it is a member of query group ... | Remove the node from the query group first | | Unsnapshotted WAL remains | node 'NODE_ID' has unsnapshotted WAL (wal file N, snapshotted through M) | Restart the node, stop it gracefully, then remove it—see Recover a crashed node | | Current state | Can register again? | | --- | --- | | Current state | Can register again? | | stopped | Yes—any instance can take over the node ID | | running or stopping | Only the same instance ID (an idempotent retry or a restart) | | removing | No— removing is terminal for the node ID | | Goal | Operation | | --- | --- | | Goal | Operation | | Rolling upgrade, config change, pod reschedule, host reboot | Restart the process with the same node ID—don’t remove the node | | Permanently scale down or decommission hardware | stop node , verify stopped , then remove node | | Replace a failed host, keeping its data | Start a node with the same node ID and object store configuration | --- # Configure object storage Source: https://docs.influxdata.com/influxdb3/enterprise/admin/object-storage/ InfluxDB 3 Enterprise can be configured to use different object storage providers to store time series data in Parquet format. The process of configuring and connecting to different object storage providers varies. The following guides walk through configuring, connecting to, and using different object storage providers as your InfluxDB 3 Enterprise object store. ### [Amazon S3](/influxdb3/enterprise/admin/object-storage/s3/) Use [Amazon S3](https://aws.amazon.com/s3/) as the object store for your InfluxDB 3 Enterprise instance. ### [MinIO](/influxdb3/enterprise/admin/object-storage/minio/) Use [MinIO](https://min.io) as the object store for your InfluxDB 3 Enterprise instance. InfluxDB uses the MinIO S3-compatible API to interact with your MinIO server or cluster. ### [Google Cloud Storage](/influxdb3/enterprise/admin/object-storage/gcs/) Use [Google Cloud Storage](https://cloud.google.com/storage) as the object store for your InfluxDB 3 Enterprise instance. ### [Azure Blob Storage](/influxdb3/enterprise/admin/object-storage/azure/) Use [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs)as the object store for your InfluxDB 3 Enterprise instance. ## Object store requirements InfluxDB 3 Enterprise uses the object store as the source of truth for catalog state. The catalog write path relies on conditional PUT (PUT-if-not-exists) to serialize catalog log writes, and every node depends on immediate visibility of writes made by any other node. Your object store deployment must provide the object store semantics InfluxDB 3 Enterprise depends on. ### Consistency semantics InfluxDB 3 Enterprise requires at least the following from any object store: * **Strong read-after-write consistency**: a `GET` immediately after a successful `PUT` returns the new object. * **Strong list-after-write consistency**: a `LIST` immediately after a successful `PUT` includes the new key. * **Conditional PUT (PUT-if-not-exists) semantics**: concurrent creates of the same key serialize so that exactly one write succeeds and the other returns`AlreadyExists`. A backend that violates these semantics can cause catalog split-brain, stale reads on node startup, and unexpected node-state warnings. ### Verify your object store InfluxDB 3 Enterprise 3.10.0 and later includes the[`influxdb3 debug object-store-check`](/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/)command that validates an object store against the preceding semantic requirements. Run it against your object store endpoint before putting the deployment into production, and again after any change to its topology or backing storage: ```bash influxdb3 debug object-store-check \ --object-store s3 \ --bucket influxdb3 \ --aws-endpoint http://localhost:9000 \ --aws-access-key-id MINIO_USERNAME \ --aws-secret-access-key MINIO_PASSWORD \ --aws-allow-http \ --check-prefix CHECK_PREFIX ``` Replace the following: * `CHECK_PREFIX`: a prefix for the tool’s synthetic test objects, such as `oscheck`.`--check-prefix` is required. 1. The tool writes and deletes synthetic test objects under`/oscheck-/`, where `oscheck-` is a fixed, tool-generated subdirectory name (not user-configurable) that’s unique to each run, and reports any semantic violation it finds. 2. If every check passes, your object store meets the semantics InfluxDB 3 Enterprise depends on. 3. If a check fails, resolve the reported issue in your object store or its configuration, and then rerun the command. 4. If the synthetic checks pass but a real catalog is still failing to load, add `--probe-prefix ` to also replay the loader’s object store operations against your real catalog. The probe itself is read-only, but the command still runs the synthetic write and delete checks under `--check-prefix` first, so that prefix must remain writable and the credentials need read, write, delete, and list permissions on it–for example, an AWS IAM policy needs `s3:GetObject`,`s3:PutObject`, `s3:DeleteObject`, and `s3:ListBucket`(see [Create IAM credentials](/influxdb3/enterprise/admin/object-storage/s3/#create-iam-credentials)for a sample policy). See [`influxdb3 debug object-store-check`](/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/)for the full flag reference. #### Related * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) [object storage](/influxdb3/enterprise/tags/object-storage/)[S3](/influxdb3/enterprise/tags/s3/) --- # Use Azure Blob Storage for object storage Source: https://docs.influxdata.com/influxdb3/enterprise/admin/object-storage/azure/ Use [Azure Blob Storage](https://azure.microsoft.com/products/storage/blobs)as the object store for your InfluxDB 3 Enterprise instance. * [Object store requirements](#object-store-requirements) * [Create a storage container](#create-a-storage-container) * [Get your storage account access key](#get-your-storage-account-access-key) * [Configure InfluxDB to connect to Azure Blob Storage](#configure-influxdb-to-connect-to-azure-blob-storage) * [Confirm the object store is working](#confirm-the-object-store-is-working) ## Object store requirements InfluxDB 3 Enterprise depends on strict object store consistency semantics— strong read-after-write and list-after-write consistency, and conditional PUT (PUT-if-not-exists) support. Your Azure Blob Storage container must provide these semantics. See [Object store requirements](../#object-store-requirements) for the full list of semantics InfluxDB 3 Enterprise depends on and how to verify your object store meets them. ## Create a storage container InfluxDB 3 Enterprise stores data in a container within an existing Azure Storage account. If you don’t already have a storage account, create one first. 1. Sign in to the [Azure portal](https://portal.azure.com) and go to your storage account. 2. Under **Data storage**, click **Containers**, and then click**+ Container**. 3. Enter a container name–for example, `influxdb3`–and click **Create**. Alternatively, use the[Azure CLI](https://learn.microsoft.com/cli/azure/storage/container) to create a container: ```bash az storage container create \ --name influxdb3 \ --account-name AZURE_STORAGE_ACCOUNT ``` ## Get your storage account access key InfluxDB 3 Enterprise authenticates to Azure Blob Storage with your storage account name and one of its access keys. 1. In the Azure portal, go to your storage account. 2. Under **Security + networking**, click **Access keys**. 3. Copy the storage account **name** and one of the **Key** values–you provide both to configure InfluxDB 3 Enterprise. > [!Tip] > In production, prefer[Azure Workload Identity](https://learn.microsoft.com/azure/aks/workload-identity-overview)over a static access key where your deployment environment supports it. ## Configure InfluxDB to connect to Azure Blob Storage To use your Azure container as the object store for your InfluxDB 3 Enterprise instance, provide the following options or environment variables with the`influxdb3 serve` command. The `--bucket` option specifies your Azure container name. [Command options](#command-options)[Environment variables](#environment-variables) * `--cluster-id`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `--node-id`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `--object-store`: `azure` * `--bucket`: `influxdb3` (your container name) * `--azure-storage-account`: Your storage account name (`AZURE_STORAGE_ACCOUNT`) * `--azure-storage-access-key`: One of your storage account’s access keys (`AZURE_STORAGE_ACCESS_KEY`) ``` influxdb3 serve \ --cluster-id INFLUXDB_CLUSTER_ID \ --node-id INFLUXDB_NODE_ID \ --object-store azure \ --bucket influxdb3 \ --azure-storage-account AZURE_STORAGE_ACCOUNT \ --azure-storage-access-key AZURE_STORAGE_ACCESS_KEY ``` * `INFLUXDB3_CLUSTER_ID`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `INFLUXDB3_NODE_ID`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `INFLUXDB3_OBJECT_STORE`: `azure` * `INFLUXDB3_BUCKET`: `influxdb3` (your container name) * `AZURE_STORAGE_ACCOUNT`: Your storage account name * `AZURE_STORAGE_ACCESS_KEY`: One of your storage account’s access keys ```bash export INFLUXDB3_CLUSTER_ID=INFLUXDB_CLUSTER_ID export INFLUXDB3_NODE_ID=INFLUXDB_NODE_ID export INFLUXDB3_OBJECT_STORE=azure export INFLUXDB3_BUCKET=influxdb3 export AZURE_STORAGE_ACCOUNT=AZURE_STORAGE_ACCOUNT export AZURE_STORAGE_ACCESS_KEY=AZURE_STORAGE_ACCESS_KEY influxdb3 serve ``` ## Confirm the object store is working When InfluxDB 3 Enterprise starts, it seeds your Azure container with the necessary directory structure and begins storing data there. Confirm the object store is functioning properly: 1. View the `influxdb3 serve` log output to confirm that the server is running correctly. 2. Inspect the contents of your container to confirm the necessary directory structure is created–for example, using the Azure CLI: ``` az storage blob list \ --container-name influxdb3 \ --account-name AZURE_STORAGE_ACCOUNT \ --output table ``` #### Related * [Create a container (Azure docs)](https://learn.microsoft.com/azure/storage/blobs/blob-containers-portal) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) [object storage](/influxdb3/enterprise/tags/object-storage/)[Azure](/influxdb3/enterprise/tags/azure/) --- # Use Google Cloud Storage for object storage Source: https://docs.influxdata.com/influxdb3/enterprise/admin/object-storage/gcs/ Use [Google Cloud Storage](https://cloud.google.com/storage) as the object store for your InfluxDB 3 Enterprise instance. * [Object store requirements](#object-store-requirements) * [Create a Cloud Storage bucket](#create-a-cloud-storage-bucket) * [Create a service account](#create-a-service-account) * [Configure InfluxDB to connect to Google Cloud Storage](#configure-influxdb-to-connect-to-google-cloud-storage) * [Confirm the object store is working](#confirm-the-object-store-is-working) ## Object store requirements InfluxDB 3 Enterprise depends on strict object store consistency semantics— strong read-after-write and list-after-write consistency, and conditional PUT (PUT-if-not-exists) support. Your Google Cloud Storage bucket must provide these semantics. See [Object store requirements](../#object-store-requirements) for the full list of semantics InfluxDB 3 Enterprise depends on and how to verify your object store meets them. ## Create a Cloud Storage bucket 1. Sign in to the[Google Cloud Storage console](https://console.cloud.google.com/storage/browser). 2. Click **Create bucket**. 3. Enter a bucket name–for example, `influxdb3`–and choose a location and storage class. 4. Keep the default access control settings unless your organization requires otherwise, and click **Create**. Alternatively, use the[gcloud CLI](https://cloud.google.com/sdk/gcloud/reference/storage/buckets/create)to create a bucket: ```bash gcloud storage buckets create gs://influxdb3 --project=GOOGLE_CLOUD_PROJECT ``` ## Create a service account Create a service account with permission to read from and write to your bucket, and download a JSON key for it: 1. In the[IAM & Admin console](https://console.cloud.google.com/iam-admin/serviceaccounts), create a new service account (or use an existing one). 2. Grant the service account the **Storage Object Admin**(`roles/storage.objectAdmin`) role on your bucket. 3. Under **Keys**, create a new JSON key and download it. Save the JSON key file–you provide its path to configure InfluxDB 3 Enterprise. > [!Tip] > In production, prefer[workload identity federation](https://cloud.google.com/iam/docs/workload-identity-federation)over a downloaded JSON key where your deployment environment supports it. ## Configure InfluxDB to connect to Google Cloud Storage To use your Cloud Storage bucket as the object store for your InfluxDB 3 Enterprise instance, provide the following options or environment variables with the `influxdb3 serve` command: [Command options](#command-options)[Environment variables](#environment-variables) * `--cluster-id`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `--node-id`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `--object-store`: `google` * `--bucket`: `influxdb3` * `--google-service-account`: Path to your service account JSON key file (`/path/to/service-account.json`) ``` influxdb3 serve \ --cluster-id INFLUXDB_CLUSTER_ID \ --node-id INFLUXDB_NODE_ID \ --object-store google \ --bucket influxdb3 \ --google-service-account /path/to/service-account.json ``` * `INFLUXDB3_CLUSTER_ID`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `INFLUXDB3_NODE_ID`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `INFLUXDB3_OBJECT_STORE`: `google` * `INFLUXDB3_BUCKET`: `influxdb3` * `GOOGLE_SERVICE_ACCOUNT`: Path to your service account JSON key file ```bash export INFLUXDB3_CLUSTER_ID=INFLUXDB_CLUSTER_ID export INFLUXDB3_NODE_ID=INFLUXDB_NODE_ID export INFLUXDB3_OBJECT_STORE=google export INFLUXDB3_BUCKET=influxdb3 export GOOGLE_SERVICE_ACCOUNT=/path/to/service-account.json influxdb3 serve ``` ## Confirm the object store is working When InfluxDB 3 Enterprise starts, it seeds your Cloud Storage bucket with the necessary directory structure and begins storing data there. Confirm the object store is functioning properly: 1. View the `influxdb3 serve` log output to confirm that the server is running correctly. 2. Inspect the contents of your bucket to confirm the necessary directory structure is created–for example, using the gcloud CLI: ``` gcloud storage ls gs://influxdb3 ``` #### Related * [Create a bucket (Google Cloud docs)](https://cloud.google.com/storage/docs/creating-buckets) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) [object storage](/influxdb3/enterprise/tags/object-storage/)[Google Cloud Storage](/influxdb3/enterprise/tags/google-cloud-storage/) --- # Use MinIO for object storage Source: https://docs.influxdata.com/influxdb3/enterprise/admin/object-storage/minio/ Use [MinIO](https://min.io) as the object store for your InfluxDB 3 Enterprise instance. InfluxDB uses the MinIO S3-compatible API to interact with your MinIO server or cluster. > > > MinIO is a high-performance, S3-compatible object storage solution released > under the GNU AGPL v3.0 license. Designed for speed and scalability, it powers > AI/ML, analytics, and data-intensive workloads with industry-leading performance. > > > > [MinIO GitHub repository](https://github.com/minio/minio?tab=readme-ov-file#readme) > > MinIO provides both an open source version ([MinIO Community Edition](https://min.io/open-source/download)) and an enterprise version ([MinIO AIStor](https://min.io/download)). While both can be used as your InfluxDB 3 Enterprise object store,**this guide walks through using MinIO Community Edition**. * [Object store requirements](#object-store-requirements) * [Set up MinIO](#set-up-minio) * [Configure InfluxDB to connect to MinIO](#configure-influxdb-to-connect-to-minio) * [Confirm the object store is working](#confirm-the-object-store-is-working) ## Object store requirements InfluxDB 3 Enterprise depends on strict object store consistency semantics— strong read-after-write and list-after-write consistency, and conditional PUT (PUT-if-not-exists) support. Your MinIO deployment must provide these semantics. See [Object store requirements](../#object-store-requirements) for the full list of semantics InfluxDB 3 Enterprise depends on and how to verify your object store meets them. > [!Warning] > #### Deployment topologies to avoid > > The following MinIO topologies can violate the semantics InfluxDB 3 Enterprise > depends on and are not supported for production: > > * **MinIO backed by NFS.** NFS attribute caching can return stale reads, > breaking read-after-write consistency. > Deploy MinIO on locally attached storage formatted with XFS. > * **Multiple independent MinIO instances sharing one backing volume**(for example, two containers mounting the same NFS export). If you need > multi-node MinIO, use its supported[distributed mode](https://min.io/docs/minio/linux/operations/install-deploy-manage/deploy-minio-multi-node-multi-drive.html)with erasure coding across nodes that each have their own local storage. > > For MinIO’s own guidance, see[on storage requirements](https://docs.min.io/enterprise/aistor-object-store/reference/aistor-server/requirements/storage),[against NFS](https://github.com/minio/minio/blob/master/docs/distributed/README.md#consistency-guarantees), > and[about primary storage requiring strict consistency](https://blog.min.io/strict-consistency-hard-requirement-for-primary-storage/). ## Set up MinIO 1. **Install and deploy a MinIO server or cluster**. You can install MinIO locally for testing and development or you can deploy a production MinIO cluster across multiple machines. The MinIO documentation provides detailed instructions for installing and deploying MinIO based on your target operating system: * [Install and deploy MinIO on **Linux**](https://min.io/docs/minio/linux/operations/installation.html)*(recommended for production deployments)* * [Install and deploy MinIO with **Kubernetes**](https://min.io/docs/minio/kubernetes/upstream/operations/installation.html) * [Install and deploy MinIO with **Docker**](https://min.io/docs/minio/container/operations/installation.html) * [Install and deploy MinIO on **macOS**](https://min.io/docs/minio/macos/operations/installation.html) * [Install and deploy MinIO on **Windows**](https://min.io/docs/minio/windows/operations/installation.html) 2. **Download and install the MinIO Client (`mc`)**. The MinIO client, or `mc` CLI, lets you perform administrative tasks on your MinIO server or cluster like creating users, assigning access policies, and more. Download and install the `mc` CLI for your local operating system and architecture. [See the **MinIO Client** section of the MinIO downloads page](https://min.io/open-source/download). 3. **Configure the `mc` CLI to connect to your MinIO server or cluster**. The `mc` CLI uses “aliases” to connect to a MinIO server or cluster. The alias refers to a set of connection credentials used to connect to and authorize with your MinIO server. Use the `mc alias set` command and provide the following: * **Alias**: A unique name or identifier for this credential set (`ALIAS`) * **MinIO URL**: The URL of your MinIO server or cluster (`https://localhost:9000`if running locally) * **Root username:** The root username you specified when setting up your MinIO server or cluster (`ROOT_USERNAME`) * **Root password**: The root password you specified when setting up your MinIO server or cluster (`ROOT_PASSWORD`) ``` mc alias set ALIAS http://localhost:9000 ROOT_USERNAME ROOT_PASSWORD ``` 4. **Create a MinIO bucket**. Use the *MinIO Console* or the *`mc mb` command* to create a new bucket in your MinIO server or cluster. [MinIO Console](#minio-console)[mc CLI](#mc-cli) The MinIO Console is a graphical user interface that lets you manage and browse buckets in your MinIO server or cluster. By default, the console is served on port`9001`. If running MinIO on your local machine, visit [http://localhost:9001](http://localhost:9001)to access the MinIO Console. If MinIO is running on a remote server, use your custom domain or IP to access the MinIO console. 1. In the Minio Console, click **Create Bucket**. 2. Enter a bucket name. For this guide, use `influxdb3`. 3. Click **Create Bucket**. Use the `mc mb` command to create a new MinIO bucket named `influxdb3`. Provide the MinIO alias configured in [step 3](#configure-alias) and the bucket name using the `ALIAS/BUCKET_NAME` syntax–for example: ``` mc mb ALIAS/influxdb3 ``` 5. **Create a MinIO user**. Use the `mc admin user add` command to create a new user. Provide the following: * **MinIO alias**: The MinIO server alias (created in [step 3](#configure-alias)) to add the user to (`ALIAS`) * **Username**: A unique username for the user (`MINIO_USERNAME`) * **Password**: A password for the user (`MINIO_PASSWORD`) ``` mc admin user add ALIAS MINIO_USERNAME MINIO_PASSWORD ``` MinIO user credentials are equivalent to credentials you would typically use to authorize with AWS S3: * A MinIO username is equivalent to an AWS access key ID * A MinIO password is equivalent to an AWS secret key 6. **Create an access policy that grants full access to the `influxdb3` bucket**. MinIO uses S3 compatible access policies to authorize access to buckets. To create a new access policy: 1. Create a file named `influxdb3-policy.json` that contains the following JSON: ``` { "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Effect": "Allow", "Resource": ["arn:aws:s3:::influxdb3"] }, { "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Effect": "Allow", "Resource": ["arn:aws:s3:::influxdb3/*"] } ] } ``` 2. Use the `mc admin policy create` command to create the new access policy in your MinIO server or cluster. Provide the following: * **MinIO alias**: The MinIO server alias (created in [step 3](#configure-alias)) to add the access policy to (`ALIAS`) * **Policy name**: A unique name for the policy (`POLICY_NAME`) * **Policy file**: The relative or absolute file path of your`influxdb3-policy.json` policy file (`/path/to/influxdb3-policy.json`) ``` mc admin policy create \ ALIAS \ POLICY_NAME \ /path/to/influxdb3-policy.json ``` 7. **Attach the access policy to your user.** Use the `mc admin policy attach` command to attach the access policy to your user. MinIO supports attaching access policies to both users and user groups. All users in a user group inherit policies attached to the group. For information about managing MinIO user groups, see[MinIO Group Management](https://min.io/docs/minio/linux/administration/identity-access-management/minio-group-management.html). Provide the following: * **MinIO alias**: The MinIO server alias created in [step 3](#configure-alias)(`ALIAS`) * **Policy name**: A unique username for the user (`POLICY_NAME`) * **Username** or **group name**: The user or user group to assign the policy to (`MINIO_USERNAME` or`MINIO_GROUP_NAME`) [user](#user)[group](#group) ``` mc admin policy attach ALIAS POLICY_NAME --user MINIO_USERNAME ``` ``` mc admin policy attach ALIAS POLICY_NAME --group MINIO_GROUP_NAME ``` Your MinIO server or cluster is now set up and ready to be used with InfluxDB 3 Enterprise. ## Configure InfluxDB to connect to MinIO To use your MinIO server or cluster as the object store for your InfluxDB 3 Enterprise instance, provide the following options or environment variables with the`influxdb3 serve` command: [Command options](#command-options)[Environment variables](#environment-variables) * `--cluster-id`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `--node-id`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `--object-store`: `s3` * `--bucket`: `influxdb3` * `--aws-endpoint`: Your MinIO URL (`http://localhost:9000` if running locally) * `--aws-access-key-id`: Your MinIO username (`MINIO_USERNAME`) * `--aws-secret-access-key`: Your MinIO password (`MINIO_PASSWORD`) * `--aws-allow-http`: *(Optional)* Include if *not* using HTTPS to connect to your MinIO server or cluster ``` influxdb3 serve \ --cluster-id INFLUXDB_CLUSTER_ID \ --node-id INFLUXDB_NODE_ID \ --object-store s3 \ --bucket influxdb3 \ --aws-endpoint http://localhost:9000 \ --aws-access-key-id MINIO_USERNAME \ --aws-secret-access-key MINIO_PASSWORD \ --aws-allow-http ``` * `INFLUXDB3_CLUSTER_ID`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `INFLUXDB3_NODE_ID`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `INFLUXDB3_OBJECT_STORE`: `s3` * `INFLUXDB3_BUCKET`: `influxdb3` * `AWS_ENDPOINT`: Your MinIO URL (`http://localhost:9000` if running locally) * `AWS_ACCESS_KEY_ID`: Your MinIO username (`MINIO_USERNAME`) * `AWS_SECRET_ACCESS_KEY`: Your MinIO password (`MINIO_PASSWORD`) * `AWS_ALLOW_HTTP`: *(Optional)* Set to `true` if *not* using HTTPS to connect to your MinIO server or cluster (default is `false`) ```bash export INFLUXDB3_CLUSTER_ID=INFLUXDB_CLUSTER_ID export INFLUXDB3_NODE_ID=INFLUXDB_NODE_ID export INFLUXDB3_OBJECT_STORE=s3 export INFLUXDB3_BUCKET=influxdb3 export AWS_ENDPOINT=http://localhost:9000 export AWS_ACCESS_KEY_ID=MINIO_USERNAME export AWS_SECRET_ACCESS_KEY=MINIO_PASSWORD export AWS_ALLOW_HTTP=true influxdb3 serve ``` ## Confirm the object store is working When InfluxDB 3 Enterprise starts, it will seed your MinIO object store with the necessary directory structure and begin storing data there. Confirm the object store is functioning properly: 1. View the `influxdb3 serve` log output to confirm that the server is running correctly. 2. Inspect the contents of your MinIO `influxdb3` bucket to confirm that the necessary directory structure is created. You can use the **MinIO Console**or the **`mc ls` command** to view the contents of a bucket–for example: ``` mc ls ALIAS/influxdb3 ``` #### Related * [Install and Deploy MinIO](https://min.io/docs/minio/linux/operations/installation.html) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) [object storage](/influxdb3/enterprise/tags/object-storage/)[S3](/influxdb3/enterprise/tags/s3/) --- # Use Amazon S3 for object storage Source: https://docs.influxdata.com/influxdb3/enterprise/admin/object-storage/s3/ Use [Amazon S3](https://aws.amazon.com/s3/) as the object store for your InfluxDB 3 Enterprise instance. Amazon S3 is the reference object storage implementation InfluxDB 3 Enterprise is built against and tested with. * [Object store requirements](#object-store-requirements) * [Create an S3 bucket](#create-an-s3-bucket) * [Create IAM credentials](#create-iam-credentials) * [Configure InfluxDB to connect to S3](#configure-influxdb-to-connect-to-s3) * [Confirm the object store is working](#confirm-the-object-store-is-working) ## Object store requirements InfluxDB 3 Enterprise depends on strict object store consistency semantics— strong read-after-write and list-after-write consistency, and conditional PUT (PUT-if-not-exists) support. Amazon S3 provides these semantics. See [Object store requirements](../#object-store-requirements) for the full list of semantics InfluxDB 3 Enterprise depends on and how to verify your object store meets them. ## Create an S3 bucket 1. Sign in to the [Amazon S3 console](https://console.aws.amazon.com/s3/). 2. Click **Create bucket**. 3. Enter a bucket name–for example, `influxdb3`–and choose an AWS Region. 4. Keep the default **Block Public Access** and versioning settings unless your organization requires otherwise, and click **Create bucket**. Alternatively, use the[AWS CLI](https://docs.aws.amazon.com/cli/latest/reference/s3/mb.html) to create a bucket: ```bash aws s3 mb s3://influxdb3 --region AWS_REGION ``` ## Create IAM credentials Create an IAM user or role with permission to read from and write to your bucket, and generate an access key for it: 1. In the [IAM console](https://console.aws.amazon.com/iam/), create a new user (or use an existing one) and attach a policy that grants access to your bucket–for example: ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:GetBucketLocation", "s3:ListBucket"], "Resource": ["arn:aws:s3:::influxdb3"] }, { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"], "Resource": ["arn:aws:s3:::influxdb3/*"] } ] } ``` 2. Generate an access key for the user under **Security credentials**. Save the access key ID and secret access key–you use them to configure InfluxDB 3 Enterprise. > [!Tip] > In production, prefer an IAM role (for example, an EC2 instance profile or > IAM roles for service accounts on EKS) over long-lived access keys where > your deployment environment supports it. ## Configure InfluxDB to connect to S3 To use your S3 bucket as the object store for your InfluxDB 3 Enterprise instance, provide the following options or environment variables with the`influxdb3 serve` command: [Command options](#command-options)[Environment variables](#environment-variables) * `--cluster-id`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `--node-id`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `--object-store`: `s3` * `--bucket`: `influxdb3` * `--aws-access-key-id`: Your access key ID (`AWS_ACCESS_KEY_ID`) * `--aws-secret-access-key`: Your secret access key (`AWS_SECRET_ACCESS_KEY`) * `--aws-default-region`: *(Optional)* Your bucket’s AWS Region (`AWS_REGION`); defaults to `us-east-1` ``` influxdb3 serve \ --cluster-id INFLUXDB_CLUSTER_ID \ --node-id INFLUXDB_NODE_ID \ --object-store s3 \ --bucket influxdb3 \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --aws-default-region AWS_REGION ``` * `INFLUXDB3_CLUSTER_ID`: Your InfluxDB 3 Enterprise cluster ID (`INFLUXDB_CLUSTER_ID`) * `INFLUXDB3_NODE_ID`: Your InfluxDB 3 Enterprise node ID (`INFLUXDB_NODE_ID`) * `INFLUXDB3_OBJECT_STORE`: `s3` * `INFLUXDB3_BUCKET`: `influxdb3` * `AWS_ACCESS_KEY_ID`: Your access key ID * `AWS_SECRET_ACCESS_KEY`: Your secret access key * `AWS_DEFAULT_REGION`: *(Optional)* Your bucket’s AWS Region; defaults to`us-east-1` ```bash export INFLUXDB3_CLUSTER_ID=INFLUXDB_CLUSTER_ID export INFLUXDB3_NODE_ID=INFLUXDB_NODE_ID export INFLUXDB3_OBJECT_STORE=s3 export INFLUXDB3_BUCKET=influxdb3 export AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID export AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY export AWS_DEFAULT_REGION=AWS_REGION influxdb3 serve ``` ## Confirm the object store is working When InfluxDB 3 Enterprise starts, it seeds your S3 bucket with the necessary directory structure and begins storing data there. Confirm the object store is functioning properly: 1. View the `influxdb3 serve` log output to confirm that the server is running correctly. 2. Inspect the contents of your bucket to confirm the necessary directory structure is created–for example, using the AWS CLI: ``` aws s3 ls s3://influxdb3 ``` #### Related * [Create a bucket (Amazon S3 docs)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/creating-bucket.html) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) [object storage](/influxdb3/enterprise/tags/object-storage/)[S3](/influxdb3/enterprise/tags/s3/) --- # Orphaned file cleanup Source: https://docs.influxdata.com/influxdb3/enterprise/admin/orphaned-file-cleanup/ InfluxDB 3 Enterprise clusters running the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)automatically find and delete compacted files in object storage that nothing published references, known as **orphaned files**. Compaction writes output files and then publishes references to them; a crash, a lease change, or a superseded write between those two steps can leave files that nothing ever references. Orphaned file cleanup, also called the orphan sweep, is what reclaims that storage over time. Orphaned file cleanup runs only on the primary compactor, the same node that plans and dispatches compaction work. With [distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/)enabled, orphaned files also come from ordinary distributed-tier failure handling, such as a worker that dies mid-job, so cleanup is part of running that feature safely. * [What gets deleted](#what-gets-deleted) * [Grace period](#grace-period) * [Cadence and the first pass](#cadence-and-the-first-pass) * [Dry run and turning cleanup off](#dry-run-and-turning-cleanup-off) * [Restores pause cleanup](#restores-pause-cleanup) * [Inspect cleanup passes](#inspect-cleanup-passes) ## What gets deleted A cleanup pass only considers objects under the prefixes the compactor writes run-set data and index files to. Ingest snapshots, the write-ahead log, the catalog, import staging, the compactor lease, and checkpoints are never candidates, because each has its own lifecycle and its own owner. Within that scope, an object is a deletion candidate only when it’s both: * **Unreferenced**: absent from the published checkpoint, the delete queue, every table or shard with in-flight or committed-but-unpublished work, and every run-set ID reserved by an outstanding compaction job. * **Past the grace period**: its object store `last_modified` timestamp is older than the pass’s grace window (see [Grace period](#grace-period)). ## Grace period Objects modified more recently than the grace period are never deletion candidates, no matter what else is true about them. The grace period defaults to `24h` and has a floor of `6h`: | Option |Default| Notes | |-------------------------|-------|---------------------------------------------------------------------------------------------| |`--compactor-sweep-grace`| `24h` |Values below `6h` are raised to `6h`. Environment variable: `INFLUXDB3_COMPACTOR_SWEEP_GRACE`| The grace period is a backstop, not the primary safety mechanism, since in-flight work is already protected because it’s tracked as referenced. The grace period mainly absorbs clock skew between the primary and the object store, and gives lingering multipart uploads time to complete. ## Cadence and the first pass A scheduled cleanup pass runs every `7d` by default: | Option |Default| Notes | |----------------------------|-------|----------------------------------------------------------------------------------------------------| |`--compactor-sweep-interval`| `7d` |Set to `off` to disable scheduled passes. Environment variable: `INFLUXDB3_COMPACTOR_SWEEP_INTERVAL`| The **first pass on a 3.12 cluster starts a few minutes after the primary compactor starts**, rather than waiting a full interval, because no prior pass is recorded yet. After that, passes repeat on the configured interval. The cadence survives restarts: if the newest recorded pass is older than the interval, a pass fires on the next startup check, and an unfinished scheduled pass resumes automatically instead of waiting out the full interval. Audit-trail objects for completed passes are retained for review: | Option |Default| Notes | |-----------------------------------|-------|------------------------------------------------------------------------------------------------------------------------------------------------| |`--compactor-sweep-audit-retention`| `7d` |Older audit objects are deleted as housekeeping at the start of the next pass. Environment variable: `INFLUXDB3_COMPACTOR_SWEEP_AUDIT_RETENTION`| ## Dry run and turning cleanup off Set the cleanup mode with `--compactor-sweep-mode`: | Option |Default| Notes | |------------------------|-------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`--compactor-sweep-mode`|`armed`|`armed` enqueues candidates into the delete queue, so orphaned files are actually deleted. `dry-run` only records candidates in the audit trail. Environment variable: `INFLUXDB3_COMPACTOR_SWEEP_MODE`| Use `dry-run` to review what a pass would delete before you trust it to delete anything. Compare the candidates it reports against what you expect using[`influxdb3 debug sweep report`](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/report/). To turn scheduled cleanup off entirely, set: ```bash influxdb3 serve \ # ... --compactor-sweep-interval off ``` ## Restores pause cleanup Cleanup passes don’t run while a[restore](/influxdb3/enterprise/admin/backup-restore/) is in progress. The compactor holds every delete, including orphaned-file candidates, until the restore reaches it and completes, so a cleanup pass can never delete a file a restore just copied back. A pass that was running when a restore started abandons at its next segment boundary and picks back up on the next scheduled fire. ## Inspect cleanup passes Two commands read cleanup activity directly from object storage. Both are offline and read-only, and neither needs a running InfluxDB 3 Enterprise server: * [`influxdb3 debug sweep list`](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/list/)names the recorded cleanup passes for a cluster, oldest to newest. * [`influxdb3 debug sweep report`](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/report/)reads one pass’s audit trail back and summarizes it, or, with `--json`, emits the full per-segment detail as a diffable artifact. A third command checks the opposite direction, confirming that nothing a checkpoint references is missing, which is useful after an incident or a migration, independent of whether cleanup ever ran: * [`influxdb3 debug verify-references`](/influxdb3/enterprise/reference/cli/influxdb3/debug/verify-references/)checks that every object the newest compactor checkpoint references still exists, and exits nonzero if anything is missing, so you can gate automation on its exit code. #### Related * [Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/) * [influxdb3 debug sweep](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/) * [influxdb3 debug verify-references](/influxdb3/enterprise/reference/cli/influxdb3/debug/verify-references/) * [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) [compaction](/influxdb3/enterprise/tags/compaction/)[storage](/influxdb3/enterprise/tags/storage/)[maintenance](/influxdb3/enterprise/tags/maintenance/) | Option | Default | Notes | | --- | --- | --- | | Option | Default | Notes | | --compactor-sweep-grace | 24h | Values below 6h are raised to 6h . Environment variable: INFLUXDB3_COMPACTOR_SWEEP_GRACE | | Option | Default | Notes | | --- | --- | --- | | Option | Default | Notes | | --compactor-sweep-interval | 7d | Set to off to disable scheduled passes. Environment variable: INFLUXDB3_COMPACTOR_SWEEP_INTERVAL | | Option | Default | Notes | | --- | --- | --- | | Option | Default | Notes | | --compactor-sweep-audit-retention | 7d | Older audit objects are deleted as housekeeping at the start of the next pass. Environment variable: INFLUXDB3_COMPACTOR_SWEEP_AUDIT_RETENTION | | Option | Default | Notes | | --- | --- | --- | | Option | Default | Notes | | --compactor-sweep-mode | armed | armed enqueues candidates into the delete queue, so orphaned files are actually deleted. dry-run only records candidates in the audit trail. Environment variable: INFLUXDB3_COMPACTOR_SWEEP_MODE | --- # Performance tuning Source: https://docs.influxdata.com/influxdb3/enterprise/admin/performance-tuning/ Configure thread allocation, memory settings, and other parameters to optimize InfluxDB 3 Enterprise performance based on your workload characteristics. * [Best practices](#best-practices) * [General monitoring principles](#general-monitoring-principles) * [Essential settings for performance](#essential-settings-for-performance) * [Common performance issues](#common-performance-issues) * [Configuration examples by workload](#configuration-examples-by-workload) * [Thread allocation details](#thread-allocation-details) * [Enterprise mode-specific tuning](#enterprise-mode-specific-tuning) * [Memory tuning](#memory-tuning) * [Advanced tuning options](#advanced-tuning-options) * [Startup optimization](#startup-optimization) * [Monitoring and validation](#monitoring-and-validation) * [Common performance issues](#common-performance-issues-1) ## Best practices 1. **Start with monitoring**: Understand your current bottlenecks before tuning 2. **Change one parameter at a time**: Isolate the impact of each change 3. **Test with production-like workloads**: Use realistic data and query patterns 4. **Document your configuration**: Keep track of what works for your workload 5. **Plan for growth**: Leave headroom for traffic increases 6. **Regular review**: Periodically reassess as workloads evolve ## General monitoring principles Before tuning performance, establish baseline metrics to identify bottlenecks: ### Key metrics to monitor 1. **CPU usage per core** * Monitor individual core utilization to identify thread pool imbalances * Watch for cores at 100% while others are idle (indicates thread allocation issues) * Use `top -H` or `htop` to view per-thread CPU usage 2. **Memory consumption** * Track heap usage vs available RAM * Monitor query execution memory pool utilization * Watch for OOM errors or excessive swapping 3. **IO and network** * Measure write throughput (points/second) * Track query response times * Monitor object store latency for cloud deployments * Check disk IO wait times with `iostat` On clusters running the upgraded storage engine (the default for new clusters in 3.11+), use these thresholds: | Metric | Good |Warning | Action | |---------------------------|-------|--------|------------------------------------------------------| | Cache hit rate | \>80% | \<60% |Increase `--file-cache-size` or `--file-cache-recency`| |Rows read vs returned ratio|\<100:1|\>1000:1| Add more selective predicates | | WAL file count | \<50 | \>100 | Increase `--wal-flush-concurrency` | | Gen0 file count | \<100 | \>200 | Increase `--compactor-input-size-budget` | For background on how these metrics relate to the storage engine’s internals, see[Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). ### Establish baselines ```bash # Monitor CPU per thread top -H -p $(pgrep influxdb3) # Track memory usage free -h watch -n 1 "free -h" # Check IO wait iostat -x 1 ``` > [!Tip] > For comprehensive metrics monitoring, see [Monitor metrics](/influxdb3/enterprise/admin/monitor-metrics/). ## Essential settings for performance Use the following to tune performance in *all-in-one* deployments: > [!Note] > For specialized cluster nodes (ingest-only, query-only, etc.), see [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) for mode-specific optimizations. ### Thread allocation (–num-io-threads, –datafusion-num-threads ) **IO threads** handle HTTP requests and line protocol parsing. **Default: 2** (often insufficient).**DataFusion threads** process queries and snapshots. > [!Note] > InfluxDB 3 Enterprise automatically allocates remaining cores to DataFusion after reserving IO threads. You can configure both thread pools explicitly by setting the `--num-io-threads` and `--datafusion-num-threads` options. > [!Important] > #### Thread defaults on the upgraded storage engine > > The 2-IO-thread default and the manual thread-allocation guidance in this > section apply to Parquet-engine clusters (clusters that started on 3.10 or > earlier that have not run the storage engine upgrade). > On clusters running the upgraded storage engine (the default for new > clusters), the IO and DataFusion > runtimes each default to the licensed core count, and thread counts set > above the licensed core count are capped with a startup warning. > An at-home license always runs 1 IO thread and 1 DataFusion thread. ```bash # Write-heavy: More IO threads, adequate DataFusion influxdb3 --num-io-threads=12 serve \ --datafusion-num-threads=20 \ --node-id=node0 --cluster-id=cluster0 \ --object-store=file --data-dir=~/.influxdb3 # Query-heavy: Fewer IO threads, more DataFusion influxdb3 --num-io-threads=4 serve \ --datafusion-num-threads=28 \ --node-id=node0 --cluster-id=cluster0 \ --object-store=file --data-dir=~/.influxdb3 ``` > [!Warning] > #### Increase IO threads for concurrent writers > > If you have multiple concurrent writers (for example, Telegraf agents), the default of 2 IO threads can bottleneck write performance. ### Memory pool (–exec-mem-pool-size) Controls memory for query execution. Default: 20% of RAM. ```bash # Increase for query-heavy workloads influxdb3 serve \ --exec-mem-pool-size=90% \ --node-id=node0 --cluster-id=cluster0 \ --object-store=file --data-dir=~/.influxdb3 # Decrease if experiencing memory pressure influxdb3 serve \ --exec-mem-pool-size=60% \ --node-id=node0 --cluster-id=cluster0 \ --object-store=file --data-dir=~/.influxdb3 ``` ### Data file cache (–file-cache-size) Caches frequently accessed data files in memory. ```bash # Enable caching for better query performance influxdb3 serve \ --file-cache-size=4GB \ --node-id=node0 --cluster-id=cluster0 \ --object-store=file --data-dir=~/.influxdb3 ``` ### WAL flush interval (–wal-flush-interval) Controls write latency vs throughput. Default: 1s. ```bash # Reduce latency for real-time data influxdb3 serve \ --wal-flush-interval=100ms \ --node-id=node0 --cluster-id=cluster0 \ --object-store=file --data-dir=~/.influxdb3 ``` ## Common performance issues ### High write latency **Symptoms:** Increasing write response times, timeouts, points dropped **Solutions:** 1. Increase [IO threads](#thread-allocation-num-io-threads-datafusion-num-threads) (default is only 2) 2. Reduce [WAL flush interval](#wal-flush-interval-wal-flush-interval) (from 1s to 100ms) 3. Check disk IO performance ### Slow query performance **Symptoms:** Long execution times, high memory usage, query timeouts **Solutions:** 1. Increase [DataFusion threads](#thread-allocation-num-io-threads-datafusion-num-threads) 2. Increase [execution memory pool](#memory-pool-exec-mem-pool-size) (to 90%) 3. Enable [data file caching](#data-file-cache-file-cache-size) ### Memory pressure **Symptoms:** OOM errors, swapping, high memory usage **Solutions:** 1. Reduce [execution memory pool](#memory-pool-exec-mem-pool-size) (to 60%) 2. Lower snapshot threshold (`--force-snapshot-mem-size=70%`) ### CPU bottlenecks **Symptoms:** 100% CPU utilization, uneven thread usage (only 2 cores for writes) **Solutions:** 1. Rebalance [thread allocation](#thread-allocation-num-io-threads-datafusion-num-threads) 2. Check if only 2 cores are used for write parsing (increase IO threads) > [!Important] > #### “My ingesters are only using 2 cores” > > Increase `--num-io-threads` to 8-16+ for ingest nodes. For dedicated ingest nodes with `--mode=ingest`, see [Configure ingest nodes](/influxdb3/enterprise/admin/clustering/#configure-ingest-nodes). The following issues are specific to the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters in 3.11+). ### High WAL file count (upgraded storage engine) **Symptoms:** `system.pt_ingest_wal` shows many accumulated files **Possible causes:** Merge operations falling behind write rate, insufficient flush concurrency, object storage latency. **Solutions:** 1. Increase flush concurrency: `--wal-flush-concurrency=8` 2. Increase WAL flush interval to create larger, fewer files:`--wal-flush-interval=5s` 3. Increase the WAL buffer size so each flush produces a larger file:`--wal-buffer-size=30MB` 4. Check object storage performance and connectivity. ### High cache miss rate (upgraded storage engine) **Symptoms:** `cache_stats` from the query telemetry endpoint shows \>40% miss rate **Possible causes:** Cache size too small for working set, cache recency window too narrow, random access patterns across time ranges. **Solutions:** 1. Increase cache size: `--file-cache-size=16GB` 2. Extend cache recency window: `--file-cache-recency=24h` 3. Extend eviction timeout: `--file-cache-evict-after=48h` ### Slow compaction (upgraded storage engine) **Symptoms:** Gen0 file count continues to grow **Possible causes:** Compaction budget too low for write volume, high write rate overwhelming compaction, snapshot size too large creating oversized Gen0 files. **Solutions:** 1. Increase the compaction input size budget:`--compactor-input-size-budget=12GB` 2. Reduce snapshot size to create smaller, more frequent Gen0 files:`--snapshot-size=125MB` 3. For distributed deployments, add a dedicated compactor node (`--mode=compact`). ### Query node lag (upgraded storage engine) **Symptoms:** Query nodes return stale data **Possible causes:** Replication falling behind, network latency to object storage, insufficient replica concurrency. **Solutions:** 1. Increase replication concurrency: `--wal-replica-steady-concurrency=8` 2. Reduce the replication polling interval: `--replication-interval=100ms` 3. Increase replica queue length: `--wal-replica-queue-length=200` For a full list of replication options, see[Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/#replication-query-nodes). ## Configuration examples by workload ### Write-heavy workloads (\>100k points/second) ```bash # 32-core system, high ingest rate influxdb3 --num-io-threads=12 serve \ --datafusion-num-threads=20 \ --exec-mem-pool-size=80% \ --wal-flush-interval=100ms \ --node-id=node0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` ### Query-heavy workloads (complex analytics) ```bash # 32-core system, analytical queries influxdb3 --num-io-threads=4 serve \ --datafusion-num-threads=28 \ --exec-mem-pool-size=90% \ --file-cache-size=2GB \ --node-id=node0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` ### Mixed workloads (real-time dashboards) ```bash # 32-core system, balanced operations influxdb3 --num-io-threads=8 serve \ --datafusion-num-threads=24 \ --exec-mem-pool-size=70% \ --file-cache-size=1GB \ --node-id=node0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` ## Thread allocation details ### Calculate optimal thread counts Use this formula as a starting point: ``` Total cores = N Concurrent writers = W Query complexity factor = Q (1-10, where 10 is most complex) IO threads = min(W + 2, N * 0.4) DataFusion threads = N - IO threads ``` > [!Important] > This formula applies to the Parquet storage engine, where > DataFusion threads handle queries, snapshots, and compaction. It doesn’t > apply to the [upgraded storage engine](/influxdb3/enterprise/reference/storage-engine-config-options/), > which runs ingest and compaction on the IO runtime and requires setting > both thread options explicitly. ### Example configurations by system size #### Small system (4 cores, 16 GB RAM) ```bash # Balanced configuration influxdb3 --num-io-threads=2 serve \ --exec-mem-pool-size=10GB \ --file-cache-size=500MB \ --node-id=node0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` #### Medium system (16 cores, 64 GB RAM) ```bash # Write-optimized configuration influxdb3 --num-io-threads=6 serve \ --datafusion-num-threads=10 \ --exec-mem-pool-size=45GB \ --file-cache-size=2GB \ --node-id=node0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` #### Large system (64 cores, 256 GB RAM) ```bash # Query-optimized configuration influxdb3 --num-io-threads=8 serve \ --datafusion-num-threads=56 \ --exec-mem-pool-size=200GB \ --file-cache-size=10GB \ --object-store-connection-limit=200 \ --node-id=node0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` ## Enterprise mode-specific tuning ### Ingest mode optimization Dedicated ingest nodes require significant IO threads: ```bash # High-throughput ingester (96 cores) influxdb3 --num-io-threads=24 serve \ --mode=ingest \ --num-cores=96 \ --datafusion-num-threads=72 \ --force-snapshot-mem-size=90% \ --node-id=ingester0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` > [!Warning] > Without explicitly setting `--num-io-threads`, a 96-core ingester uses only 2 cores > for parsing line protocol, wasting 94% of available CPU for ingest operations. ### Query mode optimization Query nodes should maximize DataFusion threads: ```bash # Query-optimized node (64 cores) influxdb3 --num-io-threads=4 serve \ --mode=query \ --num-cores=64 \ --datafusion-num-threads=60 \ --exec-mem-pool-size=90% \ --file-cache-size=4GB \ --node-id=query0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 ``` ### Compactor mode optimization Compaction is DataFusion-intensive: ```bash # Dedicated compactor (32 cores) influxdb3 --num-io-threads=2 serve \ --mode=compact \ --num-cores=32 \ --datafusion-num-threads=30 \ --node-id=compactor0 \ --cluster-id=cluster0 \ --object-store=file \ --data-dir=~/.influxdb3 # Note: --compaction-row-limit option is not yet released in v3.5.0 # Uncomment when available in a future release: # --compaction-row-limit=1000000 \ ``` ## Memory tuning ### Execution memory pool Configure the query execution memory pool: ```bash # Absolute value with a unit suffix --exec-mem-pool-size=8gb # Percentage of available RAM --exec-mem-pool-size=80% # 80% of system RAM ``` **Guidelines:** * **Write-heavy**: 60-70% (leave room for OS cache) * **Query-heavy**: 80-90% (maximize query memory) * **Mixed**: 70% (balanced approach) ### Data file cache configuration Cache frequently accessed data files: ```bash # Set cache size --file-cache-size=2gb # Configure cache behavior --parquet-mem-cache-prune-interval=1m \ --parquet-mem-cache-prune-percentage=20 ``` ### WAL and snapshot tuning Control memory pressure from write buffers: ```bash # Force snapshot when memory usage exceeds threshold --force-snapshot-mem-size=80% # Configure WAL rotation --wal-flush-interval=10s \ --wal-files-per-snapshot=100 ``` ## Advanced tuning options ### Specialized cluster nodes For performance optimizations using dedicated ingest, query, compaction, or processing nodes, see [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/). For less common performance optimizations and detailed configuration options, see: ### DataFusion engine tuning Advanced DataFusion runtime parameters: * [`--datafusion-config`](/influxdb3/enterprise/reference/cli/influxdb3/serve/#datafusion-config) ### HTTP and network tuning Request size and network optimization: * [`--max-http-request-size`](/influxdb3/enterprise/reference/cli/influxdb3/serve/#max-http-request-size) - For large batches (default: 10 MB) * [`--http-bind`](/influxdb3/enterprise/reference/cli/influxdb3/serve/#http-bind) - Bind address ### Object store optimization Performance tuning for cloud object stores: * [`--object-store-connection-limit`](/influxdb3/enterprise/reference/cli/influxdb3/serve/#object-store-connection-limit) - Connection pool size * [`--object-store-max-retries`](/influxdb3/enterprise/reference/cli/influxdb3/serve/#object-store-max-retries) - Retry configuration * [`--object-store-http2-only`](/influxdb3/enterprise/reference/cli/influxdb3/serve/#object-store-http2-only) - Force HTTP/2 ### Complete configuration reference For all available configuration options, see: * [CLI serve command reference](/influxdb3/enterprise/reference/cli/influxdb3/serve/) * [Configuration options](/influxdb3/enterprise/reference/config-options/) ## Startup optimization Server startup time scales with the number of[snapshots](/influxdb3/enterprise/admin/backup-restore/#file-structure)stored in the object store. Snapshots accumulate over time and are not automatically deleted. Without checkpointing, the server loads individual snapshots on startup. The number of snapshots is determined by the lookback window ([`gen1-lookback-duration`](/influxdb3/enterprise/reference/config-options/#gen1-lookback-duration), default 1 month) divided by[`gen1-duration`](/influxdb3/enterprise/reference/config-options/#gen1-duration)(default 10 minutes), with a minimum of 100. With default settings, a long-running server can accumulate up to \~4,320 snapshots, causing slow restarts. Two configuration options reduce startup time: * [`--checkpoint-interval`](/influxdb3/enterprise/reference/config-options/#checkpoint-interval)– periodically consolidates snapshot metadata into monthly checkpoints. On startup, the server loads one to two checkpoints per calendar month, then loads only snapshots created since the last checkpoint. * [`--gen1-lookback-duration`](/influxdb3/enterprise/reference/config-options/#gen1-lookback-duration)– limits how far back the server loads gen1 file index metadata on startup. Files outside this window still exist in object storage but are not indexed. > [!Note] > Enabling checkpointing does not delete old snapshots. > They remain in object storage but are no longer needed for startup. ### Recommended checkpoint intervals | Scenario |Recommended interval| |---------------------|--------------------| | Production servers | `1h` | |Development / testing| `10m` | ### Enable checkpoint creation ```bash influxdb3 serve --checkpoint-interval 1h ``` For all checkpoint configuration options, see[checkpoint-interval](/influxdb3/enterprise/reference/config-options/#checkpoint-interval). ## Monitoring and validation ### Monitor thread utilization ```bash # Linux: View per-thread CPU usage top -H -p $(pgrep influxdb3) # Monitor specific threads watch -n 1 "ps -eLf | grep influxdb3 | head -20" ``` ### Check performance metrics Monitor key indicators: ```sql -- Query system.threads table (Enterprise) SELECT * FROM system.threads WHERE cpu_usage > 90 ORDER BY cpu_usage DESC; -- Check write throughput SELECT count(*) as points_written, max(timestamp) - min(timestamp) as time_range FROM your_measurement WHERE timestamp > now() - INTERVAL '1 minute'; ``` ### Validate configuration Verify your tuning changes: ```bash # Check effective configuration influxdb3 serve --help-all | grep -E "num-io-threads|num-datafusion-threads" # Monitor memory usage free -h watch -n 1 "free -h" # Check IO wait iostat -x 1 ``` ## Common performance issues ### High write latency **Symptoms:** * Increasing write response times * Timeouts from write clients * Points dropped or rejected **Solutions:** 1. Increase IO threads: `--num-io-threads=16` 2. Reduce batch sizes in writers 3. Increase WAL flush frequency 4. Check disk IO performance ### Slow query performance **Symptoms:** * Long query execution times * High memory usage during queries * Query timeouts **Solutions:** 1. Increase DataFusion threads: `--datafusion-num-threads=30` 2. Increase execution memory pool: `--exec-mem-pool-size=90%` 3. Enable data file caching: `--file-cache-size=4GB` 4. Optimize query patterns (smaller time ranges, fewer fields) ### Memory pressure **Symptoms:** * Out of memory errors * Frequent garbage collection * System swapping **Solutions:** 1. Reduce execution memory pool: `--exec-mem-pool-size=60%` 2. Lower snapshot threshold: `--force-snapshot-mem-size=70%` 3. Decrease cache sizes 4. Add more RAM or reduce workload ### CPU bottlenecks **Symptoms:** * 100% CPU utilization * Uneven thread pool usage * Performance plateaus **Solutions:** 1. Rebalance thread allocation based on workload 2. Add more CPU cores 3. Optimize client batching 4. Distribute workload across specialized nodes #### Related * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) * [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) * [Query system data](/influxdb3/enterprise/admin/query-system-data/) | Metric | Good | Warning | Action | | --- | --- | --- | --- | | Metric | Good | Warning | Action | | Cache hit rate | >80% | <60% | Increase --file-cache-size or --file-cache-recency | | Rows read vs returned ratio | <100:1 | >1000:1 | Add more selective predicates | | WAL file count | <50 | >100 | Increase --wal-flush-concurrency | | Gen0 file count | <100 | >200 | Increase --compactor-input-size-budget | | Scenario | Recommended interval | | --- | --- | | Scenario | Recommended interval | | Production servers | 1h | | Development / testing | 10m | --- # Configure a standalone process node Source: https://docs.influxdata.com/influxdb3/enterprise/admin/processing-engine/ A *standalone process node* is a node that runs only in `process` mode. It runs [Processing Engine](/influxdb3/enterprise/reference/processing-engine/) plugins. It doesn’t store writes or answer data queries; the other nodes in the cluster do that work. In a cluster, we recommend running the Processing Engine on a standalone process node. This keeps plugin CPU and memory use off your ingest and query nodes. You can also restart the process node, or [limit its resources](/influxdb3/enterprise/admin/security/harden-the-host/#example-add-memory-cpu-and-io-control-for-process-node), without affecting the other nodes. This page covers the standalone process node only. For an overview of process node architectures, see [Configure a process node](/influxdb3/enterprise/admin/clustering/#configure-a-process-node). > [!Important] > #### Requires InfluxDB 3 Enterprise 3.12.0 or later > > Before 3.12.0, a process node can’t read WAL files from other nodes, so WAL triggers on a process node never run. > The trigger is accepted, and the node logs `Ingest mode not configured; processing engine will not run wal flush triggers.` * [How a process node runs plugins](#how-a-process-node-runs-plugins) * [Plan the cluster](#plan-the-cluster) * [Start the nodes](#start-the-nodes) * [Verify the cluster](#verify-the-cluster) * [Add plugins and packages](#add-plugins-and-packages) * [Create triggers](#create-triggers) * [Send each request to the right node](#send-each-request-to-the-right-node) * [Manage triggers](#manage-triggers) * [Stop and restart the process node](#stop-and-restart-the-process-node) * [Troubleshoot](#troubleshoot) ## How a process node runs plugins The process node starts each trigger type differently: * **WAL triggers** (`table:` or `all_tables`): The process node reads each ingest node’s WAL files from object storage and passes new rows to the plugin’s `process_writes()` function. * **Schedule triggers** (`every:` or `cron:`): The process node calls the plugin’s `process_scheduled_call()` function on the schedule. * **Request triggers** (`request:`): The process node serves `/api/v3/engine/` and calls the plugin’s `process_request()` function for each request. For every trigger type, the plugin reaches data through other nodes over the internode protocol: * `influxdb3_local.query()` runs the query on a query node. * `influxdb3_local.write_sync()` and the other write methods send data to an ingest node. * `influxdb3_local.info()`, `warn()`, and `error()` messages are written to `system.processing_engine_logs` through an ingest node. ## Plan the cluster A cluster with a process node needs at least two nodes: the process node and one or more nodes that run `ingest` and `query`. All nodes share one object store, one `--cluster-id`, and one license. Start every ingest node and every query node with `--internode-bind-addr`: * **Ingest nodes**: Receive plugin writes and plugin log messages. * **Query nodes**: Run plugin queries. The process node doesn’t need `--internode-bind-addr`. The smallest cluster is one combined node and the process node: | Node ID | `--mode` |Processing Engine flags| |----------|----------------------|-----------------------| | `node0` |`ingest,query,compact`|`--internode-bind-addr`| |`process0`| `process` | `--plugin-dir` | To scale ingest, query, and compaction separately, run each mode on its own node: | Node ID |`--mode` |Processing Engine flags| |----------|---------|-----------------------| |`ingest0` |`ingest` |`--internode-bind-addr`| | `query0` | `query` |`--internode-bind-addr`| |`compact0`|`compact`| | |`process0`|`process`| `--plugin-dir` | ### Secure the process node A process node isn’t a security boundary: it needs the same credentials as the other nodes. * **Object storage credentials**: The process node reads the catalog and saves its trigger state in object storage. For WAL triggers, it also reads each ingest node’s WAL files from object storage. * **License**: The process node needs the cluster’s license file. * **Internode credentials**: Plugin queries, writes, and log messages use the internode protocol. If you set `--internode-jwt-primary` or the `--internode-tls-*` options on the cluster, set them on the process node too. Without these options, internode traffic is unencrypted. ## Start the nodes Start each node with `influxdb3 serve`. The following examples start the four-node cluster from [Plan the cluster](#plan-the-cluster), with each node on its own host. A multi-node cluster requires a shared object store such as Amazon S3. To keep the commands short, the examples use an S3 bucket and omit the credential options. See [Connect to your object store](/influxdb3/enterprise/get-started/multi-server/#connect-to-your-object-store). ```bash # Ingest node influxdb3 serve \ --cluster-id CLUSTER_ID \ --node-id ingest0 \ --mode ingest \ --object-store s3 \ --bucket BUCKET \ --internode-bind-addr 0.0.0.0:8083 \ --conn-info INGEST_HOST:8083 # Query node influxdb3 serve \ --cluster-id CLUSTER_ID \ --node-id query0 \ --mode query \ --object-store s3 \ --bucket BUCKET \ --internode-bind-addr 0.0.0.0:8083 \ --conn-info QUERY_HOST:8083 # Compact node influxdb3 serve \ --cluster-id CLUSTER_ID \ --node-id compact0 \ --mode compact \ --object-store s3 \ --bucket BUCKET # Process node influxdb3 serve \ --cluster-id CLUSTER_ID \ --node-id process0 \ --mode process \ --object-store s3 \ --bucket BUCKET \ --plugin-dir PLUGIN_DIR ``` Replace the following: * `CLUSTER_ID`: the same cluster ID on every node * `BUCKET`: the S3 bucket that every node uses * `PLUGIN_DIR`: the directory on the process node that holds your plugin files * `INGEST_HOST`, `QUERY_HOST`: the host name or IP address that the process node uses to reach the ingest or query node [`--conn-info`](/influxdb3/enterprise/reference/config-options/#conn-info) sets the address that other nodes use to reach the node’s internode port. Without it, a node bound to a wildcard address such as `0.0.0.0` advertises `:`. If the process node can’t resolve that host name, plugin queries, plugin writes, and plugin log messages fail. On first start, the process node creates a Python virtual environment at `PLUGIN_DIR/.venv` from the Python bundled with InfluxDB 3 Enterprise. > [!Warning] > #### Docker containers run the Processing Engine by default > > The official Docker image sets `INFLUXDB3_PLUGIN_DIR=/plugins`. > A node with a plugin directory runs the Processing Engine regardless of `--mode`, so every container started from the image becomes a process node. > On ingest, query, and compact containers, unset the variable with `-e INFLUXDB3_UNSET_VARS=INFLUXDB3_PLUGIN_DIR`. > For more information, see [Disable in Docker deployments](/influxdb3/enterprise/reference/processing-engine/#disable-in-docker-deployments). ## Verify the cluster Query `system.nodes` on a query node: ```bash influxdb3 query \ --host http://QUERY_HOST:8181 \ --token AUTH_TOKEN \ --database _internal \ "SELECT node_id, mode, state, conn_info FROM system.nodes" ``` Check the following: * `process0` has `mode` `[process]` and `state` `running`. * Each ingest and query node has a `conn_info` that the process node can reach. ## Add plugins and packages Put plugin files in the process node’s plugin directory. For the plugin API, see the [Python API reference](/influxdb3/enterprise/plugins/python-api-reference/). To install Python packages, run [`influxdb3 install package`](/influxdb3/enterprise/reference/cli/influxdb3/install/package/) against the process node: ```bash influxdb3 install package \ --host http://PROCESS_HOST:8181 \ --token AUTH_TOKEN \ PACKAGE_NAME ``` Packages are installed only on the node that receives the request. ## Create triggers 1. Create the database on any node. 2. Create the trigger on the process node. Set `--node-spec` to the process node’s ID. Other nodes reject the request with `405 Current node mode does not use the processing engine`. ```bash influxdb3 create trigger \ --host http://PROCESS_HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --path PLUGIN_FILE \ --trigger-spec "table:TABLE_NAME" \ --node-spec "nodes:process0" \ TRIGGER_NAME ``` Always set `--node-spec`. The default, `all`, makes every node that runs the Processing Engine own the trigger. You can’t change a trigger’s `--node-spec` after you create it. If you use `--upload` to send a local plugin file, the file is stored on the process node. > [!Note] > #### WAL triggers process only new writes > > The first time the process node detects an ingest node, it starts reading that node’s WAL from the current position. > A WAL trigger doesn’t run for data written before then. ## Send each request to the right node | Request | Send it to | |--------------------------------------------------------------------|----------------| | Write data | An ingest node | | Query data or `system.processing_engine_logs` | A query node | | Create a trigger |The process node| | Install Python packages |The process node| | Call a request trigger (`/api/v3/engine/`) |The process node| |Create a database; enable, disable, or delete a trigger; stop a node| Any node | A request sent to the wrong node returns an error. See [HTTP errors](#http-errors). ## Manage triggers Enable, disable, and delete triggers from any node. The process node applies the change within a few seconds. ```bash influxdb3 disable trigger \ --host http://HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ TRIGGER_NAME influxdb3 enable trigger \ --host http://HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ TRIGGER_NAME influxdb3 delete trigger \ --host http://HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --force \ TRIGGER_NAME ``` To list triggers, query `system.processing_engine_triggers` on a query node: ```bash influxdb3 show system \ --host http://QUERY_HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ table processing_engine_triggers ``` ## Stop and restart the process node Stop the process node with [`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) from any node: ```bash influxdb3 stop node \ --host http://HOST:8181 \ --token AUTH_TOKEN \ --node-id process0 ``` To restart it, run `influxdb3 serve` with the same options. The process node saves its trigger state to `//processing_engine/restart-state.postcard` in object storage. It saves the state every [`--processing-engine-restart-state-snapshot-interval`](/influxdb3/enterprise/reference/config-options/#processing-engine-restart-state-snapshot-interval) (default `1s`) and at shutdown. After a restart, the process node does the following: * **WAL triggers**: Resumes reading each ingest node’s WAL where it stopped, so it processes writes that arrived while it was down. * **Schedule triggers**: Runs each scheduled call that came due while it was down. * **Request triggers**: Doesn’t replay requests. During a graceful shutdown, in-progress requests return `503` with `{"error": "server is shutting down"}`. After a forced stop, the connection drops. Clients can retry either way. * **Retries**: Keeps pending retries. A trigger that you disable or delete while the process node is down stays disabled or deleted. > [!Important] > #### Make plugins safe to run more than once > > A WAL trigger run or scheduled call can run again after a restart, for example when the run was in progress during the restart. > Write plugins so that a repeated run gives the same result, for example by writing the same points with the same timestamps. ### Retry and queue limits * [`--trigger-retry-max-attempts`](/influxdb3/enterprise/reference/config-options/#trigger-retry-max-attempts) (default `5`): Sets how many times a failed run is attempted for triggers with `--error-behavior retry`. * Each trigger queues up to 60 pending runs. When a request trigger’s queue is full, `/api/v3/engine/` returns `503` with `{"error": "trigger queue is full"}`. ## Troubleshoot * [Startup errors](#startup-errors) * [Plugin query and write errors](#plugin-query-and-write-errors) * [HTTP errors](#http-errors) * [Trigger problems](#trigger-problems) ### Startup errors #### The process node doesn’t start `--mode process` without `--plugin-dir` fails with: ```text starting a node with mode 'process' requires a configured plugin dir for the python environment ``` Add `--plugin-dir`. #### A node runs the Processing Engine unexpectedly `--plugin-dir` or `INFLUXDB3_PLUGIN_DIR` adds `process` mode to a node, whatever its `--mode`. Remove the option or variable from every node except the process node. For Docker, see [Start the nodes](#start-the-nodes). #### The process node doesn’t start after a crash If `restart-state.postcard` is corrupt, the process node doesn’t start. Move the file out of the way and start the node. The node starts with no saved trigger state: queued runs, pending retries, and its position in each WAL are lost. ### Plugin query and write errors `influxdb3_local.query()`, `write_sync()`, and `write_sync_to_db()` raise these errors in the plugin.`write()` and `write_to_db()` don’t: they queue rows until the plugin returns, so a failed write loses the rows and the error appears only in the process node’s log. We recommend `write_sync()` or `write_sync_to_db()` so that your plugin can detect and handle failed writes. #### `Cannot query: no remote query client found` No running query node has an internode address. Restart a query node with `--internode-bind-addr`. The process node detects it within a few seconds; you don’t need to restart the process node. #### `Cannot write: no remote write client found` No running ingest node has an internode address. Until you fix it, plugin writes and plugin log messages fail. Failed log messages don’t raise an error in the plugin; look for this line in the process node’s log: ```text failed to persist processing engine log error=Cannot log: Cannot write: no remote write client found ``` Restart an ingest node with `--internode-bind-addr`. The process node detects it within a few seconds. #### `dns error`, then `circuit-open` A plugin query or `write_sync()` call fails with `dns error`. After a few failures, it fails with `all remote query clients are circuit-open` or `all remote write clients are circuit-open`. Plugin log messages are lost with the same error. An ingest or query node advertises a `conn_info` that the process node can’t resolve, usually the host name from a `0.0.0.0` bind. Check `conn_info` in [`system.nodes`](#verify-the-cluster), then restart that node with `--conn-info` set to an address the process node can reach. The process node detects the new address without a restart. #### `ModuleNotFoundError` for a package you installed The package isn’t installed on the process node. Run [`influxdb3 install package`](#add-plugins-and-packages) against the process node. ### HTTP errors | Error | Cause | Fix | |---------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------| | `500 cannot write to a read-only server: must enable ingest mode` | You wrote data to the process node. | Write to an ingest node. | | `405 this node does not run the query mode` | You queried data or `system.processing_engine_logs` on the process node. | Query a query node. | | `405 Current node mode does not use the processing engine` |You created a trigger, installed packages, or called `/api/v3/engine/` on a node without the Processing Engine.| Send the request to the process node. | | `404 {error: "not found"}` on `/api/v3/engine/` | The trigger is disabled, deleted, or owned by another node. |Enable the trigger, or call the node in its `--node-spec`.| | `500 Node doesn't have proper mode, expect process` | `--node-spec` names a node that isn’t a process node. | Set `--node-spec` to the process node. | | `500 invalid node name ()` | `--node-spec` names a node that doesn’t exist. | Check the node ID in `system.nodes`. | | `500 Plugin not found: /` | The plugin file isn’t in the process node’s plugin directory. | Copy the file to the process node. | |`404 Database not found ` right after you create the database on another node| The process node hasn’t loaded the catalog change yet. | Wait 2 seconds and retry. | Nodes load catalog changes every [`--catalog-sync-interval`](/influxdb3/enterprise/reference/config-options/#catalog-sync-interval) (default `1s`). ### Trigger problems #### A trigger disables itself With `--error-behavior disable`, the first failed run disables the trigger. The process node logs `trigger execution failed; disabling trigger`. Fix the plugin, then enable the trigger. #### A trigger stops running after you replace the process node Only the node in a trigger’s `--node-spec` owns the trigger. If that node no longer runs `process` mode, the trigger never runs, but `system.processing_engine_triggers` still shows it as enabled. Delete the trigger and create it again with the new node ID in `--node-spec`. #### A plugin error doesn’t appear in the logs table Python tracebacks appear only in the process node’s log, in lines that contain `trigger execution failed`. They don’t appear in `system.processing_engine_logs`. Rows in `system.processing_engine_logs` can appear up to 50 seconds after the run. The `node_id` column shows the node that ran the plugin. #### Related * [Configure specialized cluster nodes](/influxdb3/enterprise/admin/clustering/) * [Processing engine and Python plugins](/influxdb3/enterprise/plugins/) * [Processing engine reference](/influxdb3/enterprise/reference/processing-engine/) * [influxdb3 create trigger](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/) * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) [processing engine](/influxdb3/enterprise/tags/processing-engine/)[plugins](/influxdb3/enterprise/tags/plugins/)[clustering](/influxdb3/enterprise/tags/clustering/)[triggers](/influxdb3/enterprise/tags/triggers/)[troubleshooting](/influxdb3/enterprise/tags/troubleshooting/) | Node ID | --mode | Processing Engine flags | | --- | --- | --- | | Node ID | --mode | Processing Engine flags | | node0 | ingest,query,compact | --internode-bind-addr | | process0 | process | --plugin-dir | | Node ID | --mode | Processing Engine flags | | --- | --- | --- | | Node ID | --mode | Processing Engine flags | | ingest0 | ingest | --internode-bind-addr | | query0 | query | --internode-bind-addr | | compact0 | compact | | | process0 | process | --plugin-dir | | Request | Send it to | | --- | --- | | Request | Send it to | | Write data | An ingest node | | Query data or system.processing_engine_logs | A query node | | Create a trigger | The process node | | Install Python packages | The process node | | Call a request trigger ( /api/v3/engine/ ) | The process node | | Create a database; enable, disable, or delete a trigger; stop a node | Any node | | Error | Cause | Fix | | --- | --- | --- | | Error | Cause | Fix | | 500 cannot write to a read-only server: must enable ingest mode | You wrote data to the process node. | Write to an ingest node. | | 405 this node does not run the query mode | You queried data or system.processing_engine_logs on the process node. | Query a query node. | | 405 Current node mode does not use the processing engine | You created a trigger, installed packages, or called /api/v3/engine/ on a node without the Processing Engine. | Send the request to the process node. | | 404 {error: "not found"} on /api/v3/engine/ | The trigger is disabled, deleted, or owned by another node. | Enable the trigger, or call the node in its --node-spec . | | 500 Node doesn't have proper mode, expect process | --node-spec names a node that isn’t a process node. | Set --node-spec to the process node. | | 500 invalid node name () | --node-spec names a node that doesn’t exist. | Check the node ID in system.nodes . | | 500 Plugin not found: / | The plugin file isn’t in the process node’s plugin directory. | Copy the file to the process node. | | 404 Database not found right after you create the database on another node | The process node hasn’t loaded the catalog change yet. | Wait 2 seconds and retry. | --- # Query system data Source: https://docs.influxdata.com/influxdb3/enterprise/admin/query-system-data/ InfluxDB 3 Enterprise stores data related to the database server, queries, and tables in *system tables*. You can query the system tables for information about your running server, databases, and table schemas. ## Query system tables * [Use the HTTP query API](#use-the-http-query-api) * [Examples](#examples) * [Show tables](#show-tables) * [View column information for a table](#view-column-information-for-a-table) * [Recently executed queries](#recently-executed-queries) * [Query plugin files](#query-plugin-files) * [Query trigger logs](#query-trigger-logs) * [Query storage engine tables](#query-storage-engine-tables) ### Use the HTTP query API Use the HTTP API `/api/v3/query_sql` endpoint to retrieve system information about your database server and table schemas in InfluxDB 3 Enterprise. To execute a query, send a `GET` or `POST` request to the endpoint: * `GET`: Pass parameters in the URL query string (for simple queries) * `POST`: Pass parameters in a JSON object (for complex queries and readability in your code) Include the following parameters: * `q`: *(Required)* The SQL query to execute. * `db`: *(Required)* The database to execute the query against. * `params`: A JSON object containing parameters to be used in a *parameterized query*. * `format`: The format of the response (`json`, `jsonl`, `csv`, `pretty`, or `parquet`). JSONL (`jsonl`) is preferred because it streams results back to the client.`pretty` is for human-readable output. Default is `json`. #### Examples > [!Note] > #### system\_ sample data > > In examples, tables with `"table_name":"system_` are user-created tables for CPU, memory, disk, > network, and other resource statistics collected and written > by the user–for example, using the `psutil` Python library or[Telegraf](/telegraf/v1/get-started/) to collect > and write system metrics to an InfluxDB 3 database. ##### Show tables The following example sends a `GET` request that executes a `show tables` query to retrieve all user-created tables (`"table_schema":"iox"`), system tables, and information schema tables for a database: ```bash curl "http://localhost:8181/api/v3/query_sql?db=mydb&format=jsonl&q=show%20tables" ``` The response body contains the following JSONL: ```jsonl {"table_catalog":"public","table_schema":"iox","table_name":"system_cpu","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_cpu_cores","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_memory","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_memory_faults","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_disk_usage","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_disk_io","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_disk_performance","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_network","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"distinct_caches","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"last_caches","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"parquet_files","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"processing_engine_plugins","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"processing_engine_triggers","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"queries","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"information_schema","table_name":"tables","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"views","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"columns","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"df_settings","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"schemata","table_type":"VIEW"} ``` A table has one of the following `table_schema` values: * `iox`: tables created by the user of the database. * `system`: tables used by the system to show information about the running database server. Some of these tables show stored information such as configurations, while others, such as the `queries` table, hold ephemeral state in memory. * `information_schema`: views that show schema information for tables in the database. #### View column information for a table The following query sends a `POST` request that executes an SQL query to retrieve information about columns in the sample `system_swap` table schema: *Note: when you send a query in JSON, you must escape single quotes that surround field names.* ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "mydb", "q": "SELECT * FROM information_schema.columns WHERE table_schema = '"'iox'"' AND table_name = '"'system_swap'"'", "format": "jsonl" }' ``` The output is the following: ```jsonl {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"free","ordinal_position":0,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"host","ordinal_position":1,"is_nullable":"NO","data_type":"Dictionary(Int32, Utf8)"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"percent","ordinal_position":2,"is_nullable":"YES","data_type":"Float64","numeric_precision":24,"numeric_precision_radix":2} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"sin","ordinal_position":3,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"sout","ordinal_position":4,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"time","ordinal_position":5,"is_nullable":"NO","data_type":"Timestamp(Nanosecond, None)"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"total","ordinal_position":6,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"used","ordinal_position":7,"is_nullable":"YES","data_type":"UInt64"} ``` #### Recently executed queries To view recently executed queries, query the `queries` system table: ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "mydb", "q": "SELECT * FROM system.queries LIMIT 2", "format": "jsonl" }' ``` The output is similar to the following: ```jsonl {"id":"cdd63409-1822-4e65-8e3a-d274d553dbb3","phase":"success","issue_time":"2025-01-20T17:01:40.690067","query_type":"sql","query_text":"show tables","partitions":0,"parquet_files":0,"plan_duration":"PT0.032689S","permit_duration":"PT0.000202S","execute_duration":"PT0.000223S","end2end_duration":"PT0.033115S","compute_duration":"P0D","max_memory":0,"success":true,"running":false,"cancelled":false} {"id":"47f8d312-5e75-4db2-837a-6fcf94c09927","phase":"success","issue_time":"2025-01-20T17:02:32.627782","query_type":"sql","query_text":"show tables","partitions":0,"parquet_files":0,"plan_duration":"PT0.000583S","permit_duration":"PT0.000015S","execute_duration":"PT0.000063S","end2end_duration":"PT0.000662S","compute_duration":"P0D","max_memory":0,"success":true,"running":false,"cancelled":false} ``` #### Query plugin files To view loaded Processing Engine plugins, query the `plugin_files` system table in the `_internal` database. The `system.plugin_files` table provides information about plugin files loaded by the Processing Engine: **Columns:** * `plugin_name` (String): Name of a trigger using this plugin * `file_name` (String): Plugin filename * `file_path` (String): Full server path to the plugin file * `size_bytes` (Int64): File size in bytes * `last_modified` (Int64): Last modification timestamp (milliseconds since epoch) ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "_internal", "q": "SELECT * FROM system.plugin_files", "format": "jsonl" }' ``` The output is similar to the following: ```jsonl {"plugin_name":"my_trigger","file_name":"my_plugin.py","file_path":"/path/to/plugins/my_plugin.py","size_bytes":2048,"last_modified":1704067200000} {"plugin_name":"scheduled_trigger","file_name":"scheduler.py","file_path":"/path/to/plugins/scheduler.py","size_bytes":4096,"last_modified":1704153600000} ``` **Filter plugins by trigger name:** ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "_internal", "q": "SELECT * FROM system.plugin_files WHERE plugin_name = '"'my_trigger'"'", "format": "jsonl" }' ``` **Find plugins by file pattern:** ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "_internal", "q": "SELECT * FROM system.plugin_files WHERE file_name LIKE '"'%scheduler%'"'", "format": "jsonl" }' ``` #### Query trigger logs The `system.processing_engine_logs` table stores log entries from Processing Engine triggers. Logs are stored in **two locations**: * The trigger’s database (primary)–query here for trigger-specific debugging * The `_internal` database–contains logs from all triggers across all databases **Columns:** * `event_time` (Timestamp): When the log entry was recorded * `trigger_name` (String): Name of the trigger that generated the log * `log_level` (String): Log level (INFO, WARN, ERROR) * `log_text` (String): Log message content **Query logs for a specific trigger:** ```bash influxdb3 query \ --database DATABASE \ "SELECT event_time, log_level, log_text FROM system.processing_engine_logs WHERE trigger_name = 'TRIGGER' ORDER BY event_time DESC LIMIT 20" ``` Replace the following: * `DATABASE`: The database where you created the trigger * `TRIGGER`: The name of your trigger **Query all trigger logs (from \_internal):** ```bash influxdb3 query \ --database _internal \ "SELECT event_time, trigger_name, log_level, log_text FROM system.processing_engine_logs ORDER BY event_time DESC LIMIT 50" ``` #### Query storage engine tables The [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters in 3.11+) exposes internal state through system tables. **`system.pt_ingest_wal`** — WAL files and their shards: ```sql SELECT * FROM system.pt_ingest_wal; ``` Columns: `wal_file_id`, `node_id`, `node_name`, `shard_start_time`,`shard_duration_seconds`, `min_time`, `max_time`, `row_count`,`size_bytes`, `is_merged`. Use this table to monitor WAL accumulation, shard distribution, time coverage, and merge status (`is_merged = false` rows are unmerged). **`system.pt_ingest_files`** — Gen0 files with metadata: ```sql SELECT * FROM system.pt_ingest_files; ``` Columns: `file_id`, `node_id`, `node_name`, `generation`, `min_time`,`max_time`, `row_count`, `size_bytes`, `has_bloom_filter`. Starting in v3.12.1, after a restart this table lists the files from the node’s newest snapshot manifest plus the snapshots taken since the node started. Files from older manifests aren’t listed. Use this table to monitor file counts per generation, file sizes, and time ranges. **Compaction tables** — expose the compaction subsystem’s state: * `system.pt_compaction_active_jobs`: currently running compaction jobs (`plan_id`, `plan_type`, `state`, `shard_id`, `total_slices`,`completed_slices`). * `system.pt_compaction_ingest_nodes`: per-ingest-node compaction lag (`node_id`, `compaction_lag`, `seen_lag`,`deferred_snapshot_count`—a non-zero value means snapshots are failing to compact and accumulating; check `system.pt_compaction_deferred_snapshots`). * `system.pt_compaction_nodes`: compaction node state. * `system.pt_compaction_run_sets`: pending compaction work grouped by time window and shard. * `system.pt_compaction_deferred_snapshots`: snapshots that failed to compact; a growing list indicates a persistent compaction failure—check`error_message`. * `system.pt_shards`: identifies each time-window shard and whether it’s currently active; check `is_active`. * `system.pt_compaction_files`: per-file compaction metadata, one row per compacted file; check `level` for compaction depth and`size_bytes`/`row_count` for file growth. * `system.pt_storage_snapshots`: one row per snapshot taken from the WAL; check `gen0_file_count` and `total_size_bytes` to monitor snapshot growth. * `system.pt_storage_checkpoints`: durable checkpoint metadata; check`is_latest` to find the current checkpoint and `window_count` for its coverage. * `system.pt_storage_run_set_indexes`: per-run-set index files; check`size_bytes` and `last_modified` to spot stale or oversized indexes. **Parquet upgrade status** — if you[upgraded from Parquet](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet): ```bash # Per-node upgrade status influxdb3 query \ --database _internal \ "SELECT * FROM system.upgrade_parquet_node" # Per-file migration progress influxdb3 query \ --database _internal \ "SELECT * FROM system.upgrade_parquet" ``` `system.upgrade_parquet_node` returns an `ingest` row for each node that runs in `ingest` mode and a `compactor` row for each node that runs in `compact`mode. Nodes that run only in `query` mode don’t report upgrade status. A node finishes upgrading only when every row for that node reads `completed`. The upgrade coordinator checks progress periodically. After migration work completes, a row can remain `upgrading` until the coordinator detects completion. The `completed` status reports when the coordinator detects that the migration has finished. Starting in v3.11.5, `compactor` rows also report upgrade health. A `stalled` value of `true` means the upgrade has stopped making progress;`stalled_since` reports when it stopped. Check `last_progress_at` for the last time the upgrade advanced and `restarts`for how many times the compactor restarted during the upgrade. The `conversion_*` and `import_*` columns report conversion and import progress. These columns are null on `ingest` rows. Starting in v3.11.8 and v3.12.2, if a Parquet file fails every conversion attempt, the `status` column of the node’s `ingest` row is `failed`. The `last_message` column identifies the file and its error. To resume the upgrade, follow the[retry procedure](/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/#retry-after-an-ingest-conversion-failure). **Query telemetry** — the query telemetry endpoint provides detailed execution statistics for analyzing query performance: ```bash curl --request GET "http://localhost:8181/api/v3/query_sql_telemetry" \ --header "Authorization: Bearer AUTH_TOKEN" ``` Replace `AUTH_TOKEN` with your authentication token. The response includes `query_id`, `execution_time_us`, per-chunk statistics (`chunks`), cache hit rates by type (`cache_stats`), and file-level read statistics (`file_stats`). #### Related * [Query data with SQL](/influxdb3/enterprise/query-data/sql/) * [Execute queries](/influxdb3/enterprise/query-data/execute-queries/) [query](/influxdb3/enterprise/tags/query/)[api](/influxdb3/enterprise/tags/api/)[system information](/influxdb3/enterprise/tags/system-information/)[schemas](/influxdb3/enterprise/tags/schemas/) --- # Recover a crashed node Source: https://docs.influxdata.com/influxdb3/enterprise/admin/recover-node/ Use this procedure to recover a node that stopped **ungracefully** — for example, the process crashed, was killed (`kill -9`), or its container was force-stopped. It also applies to a node wedged in the `stopping` state. > [!Note] > This procedure applies to both the Parquet engine and the upgraded storage > engine (the default for new clusters in InfluxDB 3 Enterprise 3.11+). ## Why ungraceful stops put writes at risk A node stores recently acknowledged writes in its write-ahead log (WAL) until it captures them in a snapshot. These buffered writes are the node’s [*WAL tail*](/influxdb3/enterprise/reference/internals/durability/#wal-tail). Run the [`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)command against a running node to save the WAL tail before the node reports a`stopped` state. A node that dies without a graceful stop skips that drain: * Running `stop node` against the dead process cannot drain anything. * [Removing the node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)purges its object-store file paths, permanently deleting any acknowledged writes not covered by its last snapshot. * Sending the process a bare `SIGTERM` (without calling `stop node`) does not force a WAL snapshot on the upgraded storage engine, so a plain process shutdown can also leave a[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail)behind. On clusters that have fully adopted the upgraded storage engine,`remove node` refuses (HTTP 409) to remove a `stopped`node that still has a[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail)unless you pass `--force-finalize`. Parquet clusters and clusters still mid-upgrade do **not** have this safeguard. On those clusters, completing this recovery procedure before removal is the only protection against losing the tail. To determine which engine your cluster runs, start with how it was created: new clusters on InfluxDB 3 Enterprise 3.11+ default to the upgraded storage engine, and clusters that started on 3.10 or earlier keep the Parquet engine until you restart them with `--upgrade-pacha-tree`. If you started a storage engine upgrade, confirm it finished—query`system.upgrade_parquet_node` and check that every node reports `completed`. See [Query system data](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables). For more information about the upgraded storage engine, see[Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). ## Recover the node 1. **Restart a server process with the same `--node-id`** (environment variable: `INFLUXDB3_NODE_ID`) and the same object store configuration. This is functionally the same as restarting the original node: on startup, WAL recovery replays the WAL files that were not drained, so the acknowledged writes are safe again. ``` influxdb3 serve \ --node-id NODE_ID \ --cluster-id CLUSTER_ID \ # ...same object store configuration as the original node ``` 2. **Stop the node gracefully** and wait for it to reach `stopped`. This drains the[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail). How the node drains it depends on the storage engine: on the Parquet engine, the node persists the buffered writes to Parquet files; on the upgraded storage engine, it captures them in a WAL snapshot. ``` influxdb3 stop node --node-id NODE_ID ``` The command waits (up to `--timeout`, default `5m`) for the node to confirm the stop. 3. **Remove the node** if you intend to permanently remove it from the cluster. If you are recovering the node to keep using it, skip this step—after a graceful stop, you can start it again at any time. ``` influxdb3 remove node --node-id NODE_ID ``` ## If the node cannot be brought back If the node cannot be restarted (for example, its data is unrecoverable or the hardware is gone) and you accept the possible loss of writes not yet captured in a snapshot, use the[`--force-finalize` option](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/#force-removal-of-a-node-that-did-not-shut-down-cleanly)to force the removal: ```bash influxdb3 remove node --node-id NODE_ID --force-finalize ``` > [!Warning] > `--force-finalize` **can lose data**: it removes the node using > whatever snapshot exists in object storage and permanently deletes any > acknowledged writes not covered by that snapshot. > If the node had finished snapshotting and only failed to report a clean > stop, nothing is lost. > Always attempt the [recovery procedure](#recover-the-node) first. ## Verify node status At any point, check node states with[`influxdb3 show nodes`](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/): ```bash influxdb3 show nodes ``` #### Related * [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) * [influxdb3 remove node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/) * [influxdb3 show nodes](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/) [clustering](/influxdb3/enterprise/tags/clustering/)[nodes](/influxdb3/enterprise/tags/nodes/)[recovery](/influxdb3/enterprise/tags/recovery/)[wal](/influxdb3/enterprise/tags/wal/) --- # Security Source: https://docs.influxdata.com/influxdb3/enterprise/admin/security/ Secure InfluxDB 3 Enterprise at the authentication, authorization, and host levels. ## Authentication Verify the identity of clients and users that connect to InfluxDB 3 Enterprise. * [Manage tokens](/influxdb3/enterprise/admin/tokens/): Authenticate API and CLI requests with `apiv3_` tokens. * [Manage users and authentication](/influxdb3/enterprise/admin/security/manage-users/): Enable multi-user authentication so users log in with individual credentials. ## Authorization Control what authenticated users and tokens can do. * [Role-based access control (RBAC)](/influxdb3/enterprise/reference/internals/rbac/): Assign built-in roles to users and understand the permissions model. ## Host hardening * [Harden the host and service](/influxdb3/enterprise/admin/security/harden-the-host/): Configure systemd sandboxing and host-level filesystem protections. --- # Harden the host and service Source: https://docs.influxdata.com/influxdb3/enterprise/admin/security/harden-the-host/ InfluxDB 3 Enterprise includes security features to protect your data and system resources. When you install using DEB or RPM packages, the default `systemd` unit file configures security sandboxing to isolate the database process from the host system. This page explains the filesystem layout, sandboxing directives, and how to customize security settings for your environment. * [Linux DEB and RPM](#linux-deb-and-rpm) * [Requirements](#requirements) * [Filesystem layout](#filesystem-layout) * [systemd unit in detail](#systemd-unit-in-detail) * [Default sandbox configuration](#default-sandbox-configuration) * [Site-specific directives](#site-specific-directives) * [Tuning the systemd unit](#tuning-the-systemd-unit) * [systemd references](#systemd-references) ## Linux DEB and RPM When you install via DEB or RPM on a `systemd`-enabled system, InfluxDB 3 Enterprise runs in a sandboxed environment configured by the provided `systemd` unit file. ### Requirements * A `systemd`-enabled Linux system * `systemd` version 248 or later for full sandbox support (Debian 12+, RHEL 9+, Ubuntu 22.04+) > [!Note] > On older systems, `systemd` logs `Unknown lvalue ''` and starts the service without sandbox protection. ### Filesystem layout The provided unit file assumes the following filesystem layout: * `/etc/influxdb3`: directory for InfluxDB 3 Enterprise configuration (`0755` permissions with `root:influxdb3` ownership by default) * `/etc/influxdb3/influxdb3-enterprise.conf`: TOML configuration file * `/usr/bin/influxdb3`: InfluxDB 3 Enterprise binary * `/usr/lib/influxdb3/python`: directory containing the embedded interpreter used by the InfluxDB 3 Enterprise processing engine * `/var/lib/influxdb3`: writable directory for InfluxDB 3 Enterprise * `/var/lib/influxdb3/data`: default directory for InfluxDB 3 Enterprise data files when `object-store` is set to `file` (the default for DEB and RPM installations) * `/var/lib/influxdb3/plugins`: default directory for InfluxDB 3 Enterprise plugin files * `/var/log/influxdb3`: writable directory for logging (unused by default) > [!Important] > If you store sensitive credentials in `/etc/influxdb3`, adjust permissions to `0750` to restrict access. ### `systemd` unit in detail The unit file is self-documenting. To view the full systemd configuration for the InfluxDB 3 service (`influxdb3-enterprise`), enter the following command: ```bash systemctl cat influxdb3-enterprise ``` The output is similar to the following: ``` # /usr/lib/systemd/system/influxdb3-enterprise.service [Unit] Description=InfluxDB 3 Enterprise After=network-online.target [Service] Type=simple ... ... ``` #### Default sandbox configuration The default sandbox configuration provides security isolation without breaking common use cases. The following options are set by default: ##### Basic Security Defaults for basic security, such as filesystem and user: * `StateDirectory=influxdb3` - writable area relative to `/var/lib` * `LogsDirectory=influxdb3` - writable area relative to `/var/log` (the unit is configured with `StandardOutput=journal` and `StandardError=journal` by default and will not use this directory) * `User=influxdb3`, `Group=influxdb3`, `SupplementaryGroups=` - run InfluxDB 3 Enterprise as the unprivileged `influxdb3:influxdb3` user. InfluxDB 3 Enterprise does not require any special privileges to run and this should always be set to an unprivileged user * `UMask=0027` - restrictive default file mode creation mask ##### Limit kernal attack surface Defaults to limit the kernal attack surface: * `SystemCallFilter=@system-service`, `SystemCallArchitectures=native`, `SystemCallFilter=~io_uring_setup keyctl userfaultfd`, and `LockPersonality=true` - basic set of allowed Linux system calls excluding a few unneeded ones that can be abused * `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX` - limit allowed address families to those needed for basic functionality (such as, IP networking and DNS resolution). Custom processing engine plugins that need kernel socket of route introspection may need to add `AF_NETLINK` to this list * `RestrictNamespaces=true` - disallow use of Linux namespaces ##### Limit privileges Defaults to limit privileges and disallow gaining or inheriting permissions and capabilities (blocks ICMP `ping`, `passwd`): * `NoNewPrivileges=true` * `RestrictSUIDSGID=true` * `CapabilityBoundingSet=` * `AmbientCapabilities=` ##### Host protection Defaults for host protection: * `ProtectSystem=strict` - make host files read-only * `ProtectHome=true` - disallow access to `/home` (tip: put credentials, configuration, etc in `/etc/influxdb3` or somewhere in `/var/lib/influxdb3` instead) * `PrivateTmp=true` - use separate `/tmp` and `/var/tmp` from host * `TemporaryFileSystem=/dev/shm:mode=1777` - use separate `/dev/shm` (override with `size=` to limit size too) * `PrivateDevices=true` - allow only pseudo devices with no host mount propagation * `ProtectKernelLogs=true` - disallow access to the kernel log ring buffer (needed if `PrivateDevices=false`) * `PrivateIPC=true` - use separate SysV IPC from host * `InaccessiblePaths=...` - disallow well-known system and user services’ named sockets (needed since `AF_UNIX` is allowed) * `ProtectProc=invisible` - hide processes not owned by this [user ( `influxdb3:influxdb3`)](#basic-security). This provides strong isolation but means that plugins can’t see other processes on the system, which could affect custom processing engine plugins that require this #### Site-specific directives The default configuration omits directives that depend on your environment–for example, the following directives require tuning based on your deployment requirements and resource constraints: * `IPAddressDeny` and `IPAddressAllow` for limiting communications by the database and processing engine to certain IP addresses * `MemoryHigh` and `MemoryMax` for limiting memory usage (the database process already has configurable controls for memory so this is primarily useful to limit the processing engine) * `Nice`, `CPUQuota`, `CPUSchedulingPolicy`, `LimitNPROC`, and `TasksMax` for limiting CPU usage (the database process already has configurable controls for CPU so this is primarily useful to limit the processing engine) * `IOWeight`, etc for limiting I/O operations (primarily useful for limiting the processing engine) * `ReadOnlyPaths`, `ReadWritePaths`, and `InaccessiblePaths` to allow/disallow other paths not covered by the default sandbox > [!Important] > Due to a limit in InfluxDB 3 Enterprise related to socket activation, `PrivateNetwork=true` cannot be used at this time. ### Tuning the `systemd` unit While the `systemd` unit is verified to work with InfluxDB 3 Enterprise and [official plugins](/influxdb3/enterprise/plugins/library/official/), you may want to harden the unit further or loosen its restrictions in certain situations. To edit the unit file, enter the following command: ```bash systemctl edit influxdb3-enterprise ``` > [!Note] > Avoid modifying the `influxdb3-enterprise.service` file directly. > Use `systemctl edit` to add overrides. #### Example: loosen for ProtectProc=default If a custom plugin needs to read other processes’ information from `/proc`: 1. Run `sudo systemctl edit influxdb3-enterprise` 2. Edit the file to contain: ``` ### Editing /etc/systemd/system/influxdb3-enterprise.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file [Service] # the 'foo' plugin needs to see other user's processes ProtectProc=default ### Lines below this comment will be discarded ... ``` 3. Verify the changes (the shipped unit is listed first followed by overrides): ``` $ sudo systemctl daemon-reload && systemctl cat --no-pager influxdb3-enterprise [Unit] Description=InfluxDB 3 Enterprise After=network-online.target [Service] ... ProtectProc=invisible ... # /etc/systemd/system/influxdb3-enterprise.service.d/override.conf [Service] # the 'foo' plugin needs to see other users' processes ProtectProc=default $ ``` 4. Restart the unit with `sudo systemctl restart influxdb3-enterprise` #### Example: restrict networking `systemd` supports network filtering via BPF. When adding directives, the rule evaluation order is: 1. Access is granted if matches entry in IPAddressAllow 2. Otherwise access is denied if matches entry in IPAddressDeny 3. Otherwise access is granted For egress, the IP matches against sender and for ingress, it matches against the receiver. This filtering only matches on IP addresses, not ports; if you need more flexibility, use host firewall tools/cloud security groups instead. As an example, to limit communications to only localhost, use`systemctl edit influxdb3-enterprise` to add: ``` ### Editing /etc/systemd/system/influxdb3-enterprise.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file IPAddressDeny=any IPAddressAllow=localhost ### Lines below this comment will be discarded ... ``` Alternatively, to restrict networking to only public IP ranges, use this instead: ``` ### Editing /etc/systemd/system/influxdb3-enterprise.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file IPAddressDeny=0.0.0.0/32 # 0.0.0.0 treated as 127.0.0.1 IPAddressDeny=127.0.0.0/8 # IPv4 loopback IPAddressDeny=10.0.0.0/8 # IPv4 internal (RFC1918) IPAddressDeny=172.16.0.0/12 # IPv4 internal (RFC1918) IPAddressDeny=192.168.0.0/16 # IPv4 internal (RFC1918) IPAddressDeny=169.254.0.0/16 # IPv4 link-local (RFC3927) IPAddressDeny=224.0.0.0/4 # IPv4 multicast IPAddressDeny=::1/128 # IPv6 loopback IPAddressDeny=fe80::/64 # IPv6 link-local IPAddressDeny=fc00::/7 # IPv6 unique local addr IPAddressDeny=ff00::/8 # IPv6 multicast ### Lines below this comment will be discarded ... ``` #### Example: add memory, CPU and I/O control for process node If InfluxDB 3 Enterprise is configured to start as a standalone processing engine node (for example, started with `--mode="process"`), then it could utilize different security directives than the database itself. For example, consider the following`systemd` override for limiting a processing engine-only node: ``` ### Editing /etc/systemd/system/influxdb3-enterprise.service.d/override.conf ### Anything between here and the comment below will become the new contents of the file [Service] # Memory - limit to 20% of the memory, killing it and restarting the service # if it reaches 30% MemoryHigh=20% MemoryMax=30% Restart=on-failure RestartSec=5 StartLimitIntervalSec=600 StartLimitBurst=5 OOMPolicy=continue # CPU - Limit to maximum of 2 CPUs with deprioritized nice value Nice=10 CPUQuota=200% CPUSchedulingPolicy=batch LimitNPROC=256 TasksMax=256 # I/O - Limit I/O to not starve main database IOWeight=50 ### Lines below this comment will be discarded ... ``` ### systemd references See the `systemd` documentation for additional information: * [systemd](https://www.freedesktop.org/software/systemd/man/latest/) * [systemd.service](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html) * [systemd.exec](https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html) * [systemd.resource-control](https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html) #### Related --- # Manage users and authentication Source: https://docs.influxdata.com/influxdb3/enterprise/admin/security/manage-users/ > [!Note] > #### User authentication is off by default > > Multi-user authentication and role-based access control (RBAC) are > generally available in InfluxDB 3 Enterprise 3.12, but remain **opt-in**. > Start the server with `--user-auth-type` to turn it on. Existing `apiv3_`token workflows are unaffected. Multi-user authentication lets users log in to InfluxDB 3 Enterprise with individual credentials that issue JSON Web Tokens (JWTs). Access is governed by[role-based access control (RBAC)](/influxdb3/enterprise/reference/internals/rbac/). Multi-user authentication complements, but doesn’t replace, `apiv3_`[token authentication](/influxdb3/enterprise/admin/tokens/). ## Enable user authentication User authentication is off by default (`--user-auth-type none`). To enable it, start the server with `--user-auth-type` set to one or both authentication methods: ```bash # Username and password sign-in influxdb3 serve --user-auth-type basic # OAuth/OIDC sign-in influxdb3 serve --user-auth-type oauth # Both influxdb3 serve --user-auth-type basic,oauth ``` `none` disables user authentication and can’t be combined with other values.`--user-auth-type` has no effect when the server starts with `--without-auth`. For the complete list of authentication serve flags, see the[`influxdb3 serve`](/influxdb3/enterprise/reference/cli/influxdb3/serve/) CLI reference and[configuration options](/influxdb3/enterprise/reference/config-options/#user-auth-type). ## Configure JWT signing keys `basic` sign-in requires an RSA key pair to sign JWTs. Provide the key ID and private key with `--jwt-key-id` and`--jwt-private-key`. The private key **must be in PKCS#1 format**. Generate a compatible key with the `-traditional` flag: ```bash openssl genrsa -traditional -out jwt-private-key.pem 2048 ``` ```bash influxdb3 serve \ --user-auth-type basic \ --jwt-key-id JWT_KEY_ID \ --jwt-private-key "$(cat jwt-private-key.pem)" ``` Replace the following: * `JWT_KEY_ID`: an identifier for the key (used in the JWT `kid` header; any string you choose) > [!Warning] > #### Use PKCS#1 keys, not PKCS#8 > > A PKCS#8 key (the default `openssl genrsa` output without `-traditional`) > is rejected at startup: the server logs a warning and disables password > sign-in. Always generate the key with `openssl genrsa -traditional`. If only one of `--jwt-key-id` and `--jwt-private-key` is set, or the key is invalid, the server logs a warning, starts anyway, and disables password sign-in. If neither is set, it starts with password sign-in disabled and logs nothing at the default log level. `oauth` sign-in and `apiv3_` token authentication are unaffected. JWTs expire one hour after they’re issued. ## Bootstrap the initial admin On a new deployment with no users and no operator token, create the initial admin user and operator token with `influxdb3 manage init-admin`: ```bash influxdb3 manage init-admin ``` `init-admin` only works while the cluster has no users and no operator token. On a deployment that already has an operator token (for example, one that used token authentication before you enabled user authentication), it returns `409 operator token already configured`. Create the first admin user with the operator token instead: ```bash influxdb3 create user --username USERNAME --role Admin --token OPERATOR_TOKEN ``` If you enabled only `oauth` sign-in (no `--jwt-key-id` and`--jwt-private-key`), pass the subject (`sub`) claim your identity provider issues for that person instead of a username and password: ```bash influxdb3 manage init-admin --oauth-id OAUTH_SUBJECT ``` Replace `OAUTH_SUBJECT`with the `sub` claim your identity provider issues for that person. `init-admin` prints the operator token once. Store it securely: there’s no way to retrieve it again. For complete syntax, see the[`influxdb3 manage`](/influxdb3/enterprise/reference/cli/influxdb3/manage/) CLI reference. ## Log in and out Users authenticate with `influxdb3 auth login` and end their session with`influxdb3 auth logout`: ```bash influxdb3 auth login ``` Credentials are stored at `~/.influxdb3/credentials.json` and refreshed automatically. > [!Note] > `influxdb3 auth logout` removes the local credentials but does **not** revoke > the issued JWT server-side. The token remains valid until it expires, one > hour after it was issued. ## Upgrading to 3.12: sign in again if a token is rejected 3.12 requires each JWT’s `aud` (audience) claim to match the cluster’s catalog UUID. Access tokens issued to `basic` sign-in users **before** upgrading to 3.12 don’t have a matching claim and are rejected after the upgrade. Clients that call `POST /api/v3/authorize/refresh` with a stored refresh token get a valid access token. The `influxdb3` CLI refreshes only when its stored token is within 30 minutes of expiry, so if a command is rejected after the upgrade, sign in again with `influxdb3 auth login`. `apiv3_` API tokens are unaffected by this change. ## Optional: Authenticate with OAuth/OIDC To delegate authentication to an OAuth/OIDC identity provider, start the server with `--user-auth-type oauth` (or `basic,oauth`) and set`--oauth-issuer`, `--oauth-audience`, and `--oauth-client-id`: ```bash influxdb3 serve \ --user-auth-type oauth \ --oauth-issuer OAUTH_ISSUER_URL \ --oauth-audience OAUTH_AUDIENCE \ --oauth-client-id OAUTH_CLIENT_ID ``` Replace the following: * `OAUTH_ISSUER_URL`: the identity provider’s issuer URL * `OAUTH_AUDIENCE`: the audience to validate incoming OAuth tokens against * `OAUTH_CLIENT_ID`: the OAuth client ID registered with the identity provider `--oauth-client-id` is also required for `influxdb3 auth login --oauth` and for browser sign-in to the[integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/). For browser-based SSO through the integrated Explorer UI, also set`--webui-public-uri`. See[Use the integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/#configure-sso-for-the-explorer-ui). ## Roles InfluxDB 3 Enterprise includes three built-in roles–**Admin**, **Auditor**, and**Member**. Assign roles to users to control what they can do. For details on each role and the permissions model, see[Role-based access control (RBAC)](/influxdb3/enterprise/reference/internals/rbac/). To add users, assign roles, and delete users in the Explorer UI, see[Manage users with InfluxDB 3 Explorer](/influxdb3/explorer/manage-users/). #### Related * [Role-based access control (RBAC)](/influxdb3/enterprise/reference/internals/rbac/) * [Manage tokens](/influxdb3/enterprise/admin/tokens/) * [Use the integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/) * [Manage users with InfluxDB 3 Explorer](/influxdb3/explorer/manage-users/) --- # Manage tables Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tables/ Tables are synonymous with measurements in InfluxDB 3 Enterprise. They are typically created automatically when you write line protocol data, but you can also manually create them to define custom schemas or apply settings. before writing data. ### [Create a table](/influxdb3/enterprise/admin/tables/create/) Use the influxdb3 CLI or HTTP API to create a table in a specified database in InfluxDB 3 Enterprise. ```sh # influxdb3 CLI influxdb3 create table \ --tags tag1,tag2,tag3 \ --retention-period 7d \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME # HTTP API curl --request POST "http://localhost:8181/api/v3/configure/table" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["tag1", "tag2", "tag3"], "retention_period": "7d" }' ``` ### [List tables](/influxdb3/enterprise/admin/tables/list/) Use the influxdb3 CLI or HTTP API to list tables in a specified database in InfluxDB 3 Enterprise. Use SQL SHOW TABLES or InfluxQL SHOW MEASUREMENTS statements. ```sh # influxdb3 CLI influxdb3 query \ --database DATABASE_NAME \ --token AUTH_TOKEN \ "SHOW TABLES" # HTTP API curl --get "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SHOW TABLES" ``` ### [Delete a table](/influxdb3/enterprise/admin/tables/delete/) Use the [`influxdb3 delete table` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/table/) or the [HTTP API](/influxdb3/enterprise/api/v3/) to delete a table from a specified database in InfluxDB 3 Enterprise. Supports both soft delete and hard delete operations. ```sh # CLI influxdb3 delete table \ --database \ --token \ # HTTP API curl -X DELETE "http://localhost:8181/api/v3/configure/table?db=&table=" \ --header "Authorization: Bearer " ``` ### [Add columns to a table](/influxdb3/enterprise/admin/tables/update/) Use the [`influxdb3 update table` command](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) or the [HTTP API](/influxdb3/enterprise/api/v3/) to add tag and field columns to an existing table in InfluxDB 3 Enterprise. ```sh # CLI influxdb3 update table \ --database \ --token \ --tags rack,zone \ --fields temp:float64 \ # HTTP API curl --request PATCH "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer " \ --header "Content-Type: application/json" \ --data '{ "db": "", "table": "", "tags": ["rack", "zone"], "fields": [{"name": "temp", "type": "float64"}] }' ``` #### Related * [InfluxDB schema design recommendations](/influxdb3/enterprise/write-data/best-practices/schema-design/) * [influxdb3 CLI](/influxdb3/enterprise/reference/cli/influxdb3/) * [Table API reference](/influxdb3/enterprise/api/v3/#tag/Table) * [Naming restrictions and conventions](/influxdb3/enterprise/reference/naming-restrictions/) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) [tables](/influxdb3/enterprise/tags/tables/) --- # Create a table Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tables/create/ Use the [`influxdb3 create table` command](/influxdb3/enterprise/reference/cli/influxdb3/create/table/)or the [HTTP API](/influxdb3/enterprise/api/v3/) to create a table in a specified database in InfluxDB 3 Enterprise. With InfluxDB 3 Enterprise, tables and measurements are synonymous. Typically, tables are created automatically on write using the table name specified in line protocol written to InfluxDB. However, you can manually create tables to define a custom schema or apply custom settings before writing data. * [Create a table using the influxdb3 CLI](#create-a-table-using-the-influxdb3-cli) * [Create a table using the HTTP API](#create-a-table-using-the-http-api) * [Retention period](#retention-period) * [Table naming restrictions](#table-naming-restrictions) > [!Important] > #### Tables with no fields can’t be queried > > In InfluxDB 3 Enterprise 3.12, a query on a table that has tags but no fields returns HTTP `500` with `field_family_projections is required`. > This applies to the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/), the default for new clusters. > Define at least one field when you create a table. ## Create a table using the influxdb3 CLI Use the `influxdb3 create table` command and provide the following: * *Required*: The name of the database to create the table in * *Required*: The name of the table to create (see [Table naming restrictions](#table-naming-restrictions)) * *Required*: Tag columns to include in the table (must have at least one tag column) * *Optional*: Field columns and their data types to include in the table * *Optional*: [Retention period](#retention-period). If omitted, uses database retention period. > [!Note] > Tables must include at least one tag column. > Field columns are optional and can be added later when you write data. > [!Important] > #### Tag order affects query performance > > When considering your schema and creating your table, order your tags by query priority. > Place the most commonly queried tags first. > Columns that appear earlier are typically faster to filter and access during query execution. > > For more information, see [Optimize writes](/influxdb3/enterprise/write-data/best-practices/optimize-writes/#sort-tags-by-query-priority). ```bash # Create a table with tag columns influxdb3 create table \ --tags tag1,tag2,tag3 \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME # Create a table with tag and field columns influxdb3 create table \ --tags room,sensor_id \ --fields temp:float64,hum:float64,co:int64 \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` ```bash # Create a table with a 7-day retention period influxdb3 create table \ --tags room,sensor_id \ --fields temp:float64,hum:float64 \ --retention-period 7d \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME # Create a table with database default retention (omit --retention-period) influxdb3 create table \ --tags room,sensor_id \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database to create the table in * `TABLE_NAME`: the name of the table to create * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ## Create a table using the HTTP API To create a table using the HTTP API, send a `POST` request to the `/api/v3/configure/table` endpoint: ``` POST http://localhost:8181/api/v3/configure/table ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your authentication token * `Content-Type: application/json` * **Request body**: JSON object with table configuration * `db` *(string, required)*: Database name * `table` *(string, required)*: Table name * `tags` *(array, required)*: Tag column names * `fields` *(array, required)*: Field definitions with name and type. Provide an empty array if no fields are defined. * `retention_period` *(string, optional)*: [Retention period](#retention-period). If omitted, uses database retention period. ```bash # Create a table with tag columns curl -X POST "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["tag1", "tag2", "tag3"], "fields": [] }' # Create a table with tag and field columns curl -X POST "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["room", "sensor_id"], "fields": [ {"name": "temp", "type": "float64"}, {"name": "hum", "type": "float64"}, {"name": "co", "type": "int64"} ] }' ``` ```bash # Create a table with a 7-day retention period curl -X POST "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["room", "sensor_id"], "fields": [ {"name": "temp", "type": "float64"}, {"name": "hum", "type": "float64"} ], "retention_period": "7d" }' # Create a table with database default retention (omit retention_period) curl -X POST "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["room", "sensor_id"], "fields": [] }' ``` Replace the following: * `DATABASE_NAME`: the name of the database to create the table in * `TABLE_NAME`: the name of the table to create * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ## Retention period By default, tables use the database retention period. When creating a table, set a table-specific *retention period* to automatically delete expired data and optimize storage for that table. A table retention period overrides the database retention period for that specific table. This allows you to maintain different retention policies for different types of data within the same database. ### Retention period syntax The retention period value is a time duration value made up of a numeric value plus a duration unit. For example, `7d` means 7 days. The retention period value cannot be negative or contain whitespace. #### Valid durations units include * **m**: minute * **h**: hour * **d**: day * **w**: week * **mo**: month * **y**: year > [!Warning] > #### Retention period constraints > > * **Minimum for data retention**: The practical minimum retention period is 1 hour (`1h`). > * **Zero-duration periods**: Setting a retention period to `0` (for example,`0d` or `0h`) is allowed but marks all data for immediate deletion at query time.*This differs from InfluxDB 1.x and 2.x where `0d` meant infinite retention.* > * **Infinite retention**: Use `none` to set an infinite retention period. ### Retention period precedence When both a database and a table have retention periods defined, the **table retention period takes precedence** for that specific table. **Example scenarios:** * **Short-term data in long-term database**: Store high-volume sensor data with 7-day retention in a database with 90-day retention * **Long-term data in short-term database**: Store audit logs with 1-year retention in a database with 30-day retention * **Mixed retention requirements**: Maintain both real-time metrics (7 days) and aggregated metrics (1 year) in the same database For complete details about retention period precedence and behavior, see[Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/#retention-period-precedence). ## Table naming restrictions Table names in InfluxDB 3 Enterprise must adhere to the following naming restrictions: * **Allowed characters**: Alphanumeric characters (a-z, A-Z, 0-9), underscore (`_`), dash (`-`) * **Starting character**: Should start with a letter or number and should not start with underscore (`_`) * **Case sensitivity**: Table names are case-sensitive * **Quoting**: Use double quotes when names contain special characters or whitespace > [!Caution] > #### Underscore prefix reserved for system use > > Names starting with an underscore (`_`) may be reserved for InfluxDB system use. > While InfluxDB 3 Enterprise might not explicitly reject these names, using them risks > conflicts with current or future system features and may result in > unexpected behavior or data loss. #### Related * [influxdb3 create table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) * [Create table API](/influxdb3/enterprise/api/table/#operation/PostConfigureTable) * [Naming restrictions and conventions](/influxdb3/enterprise/reference/naming-restrictions/) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) --- # Delete a table Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tables/delete/ Use the [`influxdb3 delete table` command](/influxdb3/enterprise/reference/cli/influxdb3/delete/table/)or the [HTTP API](/influxdb3/enterprise/api/v3/) to delete a table from a specified database in InfluxDB 3 Enterprise. With InfluxDB 3 Enterprise, tables and measurements are synonymous. By default, InfluxDB 3 Enterprise performs a soft delete, which schedules the table for deletion and makes it unavailable for querying. You can also schedule a hard deletion to permanently remove the table and its data. > [!Caution] > #### Deleting a table cannot be undone > > Deleting a table is a destructive action. > Once a table is deleted, data stored in that table cannot be recovered. * [Delete a table using the influxdb3 CLI](#delete-a-table-using-the-influxdb3-cli) * [Delete a table using the HTTP API](#delete-a-table-using-the-http-api) * [Delete data only (preserve schema and resources)](#delete-data-only-preserve-schema-and-resources) ## Delete a table using the influxdb3 CLI Use the `influxdb3 delete table` command to delete a table: ```sh influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database containing the table * `TABLE_NAME`: the name of the table to delete * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Hard delete a table immediately To permanently delete a table and its data immediately, use the `--hard-delete now` flag: ```sh influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --hard-delete now \ TABLE_NAME ``` ### Schedule a hard deletion To schedule a table for hard deletion at a specific time, use the `--hard-delete` flag with a timestamp: ```sh influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --hard-delete "2025-12-31T23:59:59Z" \ TABLE_NAME ``` ## Delete a table using the HTTP API To delete a table using the HTTP API, send a `DELETE` request to the `/api/v3/configure/table` endpoint: ``` DELETE http://localhost:8181/api/v3/configure/table ``` Include the following in your request: * **Query parameters**: * `db`: Database name * `table`: Table name to delete * `hard_delete_at`: *(Optional)* Timestamp for hard deletion * **Headers**: * `Authorization: Bearer` with your authentication token ### Soft delete a table ```bash curl -X DELETE "http://localhost:8181/api/v3/configure/table?db=DATABASE_NAME&table=TABLE_NAME" \ --header "Authorization: Bearer AUTH_TOKEN" ``` Replace the following: * `DATABASE_NAME`: the name of the database containing the table * `TABLE_NAME`: the name of the table to delete * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Schedule a hard deletion To schedule a hard deletion at a specific time, include the `hard_delete_at` parameter with an ISO 8601 timestamp: ```bash curl -X DELETE "http://localhost:8181/api/v3/configure/table?db=DATABASE_NAME&table=TABLE_NAME&hard_delete_at=2025-12-31T23:59:59Z" \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### Response A successful deletion returns HTTP status `200` with no content body. #### Example error response If the table doesn’t exist, the API returns HTTP status `404`: ```json { "error": "Table not found" } ``` ## Delete data only (preserve schema and resources) InfluxDB 3 Enterprise supports deleting only the data in a table while preserving the table schema and associated resources. This is useful when you want to clear old data and re-write new data to the same table structure without recreating resources. ### What is preserved When using the data-only deletion option, the following are preserved: * **Table schema**: Column definitions and data types * **Caches**: Last value caches (LVC) and distinct value caches (DVC) associated with the table ### Delete data only using the CLI Use the [`--data-only`](/influxdb3/enterprise/reference/cli/influxdb3/delete/table/#options) flag to delete data while preserving the table schema and resources: ```sh influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --data-only \ TABLE_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database containing the table * `TABLE_NAME`: the name of the table * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Delete data only using the HTTP API To delete only data using the HTTP API, include the `data_only=true` query parameter: ```bash curl -X DELETE "http://localhost:8181/api/v3/configure/table?db=DATABASE_NAME&table=TABLE_NAME&data_only=true" \ --header "Authorization: Bearer AUTH_TOKEN" ``` Replace the following: * `DATABASE_NAME`: the name of the database containing the table * `TABLE_NAME`: the name of the table * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) #### Related * [influxdb3 delete table](/influxdb3/enterprise/reference/cli/influxdb3/delete/table/) * [Delete table API](/influxdb3/enterprise/api/table/#operation/DeleteConfigureTable) --- # List tables Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tables/list/ Use the [`influxdb3 query` command](/influxdb3/enterprise/reference/cli/influxdb3/query/)or the [HTTP API](/influxdb3/enterprise/api/v3/) to list tables in a specified database in InfluxDB 3 Enterprise. With InfluxDB 3 Enterprise, tables and measurements are synonymous. This guide shows how to retrieve a list of all tables (measurements) in a database. * [List tables using the influxdb3 CLI](#list-tables-using-the-influxdb3-cli) * [List tables using the HTTP API](#list-tables-using-the-http-api) ## List tables using the influxdb3 CLI Use the `influxdb3 query` command with the `SHOW TABLES` SQL statement: ```sh influxdb3 query \ --database DATABASE_NAME \ --token AUTH_TOKEN \ "SHOW TABLES" ``` Replace the following: * `DATABASE_NAME`: the name of the database to list tables from * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Example output ``` +---------------+---------------+------------+------------+ | table_catalog | table_schema | table_name | table_type | +---------------+---------------+------------+------------+ | public | iox | home | BASE TABLE | | public | iox | sensors | BASE TABLE | +---------------+---------------+------------+------------+ ``` ### Alternative: List tables using InfluxQL You can also use InfluxQL syntax to list measurements (tables): ```sh influxdb3 query \ --language influxql \ --database DATABASE_NAME \ --token AUTH_TOKEN \ "SHOW MEASUREMENTS" ``` ## List tables using the HTTP API To list tables using the HTTP API, send a `GET` request to the `/api/v3/query_sql` endpoint with a `SHOW TABLES` query: ``` GET http://localhost:8181/api/v3/query_sql ``` Include the following in your request: * **Query parameters**: * `db`: Database name * `q`: The SQL query (`SHOW TABLES`) * `format`: Response format (optional, defaults to `json`) * **Headers**: * `Authorization: Bearer` with your authentication token ```bash curl --get "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SHOW TABLES" \ --data-urlencode "format=json" ``` Replace the following: * `DATABASE_NAME`: the name of the database to list tables from * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Example response ```json { "results": [ { "series": [ { "name": "tables", "columns": ["table_catalog", "table_schema", "table_name", "table_type"], "values": [ ["public", "iox", "home", "BASE TABLE"], ["public", "iox", "sensors", "BASE TABLE"] ] } ] } ] } ``` ### Get response in CSV format To get the response in CSV format, set the `format` parameter to `csv`: ```bash curl --get "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SHOW TABLES" \ --data-urlencode "format=csv" ``` #### Related * [influxdb3 query](/influxdb3/enterprise/reference/cli/influxdb3/query/) * [Query API](/influxdb3/enterprise/api/query-data/#operation/GetExecuteQuerySQL) --- # Add columns to a table Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tables/update/ Use the [`influxdb3 update table` command](/influxdb3/enterprise/reference/cli/influxdb3/update/table/)or the [HTTP API](/influxdb3/enterprise/api/v3/) to add tag and field columns to an existing table in InfluxDB 3 Enterprise. Adding columns is *additive only*. You can’t remove, rename, or change the type of an existing column. Adding a column that already exists with the same data type is a no-op. Adding a column that already exists with a different data type returns an error. This works for tables in databases that use either[schema mode](/influxdb3/enterprise/admin/databases/create/#schema-mode). In a database that uses `explicit` schema mode, declaring a column is what allows writes that reference it to succeed. For more information, see[Enforce a schema](/influxdb3/enterprise/admin/databases/enforce-schema/). * [Add columns using the influxdb3 CLI](#add-columns-using-the-influxdb3-cli) * [Add columns using the HTTP API](#add-columns-using-the-http-api) ## Add columns using the influxdb3 CLI Use the `influxdb3 update table` command with `--tags`, `--fields`, or both, and provide the following: * *Required*: The name of the database containing the table * *Required*: The name of the table to update * *Required*: At least one of `--tags` or `--fields` `--tags` accepts one or more values, so place the table name before it or follow it with another option, as in the examples below. ```sh # Add tag columns influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME \ --tags rack,zone # Add field columns influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --fields temp:float64,active:bool \ TABLE_NAME # Add tag and field columns in one command influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --tags rack \ --fields temp:float64 \ TABLE_NAME ``` Replace the following: * `DATABASE_NAME`: the name of the database containing the table * `TABLE_NAME`: the name of the table to update * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) See [Field data types](/influxdb3/enterprise/reference/cli/influxdb3/create/table/#field-data-types)for the list of valid field types. > [!Important] > #### Add columns and update retention in separate commands > > `--tags` and `--fields` add columns to the table.`--retention-period` updates the table’s retention period. > These are separate operations against separate API endpoints, so > InfluxDB 3 Enterprise rejects a command that combines `--retention-period`with `--tags` or `--fields`. > Run them as separate commands. > For more information about table retention periods, see[Data retention](/influxdb3/enterprise/reference/internals/data-retention/). ## Add columns using the HTTP API To add columns using the HTTP API, send a `PATCH` request to the `/api/v3/configure/table` endpoint: ``` PATCH http://localhost:8181/api/v3/configure/table ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your authentication token * `Content-Type: application/json` * **Request body**: JSON object with the columns to add * `db` *(string, required)*: Database name * `table` *(string, required)*: Table name * `tags` *(array, optional)*: Tag column names to add * `fields` *(array, optional)*: Field definitions to add, each with a `name` and a `type` Provide at least one of `tags` or `fields`. ```bash # Add tag columns curl --request PATCH "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["rack", "zone"], "fields": [] }' # Add field columns curl --request PATCH "http://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": [], "fields": [ {"name": "temp", "type": "float64"}, {"name": "active", "type": "bool"} ] }' ``` Replace the following: * `DATABASE_NAME`: the name of the database containing the table * `TABLE_NAME`: the name of the table to update * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Response A successful request returns HTTP status `200` with no content body. #### Example error responses An empty request that names no tags and no fields returns HTTP status `400`: ```text invalid request: at least one of tags or fields is required ``` Adding a column that already exists with a different data type returns HTTP status `400`. Patching a table that doesn’t exist returns HTTP status `404`. Use [`POST /api/v3/configure/table`](/influxdb3/enterprise/admin/tables/create/#create-a-table-using-the-http-api)to create the table first. > [!Note] > `PATCH /api/v3/configure/table` adds columns only and never touches > the table’s retention period. > To update a table’s retention period, use `PUT /api/v3/configure/table`. > [!Important] > #### PUT doesn’t add columns > > `PUT /api/v3/configure/table` updates only the retention period. > The endpoint ignores a `tags` or `fields` array in the request body and still returns HTTP status `200`. > In a database that uses `explicit` schema mode, the next write that references the undeclared column fails. > To add columns, use `PATCH /api/v3/configure/table` or `influxdb3 update table`. #### Related * [influxdb3 update table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) * [influxdb3 create table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) * [Table API reference](/influxdb3/enterprise/api/v3/#tag/Table) * [Enforce a schema](/influxdb3/enterprise/admin/databases/enforce-schema/) --- # Manage tokens Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/ Manage tokens to authenticate and authorize access to server actions, resources, and data in your InfluxDB 3 Enterprise instance. ## Provide your token If you start the InfluxDB 3 Enterprise server with authentication enabled (the default), future server actions (CLI commands and HTTP API requests) require a valid token for authorization. The first admin token you create is the *operator* token (named `_admin`), which has full administrative privileges. You can use the operator token to authenticate your requests and manage additional authorization tokens. The mechanism for providing your token depends on the client you use to interact with InfluxDB 3 Enterprise–for example: [influxdb3 CLI](#influxdb3-cli)[HTTP API](#http-api) When using the `influxdb3` CLI, you can set the `INFLUXDB3_AUTH_TOKEN` environment variable to automatically provide your authorization token to all `influxdb3` commands–for example: ```bash # Export your token as an environment variable export INFLUXDB3_AUTH_TOKEN=YOUR_AUTH_TOKEN # Run an influxdb3 command influxdb3 query \ --database DATABASE_NAME \ "SELECT * FROM 'DATABASE_NAME' WHERE time > now() - INTERVAL '10 minutes'" ``` To specify a token in the command and override the environment variable, pass the `--token` option with your authorization token–for example: ```bash # Include the --token option in your influxdb3 command influxdb3 query \ --token YOUR_AUTH_TOKEN \ --database DATABASE_NAME \ "SELECT * FROM 'DATABASE_NAME' WHERE time > now() - INTERVAL '10 minutes'" ``` You can also set the `INFLUXDB3_AUTH_TOKEN` environment variable to automatically provide your authorization token to all `influxdb3` commands. To authenticate directly to the HTTP API, you can include your authorization token in the HTTP Authorization header of your request. The `Authorization: Bearer AUTH_TOKEN` scheme works with all HTTP API endpoints that require authentication. The following examples use `curl` to show to authenticate to the HTTP API. ```bash # Add your token to the HTTP Authorization header curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM 'DATABASE_NAME' WHERE time > now() - INTERVAL '10 minutes'" ``` ### Authenticate using v1 and v2 compatibility InfluxDB 3 provides compatibility with InfluxDB v1 and v2 APIs, allowing you to use the same authentication methods as in those versions. With InfluxDB v1-compatible endpoints in InfluxDB 3, you can use database tokens in InfluxDB 1.x username and password scheme. With the InfluxDB v2-compatible `/api/v2/write` endpoint, you can use tokens in the InfluxDB v2 `Authorization: Token` scheme or in the OAuth `Authorization: Bearer` scheme. The following examples show how to authenticate with the InfluxDB v1-compatible and v2-compatible APIs in InfluxDB 3: ```bash # Token scheme with v2 /api/v2/write curl http://localhost:8181/api/v2/write\?bucket\=DATABASE_NAME \ --header "Authorization: Token AUTH_TOKEN" \ --data-raw "home,room=Kitchen temp=23.5 1622547800" ``` ```bash # Basic scheme with v1 /write # Username is ignored, but required for the request # Password is your auth token encoded in base64 curl "http://localhost:8181/write?db=DATABASE_NAME" \ --user "any:AUTH_TOKEN" \ --data-raw "home,room=Kitchen temp=23.5 1622547800" ``` ```bash # URL auth parameters with v1 /write # Username is ignored, but required for the request curl "http://localhost:8181/write?db=DATABASE_NAME&u=any&p=AUTH_TOKEN" \ --data-raw "home,room=Kitchen temp=23.5 1622547800" ``` Replace the following with your values: * `AUTH_TOKEN`: your [token](/influxdb3/enterprise/admin/tokens/) * `DATABASE_NAME`: the name of the [database](/influxdb3/enterprise/admin/databases) you want to query To use tokens with other clients for InfluxDB 3 Enterprise, see the client-specific documentation: * [InfluxDB 3 Explorer](/influxdb3/explorer/) * [InfluxDB client libraries](/influxdb3/enterprise/reference/client-libraries/) * [Telegraf](/telegraf/v1/) * [Grafana](/influxdb3/enterprise/visualize-data/grafana/) ## [Manage admin tokens](/influxdb3/enterprise/admin/tokens/admin/) Manage admin tokens in your InfluxDB 3 Enterprise instance. An admin token grants access to all actions (CLI commands and API endpoints) for the server. ## [Manage resource tokens](/influxdb3/enterprise/admin/tokens/resource/) Manage resource tokens in your InfluxDB 3 Enterprise instance. Resource tokens grant permissions on specific resources, such as databases and system information endpoints in your InfluxDB 3 Enterprise instance. Database resource tokens allow for actions like writing and querying data. #### Related * [Manage tokens with InfluxDB 3 Explorer](/influxdb3/explorer/manage-tokens/) --- # Manage admin tokens Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/admin/ Manage InfluxDB 3 Enterprise admin tokens to authorize server actions, `influxdb3` CLI commands, and HTTP API endpoints for your InfluxDB 3 Enterprise instance. Administrative (*admin*) tokens provide full system access and management capabilities for your InfluxDB 3 Enterprise instance. Admin tokens can create, edit, and delete other admin tokens, as well as manage [resource tokens](/influxdb3/enterprise/admin/tokens/resource/). InfluxDB 3 Enterprise supports two types of admin tokens: * **Operator token**: A system-generated administrative token with the name `_admin`. * Cannot be edited or deleted * Never expires * Cannot be recreated if lost (future functionality) * Can be regenerated using the CLI * **Named admin token**: User-defined administrative tokens with full admin permissions. * Can be created, edited, and deleted * Support expiration dates * Cannot modify or remove the operator token An InfluxDB 3 Enterprise instance can have one operator token and unlimited named admin tokens. ## [Create an admin token](/influxdb3/enterprise/admin/tokens/admin/create/) Use the [`influxdb3 create token --admin` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/) or the HTTP API [`/api/v3/configure/token/admin`](/influxdb3/enterprise/api/authentication/#operation/PostCreateAdminToken) endpoint to create an operator or named [admin token](/influxdb3/enterprise/admin/tokens/admin/) for your InfluxDB 3 Enterprise instance. An admin token grants access to all actions on the server. ##### CLI ```bash influxdb3 create token --admin --name TOKEN_NAME ``` #### HTTP API ```bash curl -X POST "http://localhost:8181/api/v3/configure/token/admin" \ --header 'Authorization: Bearer ADMIN_TOKEN' \ --json '{ "name": "TOKEN_NAME" }' ``` [Read more ](/influxdb3/enterprise/admin/tokens/admin/create/) ## [List admin tokens](/influxdb3/enterprise/admin/tokens/admin/list/) Use the `influxdb3` CLI or the `/api/v3` HTTP API to list admin tokens for your InfluxDB 3 Enterprise instance. Use the `influxdb3 show tokens` command to list all tokens or use SQL to query token metadata directly from the `system.tokens` table. ##### CLI ```bash influxdb3 show tokens ``` ##### HTTP API ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens WHERE permissions = '*:*:*'" \ --header 'Accept: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" ``` [Read more ](/influxdb3/enterprise/admin/tokens/admin/list/) ## [Regenerate an operator admin token](/influxdb3/enterprise/admin/tokens/admin/regenerate/) Use the [`influxdb3 create token --admin` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/) or the [HTTP API](/influxdb3/enterprise/api/v3/) to regenerate an [operator token](/influxdb3/enterprise/admin/tokens/admin/) for your InfluxDB 3 Enterprise instance. Regenerating an admin token deactivates the previous token. ##### CLI ```bash influxdb3 create token --admin \ --token OPERATOR_TOKEN \ --regenerate ``` #### HTTP API ```bash curl -X POST "http://localhost:8181/api/v3/configure/token/admin/regenerate" \ --header 'Authorization: Bearer OPERATOR_TOKEN' ``` [Read more ](/influxdb3/enterprise/admin/tokens/admin/regenerate/) ## [Use a preconfigured admin token](/influxdb3/enterprise/admin/tokens/admin/preconfigured/) Start InfluxDB 3 Enterprise with a preconfigured “offline” admin token file. If no admin tokens already exist, InfluxDB automatically creates an admin token using the provided admin token file. #### Related * [Manage tokens with InfluxDB 3 Explorer](/influxdb3/explorer/manage-tokens/) [tokens](/influxdb3/enterprise/tags/tokens/) --- # Create an admin token Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/admin/create/ Use the [`influxdb3 create token --admin` subcommand](/influxdb3/enterprise/reference/cli/influxdb3/create/token/)with the `--name` option or the HTTP API [`/api/v3/configure/token/admin`](/influxdb3/enterprise/api/v3/) endpoint to create an admin token for your InfluxDB 3 Enterprise instance. An admin token grants full access to all actions for your InfluxDB 3 instance and can be referenced by its name. InfluxDB 3 Enterprise supports two types of admin tokens: * **Operator token**: A system-generated administrative token with the name `_admin`. * Cannot be edited or deleted * Never expires * Cannot be recreated if lost (future functionality) * Can be regenerated using the CLI * **Named admin token**: User-defined administrative tokens with full admin permissions. * Can be created, edited, and deleted * Support expiration dates * Cannot modify or remove the operator token An InfluxDB 3 Enterprise instance can have one operator token and unlimited named admin tokens. [Create an operator token](#create-an-operator-token)[Create a named admin token](#create-a-named-admin-token) > [!Note] > #### Store secure tokens in a secret store > > Token strings are returned *only* on token creation. > We recommend storing database tokens in a **secure secret store**. > Anyone with access to the named admin token has full control over your InfluxDB 3 Enterprise instance. > If you lose the named admin token string, you must regenerate the token. ## Create an operator token The first admin token your create for your InfluxDB 3 Enterprise instance is the operator token. [CLI](#cli)[HTTP API](#http-api) Use the `influxdb3 create token --admin` command without a token name: ```bash influxdb3 create token --admin ``` Use the following endpoint to create an operator token: ``` POST /api/v3/configure/token/admin ``` ```bash curl -X POST "http://localhost:8181/api/v3/configure/token/admin" \ ``` The output contains the token string in plain text. ## Create a named admin token [CLI](#cli-2)[HTTP API](#http-api-2) Use the `influxdb3 create token --admin` command with a token name: ```bash influxdb3 create token --admin --token ADMIN_TOKEN --name "TOKEN_NAME" ``` Replace the following with your values: * `ADMIN_TOKEN`: your existing operator or named admin token * `TOKEN_NAME`: the name you want to assign to the new admin token The output contains the token string in plain text. Use the following endpoint to create a named admin token: ``` POST /api/v3/configure/token/admin ``` ```bash curl -X POST "http://localhost:8181/api/v3/configure/token/admin" \ --header 'Authorization: Bearer ADMIN_TOKEN' \ --json '{ "name": "TOKEN_NAME" }' ``` Replace the following with your values: * `ADMIN_TOKEN`: your existing operator or named admin token * `TOKEN_NAME`: the name you want to assign to the new admin token The response body contains the token string in plain text. *To use the token as the default for later commands, and to persist the token across sessions, assign the token string to the `INFLUXDB3_AUTH_TOKEN` environment variable.* #### Related * [Manage tokens with InfluxDB 3 Explorer](/influxdb3/explorer/manage-tokens/) --- # List admin tokens Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/admin/list/ Use the `influxdb3` CLI or the `/api/v3` HTTP API to list *admin* tokens for your InfluxDB 3 Enterprise instance. Admin tokens have `permissions=*:*:*`, which allows access to all data and resources in your InfluxDB 3 instance. > [!Note] > Token metadata includes the hashed token string. > InfluxDB 3 does not store the raw token string. > [!Important] > #### Required permissions > > Listing admin tokens requires a valid InfluxDB [admin token](/influxdb3/enterprise/admin/tokens/)or a token with read access to the `_internal` system database > . > For more information, see how to [provide your token](/influxdb3/enterprise/admin/tokens/#provide-your-token). ## List all tokens The `influxdb3 show tokens` CLI command lists all admin and resource tokens in your InfluxDB 3 instance. ```bash influxdb3 show tokens ``` ## Query token metadata To filter tokens and retrieve specific details using SQL, query the `system.tokens` table in the`_internal` system database–for example: ### Filter for admin tokens [CLI](#cli)[HTTP API](#http-api) ```bash influxdb3 query \ --database _internal \ --format csv \ "SELECT name, permissions FROM system.tokens WHERE permissions = '*:*:*'" ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT name, permissions FROM system.tokens WHERE permissions = '*:*:*'" \ --data-urlencode "format=csv" \ --header 'Accept: text/csv' \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### Filter by date [CLI](#cli-2)[HTTP API](#http-api-2) ```bash influxdb3 query \ --db _internal \ "SELECT name, permissions FROM system.tokens WHERE created_at > '2025-01-01 00:00:00'" ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT name, permissions FROM system.tokens WHERE created_at > '2025-01-01 00:00:00'" \ --header "Accept: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" ``` ## Output formats Use the format option to specify the output format for commands. InfluxDB 3 Enterprise commands used in this guide support the following output formats: * `json` *(default for HTTP API)* * `pretty` *(default for CLI)* * `jsonl` * `csv` * `parquet` *([output to a file](#output-to-a-parquet-file))* [CLI](#cli-3)[HTTP API](#http-api-3) ```bash influxdb3 show tokens \ --format jsonl ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=csv" \ --header 'Accept: text/csv' \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### Output to a Parquet file [Parquet](https://parquet.apache.org/) is a binary format. To output to a Parquet file using the CLI, include the `--output` option with a destination path for the file. To output a Parquet file using the HTTP API, your client must be able to handle binary data–for example, using cURL’s `--output` option. [CLI](#cli-4)[HTTP API](#http-api-4) ```bash influxdb3 show tokens \ --format parquet \ --output /PATH/TO/FILE.parquet ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=parquet" \ --header "Accept: application/parquet" \ --header "Authorization: Bearer AUTH_TOKEN" \ --output /PATH/TO/FILE.parquet ``` Replace `/PATH/TO/FILE.parquet`with the path to the file where you want to save the Parquet data. ## Filter the token list Use command-line tools such as `grep` or `jq` to filter the output of the`influxdb3 show tokens` command or the HTTP API response–for example: [CLI](#cli-5)[HTTP API](#http-api-5) ```bash influxdb3 show tokens --format pretty | grep _admin ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=pretty" \ --header "Accept: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" | grep _admin ``` [CLI](#cli-6)[HTTP API](#http-api-6) ```bash influxdb3 show tokens --format json | jq '.[] | {name: .name, permissions: .permissions}' ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT name, created_at FROM system.tokens WHERE permissions = '*:*:*' AND created_at > '2025-01-01 00:00:00'" \ --data-urlencode "format=json" \ --header "Accept: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" | jq '.[] | {name: .name, created_at: .created_at}' ``` #### Related * [Manage tokens with InfluxDB 3 Explorer](/influxdb3/explorer/manage-tokens/) --- # Use a preconfigured admin token Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/admin/preconfigured/ Start InfluxDB 3 Enterprise with a preconfigured “offline” admin token file. If no admin tokens already exist, InfluxDB automatically creates an admin token using the provided admin token file. Offline tokens are designed to help with automated deployments. * [Generate an offline admin token file](#generate-an-offline-admin-token-file) * [Offline admin token file schema](#offline-admin-token-file-schema) * [Start InfluxDB with the preconfigured admin token](#start-influxdb-with-the-preconfigured-admin-token) * [Use Docker Compose with preconfigured admin tokens](#use-docker-compose-with-preconfigured-admin-tokens) * [Create an admin token file](#create-an-admin-token-file) * [Configure Docker Compose with secrets](#configure-docker-compose-with-secrets) * [CI/CD setup](#cicd-setup) ## Generate an offline admin token file Use the `influxdb3 create token --admin` command to generate an offline admin token file. Include the following options: \* Required * `--name`: The name of the admin token *(default is `_admin`)**(replace `TOKEN_NAME`)* * `--expiry`: Duration for the token to remain valid, in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)format (for example, `10d` for 10 days or `1y` for 1 year).*(replace `DURATION`)* * \* `--offline` * \* `--output-file`: File path to use for the generated token file*(replace `path/to/tokens.json`)* ```bash influxdb3 create token --admin \ --name TOKEN_NAME \ --expiry DURATION \ --offline \ --output-file path/to/admin-token.json ``` > [!Note] > #### You can write or generate your own admin token file > > The `influxdb3 create token --admin --offline` command makes generating > offline admin token files easy, but it is not required. > You can also write or generate your own admin token files using the[required JSON schema](#offline-admin-token-file-schema). > > ##### Token string security standards > > If writing or generating your own admin token file, ensure that the token > string is sufficiently secure. We recommend the following: > > * Use a cryptographically secure pseudorandom number generator. > * Ensure sufficient length and entropy. Generate and base64-encode a random > string of at least 16 bytes (128 bits). > * Prepend the generated string with `apiv3_` for InfluxDB compatibility. > [!Important] > #### Token file permissions > > Token file permissions should be restricted `0600` to protect the token. ### Offline admin token file schema An offline admin token file is a JSON-formatted file that contains a single object with the following fields: * **token**: The raw token string (must begin with `apiv3_`) * **name**: The token name (default is `_admin`) * **description**: *(Optional)* A description of the token * **expiry\_millis**: *(Optional)* Token expiration time as a millisecond Unix timestamp ```json { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "_admin", "description": "Admin token for InfluxDB 3", "expiry_millis": 1756400061529 } ``` ## Start InfluxDB with the preconfigured admin token When starting InfluxDB 3 Enterprise, include the `--admin-token-file` option with the`influxdb3 serve` command or set the `INFLUXDB3_ADMIN_TOKEN_FILE` environment variable to provide the preconfigured offline admin token file: [CLI option](#cli-option)[Environment variable](#environment-variable) ```bash influxdb3 serve \ # ... \ --admin-token-file path/to/admin-token.json ``` ```bash INFLUXDB3_ADMIN_TOKEN_FILE=path/to/admin-token.json influxdb3 serve \ # ... \ ``` When the server starts, you can use the preconfigured admin token to interact with your InfluxDB 3 Enterprise cluster or instance. ## Use Docker Compose with preconfigured admin tokens For containerized deployments, you can use Docker Compose with Docker secrets to securely manage your preconfigured admin token. ### Create an admin token file Create a JSON file with your admin token using the[offline admin token file schema](#offline-admin-token-file-schema): ```json { "token": "apiv3_your_token_here", "name": "admin", "description": "Admin token for automated deployment" } ``` For security, restrict file permissions: ```bash chmod 600 path/to/admin-token.json ``` ### Configure Docker Compose with secrets Use Docker secrets to securely provide the admin token file to your container: ```yaml # compose.yaml services: influxdb3-enterprise: image: influxdb:3-enterprise ports: - 8181:8181 command: - influxdb3 - serve - --node-id=node0 - --cluster-id=cluster0 - --object-store=file - --data-dir=/var/lib/influxdb3/data - --admin-token-file=/run/secrets/admin-token environment: - INFLUXDB3_LICENSE_EMAIL=your-email@example.com secrets: - admin-token volumes: - type: bind source: ~/.influxdb3/data target: /var/lib/influxdb3/data secrets: admin-token: file: path/to/admin-token.json ``` Start the service: ```bash docker compose up -d ``` > [!Important] > #### Docker secrets security benefits > > Docker secrets provide better security than bind mounts for sensitive data: > > * Secrets are stored encrypted in memory > * Not visible in `docker inspect` output > * Not exposed in environment variables or logs > * Follow Docker and Kubernetes security best practices ### CI/CD setup For CI/CD pipelines and automated environments, create the admin token file from environment variables: ```bash # Create token file from CI/CD environment variable echo "{\"token\": \"$INFLUXDB3_ADMIN_TOKEN\", \"name\": \"admin\", \"description\": \"CI/CD admin token\"}" > admin-token.json chmod 600 admin-token.json ``` Then use the file in your Docker Compose configuration as shown above. #### Related * [Use a preconfigured permission (resource) tokens](/influxdb3/enterprise/admin/tokens/resource/preconfigured/) * [Configuration options \> admin-token-file](/influxdb3/enterprise/reference/config-options/#admin-token-file) * [influxdb3 create token --admin](/influxdb3/enterprise/reference/cli/influxdb3/create/token/admin/) --- # Regenerate an operator admin token Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/admin/regenerate/ Use the `influxdb3` CLI or the HTTP API to regenerate the operator (`_admin`) token for your InfluxDB 3 Enterprise instance. Regenerate a token to rotate it as part of your security practices or if you suspect the token has been compromised. Regenerating the operator token deactivates the previous token, stores the SHA512 hash and metadata of the new token, and returns the new token string. ## Prerequisite To regenerate an operator token, you need the current token string. ## Use the CLI or HTTP API to regenerate the operator token > [!Important] > #### Regenerating the operator token > > Regenerating the operator token invalidates the previous token. > Make sure to update any applications or scripts that use the operator token. To regenerate the operator token, use the [`influxdb3 serve create token` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/) (CLI) or the[`/api/v3/configure/token/admin/regenerate` endpoint](/influxdb3/enterprise/api/v3/configure/token/admin/regenerate) (HTTP API): [CLI](#cli)[HTTP API](#http-api) Use the `--regenerate` flag with the[`influxdb3 create token --admin`](/influxdb3/enterprise/reference/cli/influxdb3/create/token/) subcommand–for example: ```bash influxdb3 create token --admin \ --regenerate \ --token OPERATOR_TOKEN ``` In your command, replace `OPERATOR_TOKEN`with the current operator (`_admin`) token string. The output contains the new token string and InfluxDB deactivates the previous token string. Use the following HTTP API endpoint: ``` POST /api/v3/configure/token/admin/regenerate ``` In your request, send an `Authorization` header with your current operator token string –for example: ```bash curl -X POST "http://localhost:8181/api/v3/configure/token/admin/regenerate" \ --header "Authorization: Bearer OPERATOR_TOKEN" \ --header "Accept: application/json" ``` In your command, replace `OPERATOR_TOKEN` with the current token string. The response body contains the new operator token string in plain text, and InfluxDB deactivates the previous token string. To use the token as the default for later commands, and to persist the token across sessions, assign the token string to the `INFLUXDB3_AUTH_TOKEN` environment variable. ## Lost admin token recovery If you’ve lost your admin token and cannot regenerate it using the standard method, you can use the admin token recovery server: 1. Start InfluxDB 3 Enterprise with the `--admin-token-recovery-http-bind` option: ``` influxdb3 serve --admin-token-recovery-http-bind ``` 2. In a separate terminal, regenerate the admin token using the recovery endpoint: ``` influxdb3 create token --admin --regenerate --host http://127.0.0.1:8182 ``` 3. The recovery server automatically shuts down after successful token regeneration. > [!Warning] > The recovery server provides unauthenticated access to regenerate admin tokens. Only use this option when necessary and ensure the recovery endpoint (by default `127.0.0.1:8182`) is only accessible from trusted networks. ## Important considerations * Regenerating the operator token invalidates the previous token. * If you lose the operator token, use the recovery server method described above. * `--regenerate` only works for the operator token. You can’t use the `--regenerate` flag with the `influxdb3 create token --admin` command to regenerate a named admin token. * Ensure that you update any applications or scripts that use the operator token with the new token string. * Always store your operator token securely and consider implementing proper secret management practices. #### Related * [Manage tokens with InfluxDB 3 Explorer](/influxdb3/explorer/manage-tokens/) --- # Manage resource tokens Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/resource/ Manage resource tokens in your InfluxDB 3 Enterprise instance. Resource tokens provide scoped access to specific resources: * **Database tokens**: provide access to specific databases for actions like writing and querying data * **System tokens**: provide access to system-level resources, such as API endpoints for server runtime statistics and health. Resource tokens are user-defined and available only in InfluxDB 3 Enterprise. ### [Create a resource token](/influxdb3/enterprise/admin/tokens/resource/create/) Use the [`influxdb3 create token --permission` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/) or the [HTTP API](/influxdb3/enterprise/api/v3/) to create fine-grained permissions tokens that grant access to resources such as databases and system information. Database tokens allow for reading and writing data in your InfluxDB 3 Enterprise instance. System tokens allow for reading system information and metrics for your server. ##### CLI ```bash influxdb3 create token \ --permission "db:DATABASE1,DATABASE2:read,write" \ --name "Read-write on DATABASE1, DATABASE2" \ --token ADMIN_TOKEN \ --expiry 1y ``` ##### HTTP API ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Read-write for DATABASE1, DATABASE2", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE1","DATABASE2"], "actions": ["read","write"] }], "expiry_secs": 300000 }' ``` ### [List resource tokens](/influxdb3/enterprise/admin/tokens/resource/list/) Use the `influxdb3` CLI or the HTTP API to list resource tokens with fine-grained access permissions in your InfluxDB 3 Enterprise instance. Use the `influxdb3 show tokens` command to list all tokens or use SQL to query token metadata directly from the `system.tokens` table. ##### CLI ```bash influxdb3 show tokens ``` ##### HTTP API ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens WHERE permissions NOT LIKE '\*%'" \ --header 'Accept: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### [Use a preconfigured permission (resource) tokens](/influxdb3/enterprise/admin/tokens/resource/preconfigured/) Start InfluxDB 3 Enterprise with a preconfigured “offline” permission (resource) tokens file. If no tokens already exist, InfluxDB automatically creates resource tokens specified in the provided permissions (resource) tokens file. [tokens](/influxdb3/enterprise/tags/tokens/) --- # Create a resource token Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/resource/create/ Use the [`influxdb3 create token --permission` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/)or the [`/api/v3/enterprise/configure/token` HTTP API endpoint](/influxdb3/enterprise/api/auth-token/#operation/PostCreateResourceToken)to create fine-grained permissions tokens that grant access to resources such as databases and system information. Database tokens allow for reading and writing data in your InfluxDB 3 Enterprise instance. System tokens allow for reading system information and metrics for your server. After you[create an *admin token*](/influxdb3/enterprise/admin/tokens/admin/create/), you can use the token string to authenticate `influxdb3` commands and HTTP API requests for managing database and system tokens. *The HTTP API examples in this guide use [cURL](https://curl.se/) to send an API request, but you can use any HTTP client.* > [!Note] > #### Store secure tokens in a secret store > > Token strings are returned *only* on token creation. > We recommend storing database tokens in a **secure secret store**. > If you lose a resource token string, revoke the token and create a new one. ## Create a database token [CLI](#cli)[HTTP API](#http-api) Use the [`influxdb3 create token --permission` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/)to create a database token with fine-grained permissions for reading and writing data in your InfluxDB 3 Enterprise instance. In your terminal, enter `influxdb3 create token` and provide the following: * `--permission`: Token permissions (read, write, describe) in the `RESOURCE_TYPE:RESOURCE_NAMES:ACTIONS` format, for example: ``` db:DATABASE1,DATABASE2:read,write ``` * `db:`: The `db` resource type, which specifies the token is for a database * `DATABASE1,DATABASE2`: A comma-separated list of database names to grant permissions to. The resource names part supports the `*` wildcard, which grants read or write permissions to all databases. * `read,write`: A comma-separated list of permissions to grant to the token. * `--name`: A unique name for the token * *Options*, for example: * `--expiry`: The token expiration time as a duration. If an expiration isn’t set, the token does not expire until revoked. ```bash influxdb3 create token \ --permission "db:DATABASE1,DATABASE2:read,write" \ --name "Read-write on DATABASE1, DATABASE2" \ --expiry 1y ``` Replace the following: * `DATABASE1`, `DATABASE2`: your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `1y`: the token expiration time as a duration The output is the token string in plain text. Send a request to the following InfluxDB 3 Enterprise endpoint: ``` POST http://localhost:8181/api/v3/enterprise/configure/token ``` Provide the following request headers: * `Accept: application/json` to ensure the response body is JSON content * `Content-Type: application/json` to indicate the request body is JSON content * `Authorization: Bearer` and the [admin token](/influxdb3/enterprise/admin/tokens/admin/)for your instance to authorize the request In the request body, provide the following parameters: * `token_name`: a description of the token, unique within the instance * `permissions`: an array of permission objects, each with: * `resource_type`: the resource type for the token, which is always `db` * `resource_names`: an array of database names to grant permissions to. Supports the `*` wildcard, which grants the actions on all databases. * `actions`: an array of actions (`"read"`, `"write"`, `"create"`, `"describe"`, `"delete"`) * `expiry_secs`: Specify the token expiration time in seconds. The following example shows how to use the HTTP API to create a database token: ``` curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --data '{ "token_name": "Read-write for DATABASE1, DATABASE2", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE1","DATABASE2"], "actions": ["read","write"] }], "expiry_secs": 300000 }' ``` Replace the following in your request: * `DATABASE1`, `DATABASE2`: your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `300000`: the token expiration time in seconds The response body contains token details, including the `token` field with the token string in plain text. ### Examples * [Create a token with read and write access to a database](#create-a-token-with-read-and-write-access-to-a-database) * [Create a token with read and write access to all databases](#create-a-token-with-read-and-write-access-to-all-databases) * [Create a token with read-only access to a database](#create-a-token-with-read-only-access-to-a-database) * [Create a token with read-only access to multiple databases](#create-a-token-with-read-only-access-to-multiple-databases) * [Create a token that expires in seven days](#create-a-token-that-expires-in-seven-days) * [Create a token for bulk import](#create-a-token-for-bulk-import) In the examples below, replace the following: * `DATABASE_NAME`: your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE2_NAME`: your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `AUTH_TOKEN`: the [token](/influxdb3/enterprise/admin/tokens/admin)for your InfluxDB 3 Enterprise instance #### Create a token with read and write access to a database [CLI](#cli-2)[HTTP API](#http-api-2) ```bash influxdb3 create token \ --permission "db:DATABASE_NAME:read,write" \ --name "Read/write token for DATABASE_NAME" ``` ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Read/write token for DATABASE_NAME", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE_NAME"], "actions": ["read","write"] }] }' ``` #### Create a token with read and write access to all databases [CLI](#cli-3)[HTTP API](#http-api-3) ```bash influxdb3 create token \ --permission "db:*:read,write" \ --name "Read/write token for all databases" ``` ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Read/write token for all databases", "permissions": [{ "resource_type": "db", "resource_names": ["*"], "actions": ["read","write"] }] }' ``` #### Create a token with read-only access to a database [CLI](#cli-4)[HTTP API](#http-api-4) ```bash influxdb3 create token \ --permission "db:DATABASE_NAME:read" \ --name "Read-only token for DATABASE_NAME" ``` ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Read-only token for DATABASE_NAME", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE_NAME"], "actions": ["read"] }] }' ``` #### Create a token with read-only access to multiple databases [CLI](#cli-5)[HTTP API](#http-api-5) ```bash influxdb3 create token \ --permission "db:DATABASE_NAME,DATABASE2_NAME:read" \ --name "Read-only token for DATABASE_NAME, DATABASE2_NAME" ``` ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Read-only token for DATABASE_NAME, DATABASE2_NAME", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE_NAME","DATABASE2_NAME"], "actions": ["read"] }] }' ``` #### Create a token that expires in seven days [CLI](#cli-6)[HTTP API](#http-api-6) ```bash influxdb3 create token \ --permission "db:DATABASE_NAME:read,write" \ --name "Read/write token for DATABASE_NAME with 7d expiration" \ --expiry 7d ``` ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Read/write token for DATABASE_NAME with 7d expiration", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE_NAME"], "actions": ["read","write"] }], "expiry_secs": 604800 }' ``` #### Create a token for bulk import [Bulk import](/influxdb3/enterprise/admin/import-data/) needs the `write`action to create an import, and the `describe` action to list import jobs. Grant both actions on the target database to use the same token for both: [CLI](#cli-7)[HTTP API](#http-api-7) ```bash influxdb3 create token \ --permission "db:DATABASE_NAME:describe,write" \ --name "Bulk import token for DATABASE_NAME" ``` ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "Bulk import token for DATABASE_NAME", "permissions": [{ "resource_type": "db", "resource_names": ["DATABASE_NAME"], "actions": ["describe","write"] }] }' ``` ## Create a system token System tokens have the `system` resource type and allow for read-only access to system information and metrics from your server. You can create system tokens for the following system resources: * `health`: system health information from the `/health` HTTP API endpoint * `metrics`: system metrics information from the `/metrics` HTTP API endpoint * `ping`: system ping information from the `/ping` HTTP API endpoint > [!Note] > #### Database tokens can’t call /ping or /health > > A database token with the `read` action, for example, `db:DATABASE_NAME:read`, > gets an HTTP `403` response from the `/ping` and `/health` endpoints. > A health check that reuses an application’s database token reports the > server as unavailable. [CLI](#cli-8)[HTTP API](#http-api-8) Use the [`influxdb3 create token` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/)to create a system token with permissions for reading system information from your InfluxDB 3 Enterprise instance. In your terminal, run the `influxdb3 create token --permission` command and provide the following: * `--name`: A unique name for the token * *Options*, for example: * `--expiry`: The token expiration time as a duration. If an expiration isn’t set, the token does not expire until revoked. * Token permissions in the `RESOURCE_TYPE:RESOURCE_NAMES:ACTIONS` format, for example: ``` system:health:read ``` * `system:`: The `system` resource type, which specifies the token is for system information. * `health`: The specific system resource to grant permissions to. * `read`: The permission to grant to the token (system tokens are always read-only). ```bash influxdb3 create token \ --permission "system:health:read" \ --name "System health token" \ --expiry 1y ``` Replace the following: * `1y`: the token expiration time as a duration. The output is the token string in plain text. Send a request to the following InfluxDB 3 Enterprise endpoint: ``` POST http://localhost:8181/api/v3/enterprise/configure/token ``` Provide the following request headers: * `Accept: application/json` to ensure the response body is JSON content * `Content-Type: application/json` to indicate the request body is JSON content * `Authorization: Bearer` and the [admin token](/influxdb3/enterprise/admin/tokens/admin/)for your instance to authorize the request In the request body, provide the following parameters: * `token_name`: a description of the token, unique within the instance * `resource_type`: the resource type for the token, which is `system` for system tokens * `resource_names`: an array of system resource names to grant permissions to * The `resource_names` field supports the `*` wildcard, which grants read or write permissions to all system information resources. * `permissions`: an array of token permission actions (only `"read"` for system tokens) * `expiry_secs`: Specify the token expiration time in seconds. The following example shows how to use the HTTP API to create a system token: ```bash curl \ "http://localhost:8181/api/v3/enterprise/configure/token" \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "token_name": "System health token", "permissions": [{ "resource_type": "system", "resource_names": ["health"], "actions": ["read"] }], "expiry_secs": 300000 }' ``` Replace the following in your request: * `300000`: the token expiration time in seconds. The response body contains token details, including the `token` field with the token string in plain text. ## Output format The `influxdb3 create token` command supports the `--format json` option. By default, the command outputs the token string. For easier programmatic access to the command output, include `--format json`with your command to format the output as JSON. The `/api/v3/configure/token` endpoint outputs JSON format in the response body. --- # List resource tokens Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/resource/list/ Use the `influxdb3` CLI or the HTTP API to list resource tokens with fine-grained access permissions in your InfluxDB 3 Enterprise instance. Use the `influxdb3 show tokens` command to list all tokens or use SQL to query token metadata directly from the `system.tokens` table. Resource tokens have fine-grained permissions for InfluxDB 3 resources, such as databases and system tables. The permissions are in the format `RESOURCE_TYPE:RESOURCE_NAMES:ACTIONS`. * `RESOURCE_TYPE` is the type of resource (for example, `db`, `system`) * `RESOURCE_NAMES` is the specific resource name (for example, a list of database names or `*` for all) * `ACTIONS` is the permission level (`read`, `write`, or `*` for all) Database resource tokens have permissions in the format `db:database_names:actions`. System resource tokens have permissions in the format `system:system_endpoint_names:read`. \_In the following examples, replace `AUTH_TOKEN`with your InfluxDB [token](/influxdb3/enterprise/admin/tokens/) that has read permission on the `_internal` system database .\_ ## List all tokens The `influxdb3 show tokens` CLI command lists all admin and resource tokens in your InfluxDB 3 instance. ```bash influxdb3 show tokens ``` ## Query token metadata To filter tokens and retrieve specific details using SQL, query the `system.tokens` table in the`_internal` system database–for example: [CLI](#cli)[HTTP API](#http-api) ```bash influxdb3 query \ --db _internal \ "SELECT name, permissions FROM system.tokens WHERE permissions NOT LIKE '\*%'" ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens WHERE permissions NOT LIKE '\*%'" \ --header 'Accept: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" ``` ## Format the output Use the format option to specify the output format for the `influxdb3 show tokens` command or the HTTP API response. InfluxDB 3 Enterprise supports the following output formats: * `pretty` *(default for CLI)* * `json` *(default for HTTP API)* * `jsonl` * `csv` * `parquet` *([output to a file](#output-to-a-parquet-file))* [CLI](#cli-2)[HTTP API](#http-api-2) ```bash influxdb3 show tokens \ --format csv ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=csv" \ --header 'Accept: text/csv' \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### Output to a Parquet file [Parquet](https://parquet.apache.org/) is a binary format. To output to a Parquet file using the CLI, include the `--output` option with a destination path for the file. To output a Parquet file using the HTTP API, your client must be able to handle binary data–for example, using cURL’s `--output` option. [CLI](#cli-3)[HTTP API](#http-api-3) ```bash influxdb3 show tokens \ --format parquet \ --output /PATH/TO/FILE.parquet ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=parquet" \ --header "Accept: application/parquet" \ --header "Authorization: Bearer AUTH_TOKEN" \ --output /PATH/TO/FILE.parquet ``` Replace `/PATH/TO/FILE.parquet`with the path to the file where you want to save the Parquet data. ## Filter the output for resource tokens You can use tools such as `grep` or `jq` to filter the token list–for example: Use `grep` to filter for resource (non-admin) tokens: [CLI](#cli-4)[HTTP API](#http-api-4) ```bash influxdb3 show tokens --format pretty | grep -v '*:*:*' ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=jsonl" \ --header 'Accept: application/json' \ --header "Authorization: Bearer AUTH_TOKEN" | grep -v "*:*:*" ``` Use `jq` to filter for system tokens and display their permissions: [CLI](#cli-5)[HTTP API](#http-api-5) ```bash influxdb3 show tokens --format json | jq '.[] | select(.permissions | startswith("system:")) | {name: .name, permissions: .permissions}' ``` ```bash curl -G \ "http://localhost:8181/api/v3/query_sql" \ --data-urlencode "db=_internal" \ --data-urlencode "q=SELECT * FROM system.tokens" \ --data-urlencode "format=json" \ --header "Accept: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" | jq '.[] | select(.permissions | startswith("system:")) | {name: .name, permissions: .permissions}' ``` #### Related * [InfluxDB HTTP API](/influxdb3/enterprise/reference/api/) --- # Use a preconfigured permission (resource) tokens Source: https://docs.influxdata.com/influxdb3/enterprise/admin/tokens/resource/preconfigured/ Start InfluxDB 3 Enterprise with a preconfigured “offline” permission (resource) tokens file. If no tokens already exist, InfluxDB automatically creates resource tokens specified in the provided permission (resource) tokens file. * [Generate an offline permissions (resource) tokens file](#generate-an-offline-permissions-resource-tokens-file) * [Offline permission tokens file schema](#offline-permission-tokens-file-schema) * [Start InfluxDB with the preconfigured permission tokens](#start-influxdb-with-the-preconfigured-permission-tokens) * [Use Docker Compose with preconfigured resource tokens](#use-docker-compose-with-preconfigured-resource-tokens) * [Create a permission tokens file](#create-a-permission-tokens-file) * [Configure Docker Compose with secrets](#configure-docker-compose-with-secrets) * [CI/CD setup](#cicd-setup) ## Generate an offline permissions (resource) tokens file Use the `influxdb3 create token` command to generate an offline permission (resource) tokens file. You can also specify corresponding databases to create when starting InfluxDB. Include the following options: \* Required * \* `--name`: The name of the admin token*(replace `TOKEN_NAME`)* * \* `--permissions`: The [token permissions](/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/#permission-format)*(replace `TOKEN_PERMISSIONS`)* * `--expiry`: Duration for the token to remain valid, in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)format–for example `10d` for 10 days or `1y` for 1 year*(replace `DURATION`)* * \* `--offline` * `--create-databases`: Comma separated list of database names to create when starting the server*(replace `DATABASE_LIST`)* * \* `--output-file`: File path to use for the generated token file*(replace `path/to/tokens.json`)* ```bash influxdb3 create token \ --name TOKEN_NAME \ --permission "TOKEN_PERMISSIONS" \ --expiry DURATION \ --offline \ --create-databases DATABASE_LIST \ --output-file path/to/tokens.json ``` > [!Note] > #### Add multiple tokens to a permission tokens file > > If you write a new offline permission token to an existing permission token file, > the command appends the new token to the existing output file. > > #### You can write or generate your own permission tokens file > > The `influxdb3 create token --offline` command makes generating an > offline permission tokens file easy, but it is not required. > You can write or generate your own permission tokens file using the[required JSON schema](#offline-permission-tokens-file-schema). > > ##### Token string security standards > > If writing or generating your own permission tokens file, ensure that token > strings are sufficiently secure. We recommend the following: > > * Use a cryptographically secure pseudorandom number generator. > * Ensure sufficient length and entropy. Generate and base64-encode a random > string of at least 16 bytes (128 bits). > * Prepend the generated string with `apiv3_` for InfluxDB compatibility. > [!Important] > #### Token file permissions > > Token file permissions should be restricted `0600` to protect the tokens. ### Offline permission tokens file schema An offline permission tokens file is a JSON-formatted file that contains a single object with the following fields: * **create\_databases**: *(Optional)**Array of database names to create when starting the server* * **tokens**: *Array of token objects* * **token**: The raw token string (must begin with `apiv3_`) * **name**: A unique token name * **expiry\_millis**: *(Optional)* Token expiration time as a millisecond Unix timestamp * **permissions**: Array of [token permission](/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/#permission-format) strings. ```json { "create_databases": [ "db1", "db2", "db3", "db4" ], "tokens": [ { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "token-1", "expiry_millis": 1756400061529, "permissions": [ "db:db1,db2:read,write", "db:db3:read" ] }, { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "token-2", "expiry_millis": 1756400061529, "permissions": [ "db:db4:read,write" ] } ] } ``` ## Start InfluxDB with the preconfigured permission tokens When starting InfluxDB 3 Enterprise, include the `--permission-tokens-file`option with the `influxdb3 serve` command or set the`INFLUXDB3_PERMISSION_TOKENS_FILE` environment variable to provide the preconfigured offline permission tokens file: [CLI option](#cli-option)[Environment variable](#environment-variable) ```bash influxdb3 serve \ # ... \ --permission-tokens-file path/to/admin-token.json ``` ```bash INFLUXDB3_PERMISSION_TOKENS_FILE=path/to/admin-token.json influxdb3 serve \ # ... \ ``` When the server starts, you can use the preconfigured permission (resource) tokens to write data to and query data from with your InfluxDB 3 Enterprise instance or cluster. ## Use Docker Compose with preconfigured resource tokens For containerized deployments, you can use Docker Compose with Docker secrets to securely manage your preconfigured resource tokens. ### Create a permission tokens file Create a JSON file with your resource tokens using the[offline permission tokens file schema](#offline-permission-tokens-file-schema): ```json { "create_databases": [ "sensors", "metrics" ], "tokens": [ { "token": "apiv3_your_token_here", "name": "app-writer", "permissions": [ "db:sensors,metrics:read,write" ] }, { "token": "apiv3_another_token_here", "name": "dashboard-reader", "permissions": [ "db:sensors,metrics:read" ] } ] } ``` For security, restrict file permissions: ```bash chmod 600 path/to/permission-tokens.json ``` ### Configure Docker Compose with secrets Use Docker secrets to securely provide the permission tokens file to your container: ```yaml # compose.yaml services: influxdb3-enterprise: image: influxdb:3-enterprise ports: - 8181:8181 command: - influxdb3 - serve - --node-id=node0 - --cluster-id=cluster0 - --object-store=file - --data-dir=/var/lib/influxdb3/data - --permission-tokens-file=/run/secrets/permission-tokens environment: - INFLUXDB3_LICENSE_EMAIL=your-email@example.com secrets: - permission-tokens volumes: - type: bind source: ~/.influxdb3/data target: /var/lib/influxdb3/data secrets: permission-tokens: file: path/to/permission-tokens.json ``` Start the service: ```bash docker compose up -d ``` > [!Important] > #### Docker secrets security benefits > > Docker secrets provide better security than bind mounts for sensitive data: > > * Secrets are stored encrypted in memory > * Not visible in `docker inspect` output > * Not exposed in environment variables or logs > * Follow Docker and Kubernetes security best practices ### CI/CD setup For CI/CD pipelines and automated environments, create the permission tokens file from environment variables: ```bash # Create permission tokens file from CI/CD environment variables cat > permission-tokens.json << EOF { "create_databases": ["$INFLUXDB3_DATABASE"], "tokens": [ { "token": "$INFLUXDB3_RESOURCE_TOKEN", "name": "app-token", "permissions": ["db:$INFLUXDB3_DATABASE:read,write"] } ] } EOF chmod 600 permission-tokens.json ``` Then use the file in your Docker Compose configuration as shown above. #### Related * [Configuration options \> permission-tokens-file](/influxdb3/enterprise/reference/config-options/#permission-tokens-file) * [influxdb3 create token --permission](/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/) --- # Upgrade from InfluxDB 3 Core Source: https://docs.influxdata.com/influxdb3/enterprise/admin/upgrade-from-core/ Upgrade from InfluxDB 3 Core to InfluxDB 3 Enterprise. Your existing data and plugins are compatible with Enterprise–no data migration is required. * [Before you begin](#before-you-begin) * [Upgrade to Enterprise](#upgrade-to-enterprise) > [!Warning] > #### Downgrading is not supported > > After upgrading to InfluxDB 3 Enterprise, you cannot downgrade back to > InfluxDB 3 Core. Enterprise makes catalog changes that are incompatible with Core. > To revert to Core, you must restore from a backup taken before the upgrade. > > *Before upgrading, back up your data directory.* ## Before you begin 1. **Back up your data**: Create a backup of your InfluxDB 3 Core data directory before upgrading. For more information, see [Back up and restore data](/influxdb3/core/admin/backup-restore/). 2. **Note your current configuration**: Record your Core startup options, including `--data-dir`, `--object-store`, `--plugin-dir`, and any other configuration. You’ll use the same data directory with Enterprise. For a complete list of configuration options, see the[`influxdb3 serve` CLI reference](/influxdb3/enterprise/reference/cli/influxdb3/serve/). If you use a cloud object store (S3, Azure, or Google Cloud Storage), see[Configure object storage](/influxdb3/enterprise/admin/object-storage/). 3. **Choose a license type**: Decide which InfluxDB 3 Enterprise license you need: * **Trial**: 30-day full-featured trial * **At-Home**: Free for hobbyist use (2 CPU limit, single-node only) * **Commercial**: For production and commercial use For more information about licenses, see [Manage your license](/influxdb3/enterprise/admin/license/). ## Upgrade to Enterprise Choose your installation method: [Quick install (Linux/macOS)](#quick-install-linuxmacos)[Docker](#docker-2)[DEB/RPM (systemd)](#debrpm-systemd) ### Stop InfluxDB 3 Core Stop the running Core process: ```bash # If running in foreground, press Ctrl+C # If running in background, find and stop the process pkill -f "influxdb3 serve" ``` ### Install InfluxDB 3 Enterprise Run the quick install script for Enterprise: ```bash curl -O https://www.influxdata.com/d/install_influxdb3.sh \ && sh install_influxdb3.sh enterprise ``` ### Start InfluxDB 3 Enterprise Start Enterprise with your existing data directory. Enterprise requires a`--cluster-id` option that Core doesn’t use: ```bash influxdb3 serve \ --node-id NODE_ID \ --cluster-id CLUSTER_ID \ --object-store file \ --data-dir DATA_DIR \ --license-email EMAIL_ADDRESS ``` Replace the following: * `NODE_ID`: Your existing node identifier from Core * `CLUSTER_ID`: A new cluster identifier for Enterprise–for example, `cluster0` * `DATA_DIR`: The same data directory you used with Core * `EMAIL_ADDRESS`: Your email address for license activation When prompted, select your license type (`trial` or `home`), then verify your email address. ### Verify the upgrade After starting Enterprise, verify the upgrade was successful: ```bash # Check the version influxdb3 --version # Verify your license influxdb3 show license --host http://localhost:8181 ``` Query your existing data to confirm it’s accessible. ### Stop the Core container ```bash docker stop CORE_CONTAINER_NAME ``` Replace `CORE_CONTAINER_NAME` with the name of your InfluxDB 3 Core container. ### Pull the Enterprise image ```bash docker pull influxdb:3-enterprise ``` ### Start InfluxDB 3 Enterprise Start Enterprise using the same data volume. Enterprise requires a `--cluster-id`option and license configuration: ```bash docker run -d \ --name influxdb3-enterprise \ -p 8181:8181 \ -e INFLUXDB3_LICENSE_EMAIL=EMAIL_ADDRESS \ -v DATA_VOLUME:/var/lib/influxdb3/data \ -v PLUGIN_VOLUME:/var/lib/influxdb3/plugins \ influxdb:3-enterprise \ influxdb3 serve \ --node-id NODE_ID \ --cluster-id CLUSTER_ID \ --object-store file \ --data-dir /var/lib/influxdb3/data ``` Replace the following: * `EMAIL_ADDRESS`: Your email address for license activation * `DATA_VOLUME`: The same data volume you used with Core * `PLUGIN_VOLUME`: The same plugin volume you used with Core * `NODE_ID`: Your existing node identifier from Core * `CLUSTER_ID`: A new cluster identifier for Enterprise–for example, `cluster0` > [!Important] > #### License activation in Docker > > You must provide `INFLUXDB3_LICENSE_EMAIL` as an environment > variable because the interactive license prompt doesn’t work in containers. > After starting the container, check your email and click the verification link. [](#using-docker-compose) Using Docker Compose Update your `compose.yaml` to use the Enterprise image: ```yaml services: influxdb3: image: influxdb:3-enterprise container_name: influxdb3-enterprise ports: - 8181:8181 environment: - INFLUXDB3_LICENSE_EMAIL=${EMAIL_ADDRESS} command: - influxdb3 - serve - --node-id=NODE_ID - --cluster-id=CLUSTER_ID - --object-store=file - --data-dir=/var/lib/influxdb3/data volumes: # Use the same paths as Core - type: bind source: DATA_VOLUME target: /var/lib/influxdb3/data - type: bind source: PLUGIN_VOLUME target: /var/lib/influxdb3/plugins ``` Replace the following: * `EMAIL_ADDRESS`: Your email address for license activation (or a variable from your Compose `.env` file) * `NODE_ID`: Your existing node identifier from Core * `CLUSTER_ID`: A new cluster identifier for Enterprise–for example, `cluster0` * `DATA_VOLUME`: The same host data path you used with Core–for example, `~/.influxdb3/data` * `PLUGIN_VOLUME`: The same host plugin path you used with Core–for example, `~/.influxdb3/plugins` Then run: ```bash docker compose down docker compose up -d ``` ### Verify the upgrade After starting Enterprise, verify the upgrade was successful: [Docker](#docker)[Docker Compose](#docker-compose) ```bash # Check the version docker exec influxdb3-enterprise \ influxdb3 --version # Verify your license docker exec influxdb3-enterprise \ influxdb3 show license --host http://localhost:8181 ``` ```bash # Check the version docker compose exec influxdb3-enterprise \ influxdb3 --version # Verify your license docker compose exec influxdb3-enterprise \ influxdb3 show license --host http://localhost:8181 ``` Query your existing data to confirm it’s accessible. ### Stop InfluxDB 3 Core ```bash sudo systemctl stop influxdb3-core ``` ### Install InfluxDB 3 Enterprise First, remove the Core package (this doesn’t remove your data or configuration): [](#deb-based-systems-debian-ubuntu) DEB-based systems (Debian, Ubuntu) ```bash # Debian/Ubuntu sudo apt-get remove influxdb3-core # doesn't remove /var/lib/influxdb3 or /etc/influxdb3 sudo apt-get update && sudo apt-get install influxdb3-enterprise ``` [](#rpm-based-systems-rhel-centos-fedora) RPM-based systems (RHEL, CentOS, Fedora) ```bash # RHEL/CentOS/Fedora sudo yum remove influxdb3-core # doesn't remove /var/lib/influxdb3 or /etc/influxdb3 sudo yum install influxdb3-enterprise ``` ### Configure InfluxDB 3 Enterprise The Enterprise package installs a self-documenting configuration file at`/etc/influxdb3/influxdb3-enterprise.conf`. Merge your Core settings into this file rather than overwriting it. 1. View the settings you configured in Core: ``` # Show uncommented (active) settings from Core configuration sudo grep -E '^ *[^#]' /etc/influxdb3/influxdb3-core.conf ``` 2. Merge Core settings into the Enterprise configuration: ``` # Back up the original Enterprise configuration sudo cp /etc/influxdb3/influxdb3-enterprise.conf \ /etc/influxdb3/influxdb3-enterprise.conf.orig # Merge settings from Core into Enterprise config sudo grep -E '^ *[^#]' /etc/influxdb3/influxdb3-core.conf | while IFS= read -r line; do key=$(echo "$line" | sed 's/=.*/=/') if sudo grep -qE "^ *${key}" /etc/influxdb3/influxdb3-enterprise.conf; then # Update existing uncommented line sudo sed -i "s|^ *${key}.*|${line}|" /etc/influxdb3/influxdb3-enterprise.conf else # Replace commented line with active setting sudo sed -i "s|^#${key}.*|${line}|" /etc/influxdb3/influxdb3-enterprise.conf fi done ``` 3. Add your license email to the Enterprise configuration: ``` sudo sed -i 's|^#license-email.*|license-email = "EMAIL_ADDRESS"|' \ /etc/influxdb3/influxdb3-enterprise.conf ``` Replace `EMAIL_ADDRESS` with your email address for license activation. 4. Verify the merged configuration: ``` sudo grep -E '^ *[^#]' /etc/influxdb3/influxdb3-enterprise.conf ``` The output should include your Core settings plus the license email and cluster ID–for example: ``` node-id = "primary-node" object-store = "file" data-dir = "/var/lib/influxdb3/data" plugin-dir = "/var/lib/influxdb3/plugins" cluster-id = "primary-cluster" license-email = "you@example.com" ``` The DEB/RPM packages use `primary-node` and `primary-cluster` as defaults, matching the behavior of running `influxdb3 serve` without `--node-id` or `--cluster-id`. ### Start InfluxDB 3 Enterprise ```bash sudo systemctl start influxdb3-enterprise ``` Check the logs to verify the server started successfully: ```bash journalctl --unit influxdb3-enterprise -f ``` After starting, check your email and click the verification link to activate your license. ### Verify the upgrade After starting Enterprise, verify the upgrade was successful: ```bash # Check the version influxdb3 --version # Verify your license influxdb3 show license --host http://localhost:8181 ``` Query your existing data to confirm it’s accessible. #### Related * [Back up and restore data](/influxdb3/core/admin/backup-restore/) * [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/) * [Manage your InfluxDB 3 Enterprise license](/influxdb3/enterprise/admin/license/) * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) * [Configure object storage](/influxdb3/enterprise/admin/object-storage/) [upgrade](/influxdb3/enterprise/tags/upgrade/)[core](/influxdb3/enterprise/tags/core/)[migration](/influxdb3/enterprise/tags/migration/) --- # Upgrade InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/admin/upgrade/ Upgrade your InfluxDB 3 Enterprise version. * [Before you upgrade](#before-you-upgrade) * [Other changes to review before you upgrade to 3.12](#other-changes-to-review-before-you-upgrade-to-312) * [Upgrade an InfluxDB 3 instance](#upgrade-an-influxdb-3-instance) * [Upgrade a multi-node cluster](#upgrade-a-multi-node-cluster) * [Multi-node upgrade procedure](#multi-node-upgrade-procedure) * [Troubleshooting cluster upgrades](#troubleshooting-cluster-upgrades) ## Before you upgrade Before upgrading your InfluxDB 3 Enterprise cluster, review the [release notes](/influxdb3/enterprise/release-notes/) for compatibility requirements and then plan your upgrade strategy. > [!Important] > #### Upgrading to InfluxDB 3.10 is a one-way migration > > The first time you start InfluxDB 3.10, it automatically upgrades the on-disk > catalog format from v2 to v3. After migration, 3.9.x and older > binaries are unable to read the new catalog, and fail to start on the same > cluster data. > > Before upgrading, back up `{prefix}/catalogs/` and `{prefix}/_catalog_checkpoint`. > Restoring these objects is the only way to roll back to 3.9.x. > > If your cluster uses the upgraded storage engine (the default for new clusters, or after running the storage engine upgrade with `--upgrade-pacha-tree`), data written in the new `.pt` file format is also unreadable by 3.9.x. > [!Important] > #### Upgrading to InfluxDB 3.12 removes rollback to any 3.11.x release > > InfluxDB 3.12 adds a catalog record type that 3.11.x binaries can’t read. > Once every running node in the cluster is on 3.12 (which happens at first > startup on a single node), the catalog commits the new feature level. > From then on, a 3.11.x binary refuses to load the catalog and reports that > the node’s feature level is below the cluster’s committed level. > This applies to InfluxDB 3 Enterprise whether or not you use any of the > features that require it. > > Back up everything under `{prefix}/catalog/` (the catalog snapshot and > logs under `catalog/v3/`) before you upgrade. > Restoring these objects is the only way to roll back to 3.11.x. ### Other changes to review before you upgrade to 3.12 * **Query concurrency now has a finite default**: [`--max-concurrent-queries`](/influxdb3/enterprise/reference/config-options/#max-concurrent-queries) defaults to the larger of `50` and 4 times the node’s query parallelism, instead of being effectively unlimited. Queries submitted over the limit wait for a slot instead of running immediately. * **The WAL buffer limit is now enforced**: [`--wal-max-buffered-writes`](/influxdb3/enterprise/reference/config-options/#wal-max-buffered-writes) (default `100000`) previously had no effect. Once the WAL buffer fills, writes now return `429 Too Many Requests` until it drains. * **HTTP and gRPC request metrics are split by protocol**: `http_requests*` metrics now count only HTTP requests, and `grpc_requests*` metrics count only gRPC requests. Dashboards that summed the two families report lower values after you upgrade. The `path` and `method_path` labels are now route templates, such as `/api/v3/engine/:path`, instead of literal paths; update panels that filter on a specific path. Also review these InfluxDB 3 Enterprise changes: * **Data file cache is now a hard limit (upgraded storage engine)**: [`--file-cache-size`](/influxdb3/enterprise/reference/config-options/#file-cache-size) now also counts bytes held by running queries. A query that needs more than the remaining budget fails instead of the node using memory beyond the configured limit. * **Nodes without `query` mode refuse data queries (Parquet engine)**: A node that doesn’t run `query` mode now returns `405 Method Not Allowed` for data queries instead of serving them. System table queries still work. * **Processing Engine in a cluster**: A WAL trigger now runs on a process node for writes sent to any ingest node, and the default `--node-spec all` makes every node that runs the Processing Engine own the trigger. When you upgrade a cluster, move your plugins to a standalone process node, remove `--plugin-dir` from the other nodes, and pin each trigger to the process node with `--node-spec`. See [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). * **Username and password sessions must be renewed**: Access tokens issued to users who sign in with a username and password must now carry the cluster’s catalog UUID. Tokens issued before 3.12 are rejected: refresh the token or sign in again. API tokens aren’t affected. * **Orphaned file cleanup starts automatically (upgraded storage engine)**: The primary compactor begins finding and deleting unreferenced compacted files a few minutes after it first starts on 3.12, then repeats every 7 days. To only report candidates without deleting them, set `--compactor-sweep-mode dry-run`. To turn cleanup off, set `--compactor-sweep-interval off`. See [Orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/). * **Distributed compaction is available (beta, upgraded storage engine)**: Compaction jobs can now run on every compact node instead of only the node that holds the compactor lease. It’s off by default (`--compactor-dispatch-target local`), so upgrading alone doesn’t change where compaction runs. See [Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/). For the complete list of changes, see the [release notes](/influxdb3/enterprise/release-notes/). ### Verify your current version Before upgrading, verify the InfluxDB 3 Enterprise version running on each node. [CLI](#cli)[Docker](#docker-2) ```bash influxdb3 --version ``` ```bash docker exec CONTAINER_NAME influxdb3 --version ``` Replace the following: * `CONTAINER_NAME`: The name of your InfluxDB 3 Enterprise container The command returns version information similar to the following: ``` influxdb3 3.12.2 ``` > [!Tip] > ### Verify your InfluxDB version > > Before and after upgrading, verify the InfluxDB 3 Enterprise version running on your instance. ## Upgrade an InfluxDB 3 instance [Install script](#install-script)[systemctl](#systemctl-2)[Docker](#docker-3)[Docker Compose](#docker-compose-2)[Windows](#windows) ```bash curl -O https://www.influxdata.com/d/install_influxdb3.sh \ && sh install_influxdb3.sh enterprise ``` ```bash # 1. Download the new version curl -L https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_linux_amd64.tar.gz \ -o influxdb3-enterprise.tar.gz # 2. Extract the archive tar xvzf influxdb3-enterprise.tar.gz # 3. Stop the service sudo systemctl stop influxdb3-enterprise # 4. Install the new binary sudo cp influxdb3 /usr/local/bin/ # 5. Start the service sudo systemctl start influxdb3-enterprise ``` ```bash docker stop CONTAINER_NAME docker pull influxdb:enterprise docker start CONTAINER_NAME ``` Replace the following: * `CONTAINER_NAME`: The name of your InfluxDB 3 Enterprise container ```bash docker compose down docker compose pull docker compose up -d ``` ```powershell # Download the latest Windows binary Invoke-WebRequest ` -Uri "https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2-windows_amd64.zip" ` -OutFile "influxdb3-enterprise.zip" # Extract the binary Expand-Archive -Path influxdb3-enterprise.zip -DestinationPath . -Force # Stop the service, replace the binary, and start the service Stop-Service influxdb3 Copy-Item -Path "influxdb3.exe" -Destination "C:\Program Files\InfluxData\influxdb3\" -Force Start-Service influxdb3 ``` ## Upgrade a multi-node cluster Upgrade InfluxDB 3 Enterprise instances to newer versions using rolling upgrades to minimize downtime. When upgrading multi-node clusters, you need to understand how nodes on different versions interoperate and the recommended upgrade order for different node modes. * [Mixed-version clusters](#mixed-version-clusters) * [Multi-node upgrade procedure](#multi-node-upgrade-procedure) * [Troubleshooting cluster upgrades](#troubleshooting-cluster-upgrades) ### Mixed-version clusters During a rolling upgrade, nodes on the old and new versions share the catalog and the files in object storage. How the cluster behaves depends on the versions you upgrade from and to: * **From 3.10 or later**: Nodes on the old version keep reading and writing the catalog until the last of them is upgraded. Features added by the new version stay unavailable until every running node is on it. The cluster then commits the new feature level, and binaries on the old version can no longer start against the catalog. * **To 3.9 or earlier**: Upgrades that cross the catalog format change in 3.1 or 3.4 block writes that add tables, tags, or fields on nodes on the old version until they’re upgraded. Other upgrades to 3.9 or earlier don’t affect the catalog. * **From 3.9 or earlier to 3.10 or later**: Not a rolling upgrade. The first node on the new version migrates the catalog, and nodes on 3.9 or earlier can’t read the migrated catalog. Stop every node, upgrade them all, and then start the cluster. See [Upgrading to InfluxDB 3.10 is a one-way migration](#upgrading-to-influxdb-310-is-a-one-way-migration). ### Multi-node upgrade procedure Follow these steps to upgrade your InfluxDB 3 Enterprise deployment with minimal downtime. #### Recommended node upgrade order The order depends on the versions you upgrade from and to. **Upgrading from 3.10 or later:** Upgrade the nodes that read other nodes’ output before the nodes that write it. Within a mode, upgrade one node at a time so the others keep serving. 1. **Query nodes** 2. **Process nodes** 3. **Compactor nodes** 4. **Ingest nodes** A node that runs ingest mode together with query or process mode (a plugin directory adds process mode) both reads and writes files. Upgrade such nodes back to back, or stop them together when the release notes announce a file format change. **Upgrading to 3.9 or earlier across the catalog format change in 3.1 or 3.4:** Upgrade ingest nodes first to restore writes that add tables, tags, or fields, then query, compactor, and process nodes. For other upgrades to 3.9 or earlier, the order doesn’t matter. #### Perform a rolling upgrade Follow these steps to upgrade each node in your deployment: [systemctl](#systemctl)[Docker](#docker)[Docker Compose](#docker-compose)[Helm](#helm)[Ansible](#ansible) ```bash # 1. Stop the service sudo systemctl stop influxdb3-enterprise # 2. Install the new version # Follow the installation instructions for your platform: # https://docs.influxdata.com/influxdb3/enterprise/install/ # 3. Start the service sudo systemctl start influxdb3-enterprise # 4. Verify the version influxdb3 --version # 5. Check the node's health influxdb3 query \ --database _internal \ --token ADMIN_TOKEN \ "SELECT * FROM system.queries LIMIT 5" ``` Replace the following: * `ADMIN_TOKEN`: An admin token ```bash # 1. Stop the container docker stop CONTAINER_NAME # 2. Pull the latest image docker pull influxdb:enterprise # 3. Start the container with the new image # IMPORTANT: Adjust the docker run command to match your existing # container configuration, including environment variables, volume mounts, # object store settings, and network settings. docker run -d \ --name CONTAINER_NAME \ -p 8181:8181 \ -e INFLUXDB3_LICENSE_EMAIL=your-email@example.com \ -v ~/.influxdb3/data:/var/lib/influxdb3/data \ influxdb:enterprise \ influxdb3 serve \ --node-id NODE_ID \ --cluster-id CLUSTER_ID \ --object-store OBJECT_STORE_TYPE \ --data-dir /var/lib/influxdb3/data # 4. Verify the version docker exec CONTAINER_NAME influxdb3 --version # 5. Check the node's health docker exec CONTAINER_NAME influxdb3 query \ --database _internal \ --token ADMIN_TOKEN \ "SELECT * FROM system.queries LIMIT 5" ``` Replace the following: * `CONTAINER_NAME`: The name of your InfluxDB 3 Enterprise container * `NODE_ID`: The node identifier for this instance * `CLUSTER_ID`: The cluster identifier for your deployment * `OBJECT_STORE_TYPE`: The object store type (for example, `file`, `s3`, `azure`, or `google`) * `ADMIN_TOKEN`: An admin token > [!Note] > #### Use the influxdb:enterprise image tag > > The `influxdb:enterprise` tag always points to the latest InfluxDB 3 Enterprise release. > Use `docker pull influxdb:enterprise` to pull the latest version, or specify a version tag directly (for example, `influxdb:3.12.2-enterprise`) to upgrade to a specific version. > [!Important] > If you use a cloud object store (S3, Azure, or Google Cloud), include the appropriate credentials and bucket configuration in the `docker run` command. ```bash # 1. Stop the services docker compose down # 2. Update the image in your compose.yaml file # Change the image version to: influxdb:enterprise # 3. Start the services with the new image docker compose up -d # 4. Verify the version docker compose exec influxdb3 influxdb3 --version # 5. Check the node's health docker compose exec influxdb3 influxdb3 query \ --database _internal \ --token ADMIN_TOKEN \ "SELECT * FROM system.queries LIMIT 5" ``` Replace the following: * `ADMIN_TOKEN`: An admin token > [!Note] > #### Use the influxdb:enterprise image tag > > The `influxdb:enterprise` tag always points to the latest InfluxDB 3 Enterprise release. > Update the `image:` field in your `compose.yaml` to `influxdb:enterprise` to pull the latest version, or specify a version tag directly (for example, `influxdb:3.12.2-enterprise`) to upgrade to a specific version. The [InfluxDB 3 Enterprise Helm chart](https://github.com/influxdata/helm-charts/tree/master/charts/influxdb3-enterprise)runs a separate StatefulSet for each node mode, but the image tag (`image.tag`) is a single chart-wide value. A plain `helm upgrade` therefore rolls *every* node mode at once, and Kubernetes doesn’t order rollouts across StatefulSets—so the upgrade doesn’t follow the[recommended node upgrade order](#recommended-node-upgrade-order) on its own. To control the order, freeze the modes you aren’t upgrading yet with the`updateStrategy.rollingUpdate.partition` field, then release them one mode at a time. The steps below follow the order for upgrades from 3.10 or later. Setting `partition` to a value greater than or equal to a StatefulSet’s replica count holds every pod in that StatefulSet at its current version. A partition lower than the replica count lets the pods at or above that ordinal update, so use a ceiling that your replica counts can’t reach. ```bash # 1. Freeze the modes you upgrade later, then apply the new image tag. # Any partition >= a StatefulSet's replica count holds all of its pods, so # 10000 is a ceiling no deployment reaches. Only queriers roll. helm upgrade RELEASE_NAME influxdata/influxdb3-enterprise \ --namespace NAMESPACE \ --reuse-values \ --set image.tag=VERSION-enterprise \ --set processingEngine.updateStrategy.rollingUpdate.partition=10000 \ --set compactor.updateStrategy.rollingUpdate.partition=10000 \ --set ingester.updateStrategy.rollingUpdate.partition=10000 # 2. Wait for the querier pods to roll and become ready kubectl rollout status statefulset --namespace NAMESPACE \ --selector app.kubernetes.io/component=querier # 3. Release the processing engine and wait for it to roll helm upgrade RELEASE_NAME influxdata/influxdb3-enterprise \ --namespace NAMESPACE --reuse-values \ --set processingEngine.updateStrategy.rollingUpdate.partition=0 kubectl rollout status statefulset --namespace NAMESPACE \ --selector app.kubernetes.io/component=processor # 4. Release the compactor and wait for it to roll helm upgrade RELEASE_NAME influxdata/influxdb3-enterprise \ --namespace NAMESPACE --reuse-values \ --set compactor.updateStrategy.rollingUpdate.partition=0 kubectl rollout status statefulset --namespace NAMESPACE \ --selector app.kubernetes.io/component=compactor # 5. Release the ingesters and wait for them to roll helm upgrade RELEASE_NAME influxdata/influxdb3-enterprise \ --namespace NAMESPACE --reuse-values \ --set ingester.updateStrategy.rollingUpdate.partition=0 kubectl rollout status statefulset --namespace NAMESPACE \ --selector app.kubernetes.io/component=ingester # 6. Verify every node re-registered and reports running influxdb3 show nodes ``` Replace the following: * `RELEASE_NAME`: Your Helm release name * `NAMESPACE`: The namespace of your release * `VERSION`: The target version (for example, `3.12.2`) > [!Important] > #### Raise the termination grace period before upgrading > > The chart doesn’t set `terminationGracePeriodSeconds`, so pods inherit the > Kubernetes default of 30 seconds. > If a node is still flushing its write-ahead log when Kubernetes sends`SIGKILL`, it stops ungracefully and has to replay its WAL on restart. > Raise the grace period above your observed shutdown time before you roll a > cluster—see[Deploy with an orchestrator](/influxdb3/enterprise/admin/node-lifecycle/#kubernetes-and-helm). > [!Warning] > #### A rollout is a restart, not a removal > > Each pod keeps its StatefulSet-ordinal name, so every node re-registers under > its existing node ID. > Don’t run[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)as part of an upgrade—removal permanently deletes the node’s catalog entry and > object-store files. > See [Restart compared to removal](/influxdb3/enterprise/admin/node-lifecycle/#restart-compared-to-removal). > [!Caution] > #### helm rollback doesn’t undo a catalog migration > > `helm rollback` reverts the image tag, but it can’t revert changes the newer > version already made to your cluster data. > After a node starts InfluxDB 3 Enterprise 3.10 or later, the on-disk catalog is > migrated to v3 and older binaries fail to start against it—so rolling the > release back leaves pods crash-looping on a catalog they can’t read. > Restoring the catalog objects you backed up is the only way back. > See [Upgrading to InfluxDB 3.10 is a one-way migration](#upgrading-to-influxdb-310-is-a-one-way-migration). Drive the upgrade one node at a time with `serial: 1`, and order your plays by node mode to match the[recommended node upgrade order](#recommended-node-upgrade-order). The playbook below follows the order for upgrades from 3.10 or later. ```yaml # Upgrade one host at a time, query nodes first. - hosts: influxdb3_query serial: 1 vars: # Pin the target version so every host lands on the same build. influxdb3_version: "3.12.2" tasks: - name: Stop influxdb3 gracefully ansible.builtin.systemd_service: name: influxdb3 state: stopped # Replace this task with the install method you use--for example, a # package from your own repository or the downloaded release archive. # See https://docs.influxdata.com/influxdb3/enterprise/install/ - name: Install influxdb3 {{ influxdb3_version }} ansible.builtin.include_role: name: influxdb3_install - name: Start influxdb3 ansible.builtin.systemd_service: name: influxdb3 state: started - name: Wait for the node to report healthy ansible.builtin.uri: url: "http://{{ inventory_hostname }}:8181/health" status_code: 200 register: health until: health.status == 200 retries: 30 delay: 10 # Repeat for influxdb3_process, then influxdb3_compact, then influxdb3_ingest. ``` Because `systemd` escalates to `SIGKILL` after `TimeoutStopSec`, confirm your unit file allows enough time for the final WAL flush before you roll a cluster: ```ini [Service] KillSignal=SIGTERM TimeoutStopSec=300 ``` > [!Important] > #### Restart in place—don’t remove nodes > > Each host restarts with the same[`--node-id`](/influxdb3/enterprise/reference/config-options/#node-id), so it > re-registers as the same node. > Never add[`influxdb3 remove node`](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)to an upgrade playbook—see[Restart compared to removal](/influxdb3/enterprise/admin/node-lifecycle/#restart-compared-to-removal). **Repeat these steps** for each remaining node in the recommended order. ### Troubleshooting cluster upgrades #### Upgrade order issues If you upgrade nodes out of the [recommended order](#recommended-node-upgrade-order), a query or process node on the old version may encounter files in a format it can’t read. It logs errors and keeps serving the data it already loaded; if it restarts, it fails to start. Upgrade the node to recover. #### New features stay unavailable after the upgrade The cluster commits a new feature level only when every running node is on the new version. Run `influxdb3 show nodes`; a node that stopped without a clean shutdown can still show as running and holds the feature level back. Restart that node on the new version. #### Nodes upgrade out of order in Helm deployments The InfluxDB 3 Enterprise Helm chart uses a single chart-wide `image.tag`, so a plain `helm upgrade` rolls every node mode at once instead of following the[recommended node upgrade order](#recommended-node-upgrade-order). Use `updateStrategy.rollingUpdate.partition` to release one mode at a time, as shown in the **Helm** tab of[Perform a rolling upgrade](#perform-a-rolling-upgrade). #### Nodes don’t return to running after a rollout A node that was killed before it finished flushing its write-ahead log stops ungracefully and replays its WAL on restart, which can extend startup. In Kubernetes, this usually means `terminationGracePeriodSeconds` (default 30) is shorter than the node’s shutdown time; with `systemd`, it usually means`TimeoutStopSec` is too low. See [Deploy with an orchestrator](/influxdb3/enterprise/admin/node-lifecycle/#deploy-with-an-orchestrator). #### Extra nodes appear in the catalog after an upgrade Each restart registered a new node ID instead of reclaiming the existing one. Verify that your deployment assigns a stable[`--node-id`](/influxdb3/enterprise/reference/config-options/#node-id)—a Kubernetes Deployment generates a new pod name on every rollout, so use a StatefulSet instead. See [Kubernetes and Helm](/influxdb3/enterprise/admin/node-lifecycle/#kubernetes-and-helm). #### Version compatibility problems If nodes fail to communicate after an upgrade, verify that all nodes are running compatible versions. 1. **Connect to each node** and [verify the version](#verify-your-current-version) 2. Review the [release notes](/influxdb3/enterprise/release-notes/) for your target version to identify any breaking changes or compatibility requirements. #### Related * [Back up and restore data](/influxdb3/enterprise/admin/backup-restore/) * [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) * [Manage databases](/influxdb3/enterprise/admin/databases/) * [Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/) * [Orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/) [upgrade](/influxdb3/enterprise/tags/upgrade/)[administration](/influxdb3/enterprise/tags/administration/)[catalog](/influxdb3/enterprise/tags/catalog/) --- # InfluxDB HTTP API Source: https://docs.influxdata.com/influxdb3/enterprise/api/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Use the InfluxDB 3 Enterprise HTTP API to write data, query data, and manage databases, tables, and tokens. ### [Quick start](/influxdb3/enterprise/api/quick-start/) Authenticate, write, and query with the API: ### [Authentication](/influxdb3/enterprise/api/authentication/) Use one of the following schemes to authenticate to the InfluxDB 3 Enterprise API: ### [Headers and parameters](/influxdb3/enterprise/api/headers-and-parameters/) Most InfluxDB API endpoints require parameters in the request–for example, specifying the database to use. ### [Migrate from InfluxDB v1 or v2](/influxdb3/enterprise/api/migrate-from-influxdb-v1-or-v2/) Migrate your existing InfluxDB v1 or v2 workloads to InfluxDB 3 Enterprise. ### [Auth token](/influxdb3/enterprise/api/auth-token/) Create and manage tokens used for authenticating and authorizing access to InfluxDB 3 Enterprise resources. ### [Backup and restore](/influxdb3/enterprise/api/backup-and-restore/) Create, list, retrieve, and delete backups, and start, list, and cancel restores. ### [Bulk import](/influxdb3/enterprise/api/bulk-import/) Upload Parquet files for bulk import into existing tables and list the status of bulk import jobs. ### [Cache distinct values](/influxdb3/enterprise/api/cache-distinct-values/) The Distinct Value Cache (DVC) lets you cache distinct values of one or more columns in a table, improving the performance of queries that return distinct tag and field values. ### [Cache last value](/influxdb3/enterprise/api/cache-last-value/) The Last Value Cache (LVC) lets you cache the most recent values for specific fields in a table, improving the performance of queries that return the most recent value of a field for specific series… ### [Data deletion](/influxdb3/enterprise/api/data-deletion/) Manage row-delete requests. ### [Database](/influxdb3/enterprise/api/database/) Create, list, and delete databases in InfluxDB 3 Enterprise. ### [Enterprise](/influxdb3/enterprise/api/enterprise/) Endpoints available only in InfluxDB 3 Enterprise. Operations carrying this tag also set `x-enterprise-only: true` for programmatic detection. ### [Export data](/influxdb3/enterprise/api/export-data/) Export data from InfluxDB 3 Enterprise databases. List databases, tables, export windows, and retrieve window data for data migration and backup workflows. ### [Load capture](/influxdb3/enterprise/api/load-capture/) Capture an anonymized profile of write requests, query requests, or both on a query-capable node. ### [Processing engine](/influxdb3/enterprise/api/processing-engine/) Manage Processing engine triggers, test plugins, and send requests to trigger On Request plugins. ### [Query data](/influxdb3/enterprise/api/query-data/) Query data stored in InfluxDB 3 Enterprise using SQL or InfluxQL. ### [Query groups](/influxdb3/enterprise/api/query-groups/) Manage distributed query groups–named sets of query nodes that InfluxDB 3 Enterprise load-balances queries across. ### [Server information](/influxdb3/enterprise/api/server-information/) Retrieve server metrics, health status, and version information for InfluxDB 3 Enterprise. ### [Table](/influxdb3/enterprise/api/table/) Manage table schemas in an InfluxDB 3 Enterprise database. ### [User management](/influxdb3/enterprise/api/user-management/) Manage users and their role assignments, and look up users and roles. ### [Write data](/influxdb3/enterprise/api/write-data/) Write data to InfluxDB 3 Enterprise using line protocol format. --- # All endpoints Source: https://docs.influxdata.com/influxdb3/enterprise/api/all-endpoints/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) All InfluxDB 3 Enterprise API endpoints, sorted by path. ## v3 API [POST`/api/v3/authorize`](/influxdb3/enterprise/api/enterprise/#operation/PostAuthorize)[POST`/api/v3/authorize/refresh`](/influxdb3/enterprise/api/enterprise/#operation/PostAuthorizeRefresh)[POST`/api/v3/authorize/reset-password`](/influxdb3/enterprise/api/enterprise/#operation/PostAuthorizeResetPassword)[DELETE`/api/v3/configure/database`Delete a database](/influxdb3/enterprise/api/database/#operation/DeleteConfigureDatabase)[GET`/api/v3/configure/database`List databases](/influxdb3/enterprise/api/database/#operation/GetConfigureDatabase)[POST`/api/v3/configure/database`Create a database](/influxdb3/enterprise/api/database/#operation/PostConfigureDatabase)[PUT`/api/v3/configure/database`Update a database](/influxdb3/enterprise/api/database/#operation/PutConfigureDatabase)[DELETE`/api/v3/configure/database/retention_period`Remove database retention period](/influxdb3/enterprise/api/database/#operation/DeleteDatabaseRetentionPeriod)[DELETE`/api/v3/configure/distinct_cache`Delete distinct cache](/influxdb3/enterprise/api/cache-distinct-values/#operation/DeleteConfigureDistinctCache)[DELETE`/api/v3/configure/distinct_cache`Delete distinct cache](/influxdb3/enterprise/api/table/#operation/DeleteConfigureDistinctCache)[POST`/api/v3/configure/distinct_cache`Create distinct cache](/influxdb3/enterprise/api/cache-distinct-values/#operation/PostConfigureDistinctCache)[POST`/api/v3/configure/distinct_cache`Create distinct cache](/influxdb3/enterprise/api/table/#operation/PostConfigureDistinctCache)[DELETE`/api/v3/configure/last_cache`Delete last cache](/influxdb3/enterprise/api/cache-last-value/#operation/DeleteConfigureLastCache)[DELETE`/api/v3/configure/last_cache`Delete last cache](/influxdb3/enterprise/api/table/#operation/DeleteConfigureLastCache)[POST`/api/v3/configure/last_cache`Create last cache](/influxdb3/enterprise/api/cache-last-value/#operation/PostConfigureLastCache)[POST`/api/v3/configure/last_cache`Create last cache](/influxdb3/enterprise/api/table/#operation/PostConfigureLastCache)[POST`/api/v3/configure/plugin_environment/install_packages`Install plugin packages](/influxdb3/enterprise/api/processing-engine/#operation/PostInstallPluginPackages)[POST`/api/v3/configure/plugin_environment/install_requirements`Install plugin requirements](/influxdb3/enterprise/api/processing-engine/#operation/PostInstallPluginRequirements)[DELETE`/api/v3/configure/processing_engine_trigger`Delete processing engine trigger](/influxdb3/enterprise/api/processing-engine/#operation/DeleteConfigureProcessingEngineTrigger)[POST`/api/v3/configure/processing_engine_trigger`Create processing engine trigger](/influxdb3/enterprise/api/processing-engine/#operation/PostConfigureProcessingEngineTrigger)[POST`/api/v3/configure/processing_engine_trigger/disable`Disable processing engine trigger](/influxdb3/enterprise/api/processing-engine/#operation/PostDisableProcessingEngineTrigger)[POST`/api/v3/configure/processing_engine_trigger/enable`Enable processing engine trigger](/influxdb3/enterprise/api/processing-engine/#operation/PostEnableProcessingEngineTrigger)[DELETE`/api/v3/configure/query_concurrency_limit`Reset the query concurrency limit](/influxdb3/enterprise/api/enterprise/#operation/DeleteConfigureQueryConcurrencyLimit)[DELETE`/api/v3/configure/query_concurrency_limit`Reset the query concurrency limit](/influxdb3/enterprise/api/server-information/#operation/DeleteConfigureQueryConcurrencyLimit)[GET`/api/v3/configure/query_concurrency_limit`Get the query concurrency limit](/influxdb3/enterprise/api/enterprise/#operation/GetConfigureQueryConcurrencyLimit)[GET`/api/v3/configure/query_concurrency_limit`Get the query concurrency limit](/influxdb3/enterprise/api/server-information/#operation/GetConfigureQueryConcurrencyLimit)[PUT`/api/v3/configure/query_concurrency_limit`Set the query concurrency limit](/influxdb3/enterprise/api/enterprise/#operation/PutConfigureQueryConcurrencyLimit)[PUT`/api/v3/configure/query_concurrency_limit`Set the query concurrency limit](/influxdb3/enterprise/api/server-information/#operation/PutConfigureQueryConcurrencyLimit)[DELETE`/api/v3/configure/table`Delete a table](/influxdb3/enterprise/api/table/#operation/DeleteConfigureTable)[PATCH`/api/v3/configure/table`Add columns to a table](/influxdb3/enterprise/api/table/#operation/PatchConfigureTable2)[POST`/api/v3/configure/table`Create a table](/influxdb3/enterprise/api/table/#operation/PostConfigureTable)[PUT`/api/v3/configure/table`Update a table](/influxdb3/enterprise/api/enterprise/#operation/PatchConfigureTable)[DELETE`/api/v3/configure/token`Delete token](/influxdb3/enterprise/api/auth-token/#operation/DeleteToken)[POST`/api/v3/configure/token/admin`Create admin token](/influxdb3/enterprise/api/auth-token/#operation/PostCreateAdminToken)[POST`/api/v3/configure/token/admin/regenerate`Regenerate admin token](/influxdb3/enterprise/api/auth-token/#operation/PostRegenerateAdminToken)[POST`/api/v3/configure/token/named_admin`Create named admin token](/influxdb3/enterprise/api/auth-token/#operation/PostCreateNamedAdminToken)[POST`/api/v3/configure/user`](/influxdb3/enterprise/api/enterprise/#operation/PostConfigureUser)[GET`/api/v3/engine/{request_path}`On Request processing engine plugin request](/influxdb3/enterprise/api/processing-engine/#operation/GetProcessingEnginePluginRequest)[POST`/api/v3/engine/{request_path}`On Request processing engine plugin request](/influxdb3/enterprise/api/processing-engine/#operation/PostProcessingEnginePluginRequest)[DELETE`/api/v3/enterprise/backup`Cancel a running backup](/influxdb3/enterprise/api/backup-and-restore/#operation/DeleteEnterpriseBackup)[DELETE`/api/v3/enterprise/backup`Cancel a running backup](/influxdb3/enterprise/api/enterprise/#operation/DeleteEnterpriseBackup)[GET`/api/v3/enterprise/backup`List backups](/influxdb3/enterprise/api/backup-and-restore/#operation/GetEnterpriseBackup)[GET`/api/v3/enterprise/backup`List backups](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseBackup)[POST`/api/v3/enterprise/backup`Start a backup](/influxdb3/enterprise/api/backup-and-restore/#operation/PostEnterpriseBackup)[POST`/api/v3/enterprise/backup`Start a backup](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseBackup)[DELETE`/api/v3/enterprise/backup/{name}`Delete a backup](/influxdb3/enterprise/api/backup-and-restore/#operation/DeleteEnterpriseBackupByName)[DELETE`/api/v3/enterprise/backup/{name}`Delete a backup](/influxdb3/enterprise/api/enterprise/#operation/DeleteEnterpriseBackupByName)[GET`/api/v3/enterprise/backup/{name}`Get a backup](/influxdb3/enterprise/api/backup-and-restore/#operation/GetEnterpriseBackupByName)[GET`/api/v3/enterprise/backup/{name}`Get a backup](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseBackupByName)[DELETE`/api/v3/enterprise/configure/file_index`Delete a file index](/influxdb3/enterprise/api/database/#operation/DeleteEnterpriseConfigureFileIndex)[DELETE`/api/v3/enterprise/configure/file_index`Delete a file index](/influxdb3/enterprise/api/enterprise/#operation/DeleteEnterpriseConfigureFileIndex)[DELETE`/api/v3/enterprise/configure/file_index`Delete a file index](/influxdb3/enterprise/api/table/#operation/DeleteEnterpriseConfigureFileIndex)[POST`/api/v3/enterprise/configure/file_index`Create a file index](/influxdb3/enterprise/api/database/#operation/PostEnterpriseConfigureFileIndex)[POST`/api/v3/enterprise/configure/file_index`Create a file index](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseConfigureFileIndex)[POST`/api/v3/enterprise/configure/file_index`Create a file index](/influxdb3/enterprise/api/table/#operation/PostEnterpriseConfigureFileIndex)[POST`/api/v3/enterprise/configure/node/remove`Permanently remove a stopped node](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseConfigureNodeRemove)[POST`/api/v3/enterprise/configure/node/remove`Permanently remove a stopped node](/influxdb3/enterprise/api/server-information/#operation/PostEnterpriseConfigureNodeRemove)[POST`/api/v3/enterprise/configure/node/stop`Mark a node as stopped](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseConfigureNodeStop)[POST`/api/v3/enterprise/configure/node/stop`Mark a node as stopped](/influxdb3/enterprise/api/server-information/#operation/PostEnterpriseConfigureNodeStop)[GET`/api/v3/enterprise/configure/query_groups`List distributed query groups (not yet operational)](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseConfigureQueryGroups)[GET`/api/v3/enterprise/configure/query_groups`List distributed query groups (not yet operational)](/influxdb3/enterprise/api/query-groups/#operation/GetEnterpriseConfigureQueryGroups)[POST`/api/v3/enterprise/configure/query_groups`Create a distributed query group (not yet operational)](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseConfigureQueryGroups)[POST`/api/v3/enterprise/configure/query_groups`Create a distributed query group (not yet operational)](/influxdb3/enterprise/api/query-groups/#operation/PostEnterpriseConfigureQueryGroups)[DELETE`/api/v3/enterprise/configure/query_groups/{id}`Delete a distributed query group (not yet operational)](/influxdb3/enterprise/api/enterprise/#operation/DeleteEnterpriseConfigureQueryGroupsById)[DELETE`/api/v3/enterprise/configure/query_groups/{id}`Delete a distributed query group (not yet operational)](/influxdb3/enterprise/api/query-groups/#operation/DeleteEnterpriseConfigureQueryGroupsById)[GET`/api/v3/enterprise/configure/query_groups/{id}`Retrieve a distributed query group (not yet operational)](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseConfigureQueryGroupsById)[GET`/api/v3/enterprise/configure/query_groups/{id}`Retrieve a distributed query group (not yet operational)](/influxdb3/enterprise/api/query-groups/#operation/GetEnterpriseConfigureQueryGroupsById)[PATCH`/api/v3/enterprise/configure/query_groups/{id}`Update a distributed query group (not yet operational)](/influxdb3/enterprise/api/enterprise/#operation/PatchEnterpriseConfigureQueryGroupsById)[PATCH`/api/v3/enterprise/configure/query_groups/{id}`Update a distributed query group (not yet operational)](/influxdb3/enterprise/api/query-groups/#operation/PatchEnterpriseConfigureQueryGroupsById)[PUT`/api/v3/enterprise/configure/query_groups/{id}`Replace a distributed query group (not yet operational)](/influxdb3/enterprise/api/enterprise/#operation/PutEnterpriseConfigureQueryGroupsById)[PUT`/api/v3/enterprise/configure/query_groups/{id}`Replace a distributed query group (not yet operational)](/influxdb3/enterprise/api/query-groups/#operation/PutEnterpriseConfigureQueryGroupsById)[DELETE`/api/v3/enterprise/configure/table/retention_period`Clear table retention period](/influxdb3/enterprise/api/enterprise/#operation/DeleteEnterpriseConfigureTableRetentionPeriod)[DELETE`/api/v3/enterprise/configure/table/retention_period`Clear table retention period](/influxdb3/enterprise/api/table/#operation/DeleteEnterpriseConfigureTableRetentionPeriod)[POST`/api/v3/enterprise/configure/table/retention_period`Set table retention period](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseConfigureTableRetentionPeriod)[POST`/api/v3/enterprise/configure/table/retention_period`Set table retention period](/influxdb3/enterprise/api/table/#operation/PostEnterpriseConfigureTableRetentionPeriod)[POST`/api/v3/enterprise/configure/token`Create a resource token](/influxdb3/enterprise/api/auth-token/#operation/PostCreateResourceToken)[POST`/api/v3/enterprise/configure/token`Create a resource token](/influxdb3/enterprise/api/enterprise/#operation/PostCreateResourceToken)[GET`/api/v3/enterprise/import`List bulk import statuses](/influxdb3/enterprise/api/bulk-import/#operation/GetEnterpriseImport)[GET`/api/v3/enterprise/import`List bulk import statuses](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseImport)[POST`/api/v3/enterprise/import`Import Parquet data](/influxdb3/enterprise/api/bulk-import/#operation/PostEnterpriseImport)[POST`/api/v3/enterprise/import`Import Parquet data](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseImport)[GET`/api/v3/enterprise/restore`List restores](/influxdb3/enterprise/api/backup-and-restore/#operation/GetEnterpriseRestore)[GET`/api/v3/enterprise/restore`List restores](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseRestore)[POST`/api/v3/enterprise/restore`Start a restore](/influxdb3/enterprise/api/backup-and-restore/#operation/PostEnterpriseRestore)[POST`/api/v3/enterprise/restore`Start a restore](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseRestore)[DELETE`/api/v3/enterprise/restore/{id}`Cancel a running restore](/influxdb3/enterprise/api/backup-and-restore/#operation/DeleteEnterpriseRestoreById)[DELETE`/api/v3/enterprise/restore/{id}`Cancel a running restore](/influxdb3/enterprise/api/enterprise/#operation/DeleteEnterpriseRestoreById)[GET`/api/v3/enterprise/restore/{id}`Get a restore](/influxdb3/enterprise/api/backup-and-restore/#operation/GetEnterpriseRestoreById)[GET`/api/v3/enterprise/restore/{id}`Get a restore](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseRestoreById)[GET`/api/v3/enterprise/upgrade/parquet_cleanup`Get Parquet cleanup status](/influxdb3/enterprise/api/enterprise/#operation/GetEnterpriseUpgradeParquetCleanup)[GET`/api/v3/enterprise/upgrade/parquet_cleanup`Get Parquet cleanup status](/influxdb3/enterprise/api/server-information/#operation/GetEnterpriseUpgradeParquetCleanup)[POST`/api/v3/enterprise/upgrade/parquet_cleanup`Request Parquet cleanup after a storage engine upgrade](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseUpgradeParquetCleanup)[POST`/api/v3/enterprise/upgrade/parquet_cleanup`Request Parquet cleanup after a storage engine upgrade](/influxdb3/enterprise/api/server-information/#operation/PostEnterpriseUpgradeParquetCleanup)[POST`/api/v3/enterprise/upgrade/retry_parquet_to_pacha_tree`Retry a failed Parquet-to-PachaTree upgrade](/influxdb3/enterprise/api/enterprise/#operation/PostEnterpriseUpgradeRetryParquetToPachaTree)[POST`/api/v3/enterprise/upgrade/retry_parquet_to_pacha_tree`Retry a failed Parquet-to-PachaTree upgrade](/influxdb3/enterprise/api/server-information/#operation/PostEnterpriseUpgradeRetryParquetToPachaTree)[GET`/api/v3/export/databases`List databases available for export](/influxdb3/enterprise/api/enterprise/#operation/GetExportDatabases)[GET`/api/v3/export/databases`List databases available for export](/influxdb3/enterprise/api/export-data/#operation/GetExportDatabases)[GET`/api/v3/export/tables`List tables available for export](/influxdb3/enterprise/api/enterprise/#operation/GetExportTables)[GET`/api/v3/export/tables`List tables available for export](/influxdb3/enterprise/api/export-data/#operation/GetExportTables)[GET`/api/v3/export/window_data`Export window data as Parquet](/influxdb3/enterprise/api/enterprise/#operation/GetExportWindowData)[GET`/api/v3/export/window_data`Export window data as Parquet](/influxdb3/enterprise/api/export-data/#operation/GetExportWindowData)[GET`/api/v3/export/windows`List compacted windows for a table](/influxdb3/enterprise/api/enterprise/#operation/GetExportWindows)[GET`/api/v3/export/windows`List compacted windows for a table](/influxdb3/enterprise/api/export-data/#operation/GetExportWindows)[POST`/api/v3/isAuthorized`Check whether the caller is authorized for an action](/influxdb3/enterprise/api/enterprise/#operation/PostIsAuthorized)[GET`/api/v3/loadcap/profiles`List load capture profiles](/influxdb3/enterprise/api/enterprise/#operation/GetLoadcapProfiles)[GET`/api/v3/loadcap/profiles`List load capture profiles](/influxdb3/enterprise/api/load-capture/#operation/GetLoadcapProfiles)[DELETE`/api/v3/loadcap/profiles/{profile_id}`Delete a load capture profile](/influxdb3/enterprise/api/enterprise/#operation/DeleteLoadcapProfile)[DELETE`/api/v3/loadcap/profiles/{profile_id}`Delete a load capture profile](/influxdb3/enterprise/api/load-capture/#operation/DeleteLoadcapProfile)[GET`/api/v3/loadcap/profiles/{profile_id}`Retrieve load capture profile details](/influxdb3/enterprise/api/enterprise/#operation/GetLoadcapProfile)[GET`/api/v3/loadcap/profiles/{profile_id}`Retrieve load capture profile details](/influxdb3/enterprise/api/load-capture/#operation/GetLoadcapProfile)[GET`/api/v3/loadcap/profiles/{profile_id}/download`Download a load capture profile](/influxdb3/enterprise/api/enterprise/#operation/GetLoadcapProfileDownload)[GET`/api/v3/loadcap/profiles/{profile_id}/download`Download a load capture profile](/influxdb3/enterprise/api/load-capture/#operation/GetLoadcapProfileDownload)[GET`/api/v3/loadcap/profiles/{profile_id}/files`List files in a load capture profile](/influxdb3/enterprise/api/enterprise/#operation/GetLoadcapProfileFiles)[GET`/api/v3/loadcap/profiles/{profile_id}/files`List files in a load capture profile](/influxdb3/enterprise/api/load-capture/#operation/GetLoadcapProfileFiles)[GET`/api/v3/loadcap/profiles/{profile_id}/files/{filename}`Download a single file from a load capture profile](/influxdb3/enterprise/api/enterprise/#operation/GetLoadcapProfileFile)[GET`/api/v3/loadcap/profiles/{profile_id}/files/{filename}`Download a single file from a load capture profile](/influxdb3/enterprise/api/load-capture/#operation/GetLoadcapProfileFile)[GET`/api/v3/loadcap/profiles/{profile_id}/preview`Preview a load capture profile](/influxdb3/enterprise/api/enterprise/#operation/GetLoadcapProfilePreview)[GET`/api/v3/loadcap/profiles/{profile_id}/preview`Preview a load capture profile](/influxdb3/enterprise/api/load-capture/#operation/GetLoadcapProfilePreview)[POST`/api/v3/loadcap/start`Start a load capture](/influxdb3/enterprise/api/enterprise/#operation/PostLoadcapStart)[POST`/api/v3/loadcap/start`Start a load capture](/influxdb3/enterprise/api/load-capture/#operation/PostLoadcapStart)[POST`/api/v3/plugin_test/schedule`Test scheduling plugin](/influxdb3/enterprise/api/processing-engine/#operation/PostTestSchedulingPlugin)[POST`/api/v3/plugin_test/wal`Test WAL plugin](/influxdb3/enterprise/api/processing-engine/#operation/PostTestWALPlugin)[PUT`/api/v3/plugins/directory`Update a multi-file plugin directory](/influxdb3/enterprise/api/processing-engine/#operation/PutPluginDirectory)[POST`/api/v3/plugins/files`Create a plugin file](/influxdb3/enterprise/api/processing-engine/#operation/PostPluginsFiles)[PUT`/api/v3/plugins/files`Update a plugin file](/influxdb3/enterprise/api/processing-engine/#operation/PutPluginFile)[GET`/api/v3/query_influxql`Execute InfluxQL query](/influxdb3/enterprise/api/query-data/#operation/GetExecuteInfluxQLQuery)[POST`/api/v3/query_influxql`Execute InfluxQL query](/influxdb3/enterprise/api/query-data/#operation/PostExecuteQueryInfluxQL)[GET`/api/v3/query_sql`Execute SQL query](/influxdb3/enterprise/api/query-data/#operation/GetExecuteQuerySQL)[POST`/api/v3/query_sql`Execute SQL query](/influxdb3/enterprise/api/query-data/#operation/PostExecuteQuerySQL)[GET`/api/v3/roles`](/influxdb3/enterprise/api/enterprise/#operation/GetRoles)[GET`/api/v3/roles_by_name/{roleName}`Get a role by name](/influxdb3/enterprise/api/enterprise/#operation/GetRolesByNameByRoleName)[GET`/api/v3/roles_by_name/{roleName}`Get a role by name](/influxdb3/enterprise/api/user-management/#operation/GetRolesByNameByRoleName)[GET`/api/v3/roles/{id}/permissions`Get a role's permissions](/influxdb3/enterprise/api/enterprise/#operation/GetRolesByIdPermissions)[GET`/api/v3/roles/{id}/permissions`Get a role's permissions](/influxdb3/enterprise/api/user-management/#operation/GetRolesByIdPermissions)[GET`/api/v3/row_delete_requests`](/influxdb3/enterprise/api/enterprise/#operation/GetRowDeleteRequests)[POST`/api/v3/row_delete_requests`](/influxdb3/enterprise/api/enterprise/#operation/PostRowDeleteRequests)[POST`/api/v3/row_delete_requests/{seq}/abort`Abort a pending row-delete request](/influxdb3/enterprise/api/data-deletion/#operation/PostRowDeleteRequestsBySeqAbort)[POST`/api/v3/row_delete_requests/{seq}/abort`Abort a pending row-delete request](/influxdb3/enterprise/api/enterprise/#operation/PostRowDeleteRequestsBySeqAbort)[GET`/api/v3/users`](/influxdb3/enterprise/api/enterprise/#operation/GetUsers)[POST`/api/v3/users`](/influxdb3/enterprise/api/enterprise/#operation/PostUsers)[GET`/api/v3/users_by_oauth_id/{oauth_id}`Get a user by OAuth identity-provider ID](/influxdb3/enterprise/api/enterprise/#operation/GetUsersByOauthIdByOauthId)[GET`/api/v3/users_by_oauth_id/{oauth_id}`Get a user by OAuth identity-provider ID](/influxdb3/enterprise/api/user-management/#operation/GetUsersByOauthIdByOauthId)[GET`/api/v3/users/{id}`](/influxdb3/enterprise/api/enterprise/#operation/GetUsersById)[POST`/api/v3/users/{id}/require-password-reset`Require a user to reset their password on next login](/influxdb3/enterprise/api/enterprise/#operation/PostUsersByIdRequirePasswordReset)[POST`/api/v3/users/{id}/require-password-reset`Require a user to reset their password on next login](/influxdb3/enterprise/api/user-management/#operation/PostUsersByIdRequirePasswordReset)[GET`/api/v3/users/{id}/roles`List all roles assigned to a user](/influxdb3/enterprise/api/enterprise/#operation/GetUsersByIdRoles)[GET`/api/v3/users/{id}/roles`List all roles assigned to a user](/influxdb3/enterprise/api/user-management/#operation/GetUsersByIdRoles)[PUT`/api/v3/users/{id}/roles`Replace all roles assigned to a user](/influxdb3/enterprise/api/enterprise/#operation/PutUsersByIdRoles)[PUT`/api/v3/users/{id}/roles`Replace all roles assigned to a user](/influxdb3/enterprise/api/user-management/#operation/PutUsersByIdRoles)[POST`/api/v3/write_lp`Write line protocol](/influxdb3/enterprise/api/write-data/#operation/PostWriteLP) ## v2-compatible API [POST`/api/v2/write`Write line protocol (v2-compatible)](/influxdb3/enterprise/api/write-data/#operation/PostV2Write) ## v1-compatible API [GET`/api/v1/health`Health check (v1)](/influxdb3/enterprise/api/server-information/#operation/GetHealthV1)[GET`/health`Health check](/influxdb3/enterprise/api/server-information/#operation/GetHealth)[GET`/metrics`Metrics](/influxdb3/enterprise/api/server-information/#operation/GetMetrics)[GET`/ping`Ping the server](/influxdb3/enterprise/api/server-information/#operation/GetPing)[POST`/ping`Ping the server](/influxdb3/enterprise/api/server-information/#operation/ping)[GET`/query`Execute InfluxQL query (v1-compatible)](/influxdb3/enterprise/api/query-data/#operation/GetV1ExecuteQuery)[POST`/query`Execute InfluxQL query (v1-compatible)](/influxdb3/enterprise/api/query-data/#operation/PostExecuteV1Query)[POST`/write`Write line protocol (v1-compatible)](/influxdb3/enterprise/api/write-data/#operation/PostV1Write) ## Management API [GET`/ready`](/influxdb3/enterprise/api/enterprise/#operation/ready) --- # Auth token Source: https://docs.influxdata.com/influxdb3/enterprise/api/auth-token/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Create and manage tokens used for authenticating and authorizing access to InfluxDB 3 Enterprise resources. DELETE`/api/v3/configure/token` ### Delete token Deletes a token. #### Parameters ##### Query parameters `token_name`requiredstring The name of the token to delete. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/token?token_name=TOKEN_NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The token has been deleted. 401Unauthorized access. `data`object `error`string 404Token not found. POST`/api/v3/configure/token/admin` ### Create admin token Creates an admin token. An admin token is a special type of token that has full access to all resources in the system. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/token/admin" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 201Success. The admin token has been created. The response body contains the token string and metadata. `created_at`string \ `expiry`string \ `hash`string `id`integer `name`string `token`string Example request body ```json { "created_at": "2025-04-18T14:02:45.331Z", "expiry": null, "hash": "00xx0Xx0xx00XX0x0", "id": 0, "name": "_admin", "token": "apiv3_00xx0Xx0xx00XX0x0" } ``` 401Unauthorized access. `data`object `error`string POST`/api/v3/configure/token/admin/regenerate` ### Regenerate admin token Regenerates an admin token and revokes the previous token with the same name. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/token/admin/regenerate" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 201Success. The admin token has been regenerated. `created_at`string \ `expiry`string \ `hash`string `id`integer `name`string `token`string Example request body ```json { "created_at": "2025-04-18T14:02:45.331Z", "expiry": null, "hash": "00xx0Xx0xx00XX0x0", "id": 0, "name": "_admin", "token": "apiv3_00xx0Xx0xx00XX0x0" } ``` 401Unauthorized access. `data`object `error`string POST`/api/v3/configure/token/named_admin` ### Create named admin token Creates a named admin token. A named admin token is a special type of admin token with a custom name for identification and management. #### Request bodyrequired Content-Type:`application/json` `expiry_secs`integer Optional expiration time in seconds. If not provided, the token does not expire. `token_name`requiredstring The name for the admin token. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/token/named_admin" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "expiry_secs": 0, "token_name": "TOKEN_NAME" }' ``` #### Responses 201Success. The named admin token has been created. The response body contains the token string and metadata. `created_at`string \ `expiry`string \ `hash`string `id`integer `name`string `token`string Example request body ```json { "created_at": "2025-04-18T14:02:45.331Z", "expiry": null, "hash": "00xx0Xx0xx00XX0x0", "id": 0, "name": "_admin", "token": "apiv3_00xx0Xx0xx00XX0x0" } ``` 401Unauthorized access. `data`object `error`string 409A token with this name already exists. POST`/api/v3/enterprise/configure/token` ### Create a resource token Creates a resource (fine-grained permissions) token. A resource token is a token that has access to specific resources in the system. This endpoint is only available in InfluxDB 3 Enterprise. InfluxDB 3 Cloud supports a narrower set of database-scoped read and write tokens instead of resource tokens—see [Manage tokens in InfluxDB 3 Cloud](/influxdb3/cloud/admin/tokens/). InfluxDB 3 Core doesn’t support resource tokens. #### Request bodyrequired Content-Type:`application/json` `expiry_secs`integer Optional expiration time in seconds. `permissions`requiredobject[] List of permissions to grant to the token. `actions`requiredstring[] List of actions to grant. `resource_names`requiredstring[] List of resource names. Use “\*” for all resources. `resource_type`requiredstring The type of resource. Allowed:`system`, `db` `token_name`requiredstring The name for the resource token. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/token" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "expiry_secs": 0, "permissions": [ { "actions": [], "resource_names": [], "resource_type": "system" } ], "token_name": "TOKEN_NAME" }' ``` #### Responses 201Success. The resource token has been created. The response body contains the token string and metadata. `expiry_secs`integer The expiration time in seconds. `permissions`object[] `actions`string[] `resource_names`string[] List of resource names. Use “\*” for all resources. `resource_type`string Allowed:`system`, `db` `token_name`string Example request body ```json { "expiry_secs": 300000, "permissions": [ { "actions": [ "read" ], "resource_names": [ "*" ], "resource_type": "system" } ], "token_name": "All system information" } ``` 401Unauthorized access. `data`object `error`string #### Related * [Manage tokens](/influxdb3/enterprise/admin/tokens/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Authentication Source: https://docs.influxdata.com/influxdb3/enterprise/api/authentication/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Use one of the following schemes to authenticate to the InfluxDB 3 Enterprise API: * Token authentication * Bearer authentication * Basic authentication * Querystring authentication ## Basic Authentication Use the `Authorization` header with the `Basic` scheme to authenticate v1 API requests. Works with v1 compatibility [`/write`](#operation/PostV1Write) and [`/query`](#operation/GetV1ExecuteQuery) endpoints in InfluxDB 3. When authenticating requests, InfluxDB 3 checks that the `password` part of the decoded credential is an authorized token and ignores the `username` part of the decoded credential. ### Syntax ```http Authorization: Basic ``` ### Example ```bash curl "http://localhost:8181/write?db=DATABASE_NAME&precision=s" \ --user "":"AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` Replace the following: * **`DATABASE_NAME`**: your InfluxDB 3 Enterprise database * **`AUTH_TOKEN`**: an admin token or database token authorized for the database ## Bearer Authentication Use the OAuth Bearer authentication scheme to provide an authorization token to InfluxDB 3. Bearer authentication works with all endpoints. In your API requests, send an `Authorization` header. For the header value, provide the word `Bearer` followed by a space and a database token. ### Syntax ```http Authorization: Bearer AUTH_TOKEN ``` ### Example ```bash curl http://localhost:8181/api/v3/query_influxql \ --header "Authorization: Bearer AUTH_TOKEN" ``` ## Query String Authentication Use InfluxDB 1.x API parameters to provide credentials through the query string for v1 API requests. Querystring authentication works with v1-compatible [`/write`](#operation/PostV1Write) and [`/query`](#operation/GetV1ExecuteQuery) endpoints. When authenticating requests, InfluxDB 3 checks that the `p` (*password*) query parameter is an authorized token and ignores the `u` (*username*) query parameter. ### Syntax ```http https://localhost:8181/query/?[u=any]&p=AUTH_TOKEN https://localhost:8181/write/?[u=any]&p=AUTH_TOKEN ``` ### Examples ```bash curl "http://localhost:8181/write?db=DATABASE_NAME&precision=s&p=AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` Replace the following: * **`DATABASE_NAME`**: your InfluxDB 3 Enterprise database * **`AUTH_TOKEN`**: an admin token or database token authorized for the database ```bash ####################################### # Use an InfluxDB 1.x compatible username and password # to query the InfluxDB v1 HTTP API ####################################### # Use authentication query parameters: # ?p=AUTH_TOKEN ####################################### curl --get "http://localhost:8181/query" \ --data-urlencode "p=AUTH_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM MEASUREMENT" ``` Replace the following: * **`DATABASE_NAME`**: the database to query * **`AUTH_TOKEN`**: a database token with sufficient permissions to the database ## Token Authentication Use InfluxDB v2 Token authentication to provide an authorization token to InfluxDB 3. The v2 Token scheme works with v1 and v2 compatibility endpoints in InfluxDB 3. In your API requests, send an `Authorization` header. For the header value, provide the word `Token` followed by a space and a database token. The word `Token` is case-sensitive. ### Syntax ```http Authorization: Token AUTH_TOKEN ``` ### Example ```sh ######################################################## # Use the Token authentication scheme with /api/v2/write # to write data. ######################################################## curl --request post "http://localhost:8181/api/v2/write?bucket=DATABASE_NAME&precision=s" \ --header "Authorization: Token AUTH_TOKEN" \ --data-binary 'home,room=kitchen temp=72 1463683075' ``` #### Related * [Authenticate v1 API requests](/influxdb3/enterprise/guides/api-compatibility/v1/) * [Manage tokens](/influxdb3/enterprise/admin/tokens/) --- # Backup and restore Source: https://docs.influxdata.com/influxdb3/enterprise/api/backup-and-restore/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Create, list, retrieve, and delete backups, and start, list, and cancel restores. Backup and restore endpoints require the upgraded storage engine, enabled with the `--use-pacha-tree` flag, and a compactor node that runs the backup and restore engine. If the storage engine is not present on the node, these endpoints return `503 Service Unavailable`. All endpoints require an admin (operator) token or admin user. Backups and restores run asynchronously: starting one returns immediately with a handle that you poll for status. Only one restore can run cluster-wide at a time. GET`/api/v3/enterprise/backup` ### List backups Lists all full backups. Requires an admin (operator) token. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/backup" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of backups. `backups`requiredobject[] All full backups. `completed_at`string \ The completion timestamp. Omitted when the backup is not complete. `created_at`requiredstring \ The creation timestamp. `name`requiredstring The backup name. `status`requiredstring The status of a backup or restore. Allowed:`in_progress`, `completed`, `failed` `total_files`requiredinteger \ The number of files in the backup. `total_size_bytes`requiredinteger \ The total backup size in bytes. `type`requiredstring The backup type. Allowed:`full`, `incremental` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 500Internal server error. Listing backups failed. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string POST`/api/v3/enterprise/backup` ### Start a backup Registers and asynchronously starts a full backup. Returns immediately with the resolved backup name and an in-progress acknowledgement. Backups run asynchronously. Poll [List backups](#operation/GetEnterpriseBackup) or [Get a backup](#operation/GetEnterpriseBackupByName) to track progress. Requires an admin (operator) token. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `force`boolean Overwrite an existing backup of the same name. `name`string The backup name. Defaults to a generated name when omitted. `type`requiredstring The backup type. Only `full` backups are supported. Allowed:`full` Example request body ```json { "force": false, "name": "nightly-backup", "type": "full" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/backup" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"force":false,"name":"nightly-backup","type":"full"}' ``` #### Responses 202Accepted. The backup has started. `backup_name`requiredstring The resolved backup name. `status`requiredstring The status of a newly started backup. Always `in_progress`. Allowed:`in_progress` Example request body ```json { "backup_name": "nightly-backup", "status": "in_progress" } ``` 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 409Conflict. A backup with that name already exists and `force` is `false`, or a duplicate backup job is already in flight. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string DELETE`/api/v3/enterprise/backup` ### Cancel a running backup Cancels an in-progress backup job by name. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `name`requiredstring The name of the backup job to cancel. Example request body ```json { "name": "nightly-backup" } ``` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/backup" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"name":"nightly-backup"}' ``` #### Responses 200Success. The cancellation has been accepted. `backup_name`requiredstring The cancelled backup name. Example request body ```json { "backup_name": "nightly-backup" } ``` 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No backup is running with that name. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string GET`/api/v3/enterprise/backup/{name}` ### Get a backup Returns a single backup’s manifest. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `name`requiredstring The backup name. URL-encoded names are decoded by the server. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/backup/{name}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the backup manifest. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No backup exists with that name. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string DELETE`/api/v3/enterprise/backup/{name}` ### Delete a backup Deletes a backup by name. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `name`requiredstring The backup name. URL-encoded names are decoded by the server. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/backup/{name}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The backup has been deleted. `backup_name`requiredstring The deleted backup name. `type`requiredstring The backup type. Allowed:`full`, `incremental` Example request body ```json { "backup_name": "nightly-backup", "type": "full" } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No backup exists with that name. `data`object `error`string 409Conflict. The backup is in progress and cannot be deleted, or the backup chain is in a conflicting state. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string GET`/api/v3/enterprise/restore` ### List restores Lists every restore for the cluster, newest first, including in-flight and completed restores. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/restore" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of restores. `restores`requiredobject[] All restores, newest first. `backup_name`requiredstring The source backup name. `completed_at`string \ The completion timestamp. Omitted when the restore is not complete. `created_at`requiredstring \ When the restore started. `restore_id`requiredstring The restore handle. `status`requiredstring The status of a backup or restore. Allowed:`in_progress`, `completed`, `failed` `total_files`requiredinteger \ The number of files. `total_size_bytes`requiredinteger \ The total size in bytes. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 500Internal server error. Listing restores failed. `data`object `error`string 503Service unavailable. The restore engine is not present on this node. `data`object `error`string POST`/api/v3/enterprise/restore` ### Start a restore Registers and asynchronously starts a restore from a named backup. Returns a `restore_id` handle for tracking and cancellation. Restores run asynchronously. Poll [Get a restore](#operation/GetEnterpriseRestoreById) to track progress. Only one restore can run cluster-wide at a time; if a restore is already running, this endpoint returns `409 Conflict`. Requires an admin (operator) token. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `backup_name`requiredstring The name of the backup to restore from. Must not be empty. Example request body ```json { "backup_name": "nightly-backup" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/restore" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"backup_name":"nightly-backup"}' ``` #### Responses 202Accepted. The restore has started. `restore_id`requiredstring The handle used to track or cancel the restore. `status`requiredstring The status of a newly started restore. Always `in_progress`. Allowed:`in_progress` Example request body ```json { "restore_id": "01J8X9Z0ABCDEF", "status": "in_progress" } ``` 400Bad request. `backup_name` is empty. `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 409Conflict. A restore is already running. Only one restore can run at a time. `data`object `error`string 503Service unavailable. The restore engine is not present on this node, or the restore lease could not be acquired. `data`object `error`string GET`/api/v3/enterprise/restore/{id}` ### Get a restore Reports a restore’s state by `restore_id`. Reads the persisted manifest, so it works for in-flight and completed restores. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `id`requiredstring The `restore_id` returned when the restore started. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/restore/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the restore status. `backup_name`requiredstring The source backup name. `completed_at`string \ The completion timestamp. Null when the restore is not complete. `created_at`requiredstring \ When the restore started. `restore_id`requiredstring The restore handle. `status`requiredstring The status of a backup or restore. Allowed:`in_progress`, `completed`, `failed` `total_files`requiredinteger \ The number of files. `total_size_bytes`requiredinteger \ The total size in bytes. Example request body ```json { "backup_name": "nightly-backup", "completed_at": null, "created_at": "2025-06-16T03:00:00Z", "restore_id": "01J8X9Z0ABCDEF", "status": "in_progress", "total_files": 128, "total_size_bytes": 524288000 } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No restore exists with that ID. `data`object `error`string 503Service unavailable. The restore engine is not present on this node. `data`object `error`string DELETE`/api/v3/enterprise/restore/{id}` ### Cancel a running restore Cancels an in-flight restore by `restore_id`. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `id`requiredstring The `restore_id` returned when the restore started. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/restore/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204No content. The cancellation has been accepted. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No restore exists with that ID, the restore is already completed, or the restore engine is not present on this node. `data`object `error`string #### Related * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Bulk import Source: https://docs.influxdata.com/influxdb3/enterprise/api/bulk-import/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Upload Parquet files for bulk import into existing tables and list the status of bulk import jobs. Bulk import endpoints require the upgraded storage engine, enabled with the `--use-pacha-tree` flag, and a compactor node that runs the bulk import scheduler to complete the import. All endpoints require an admin (operator) token. GET`/api/v3/enterprise/import` ### List bulk import statuses Lists the status of bulk import jobs, for both uploads and object-store pulls. Requires a token with the `describe` action on at least one database, or an admin token. The list includes only jobs for databases the token can describe. A token that can’t describe any database gets `403`. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag). This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/import" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of bulk import statuses. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 500Internal server error. Object store or read failure. `data`object `error`string POST`/api/v3/enterprise/import` ### Import Parquet data Imports Parquet data into an existing table. Send the data one of two ways: * **Upload** (`multipart/form-data`): send one Parquet file in the request. * **Object-store pull** (`application/json`): name a `source` prefix, and the server imports every Parquet file under it. The source is either a prefix in the cluster’s own object store or an `s3://bucket/prefix`URL. The server reads another bucket with its own object-store credentials; the request carries none, so those credentials must be able to read the bucket. A server whose own object store isn’t S3 can’t read another bucket and returns `400`. The request returns after the import is staged. The compactor node imports the data asynchronously. A job moves through `queued`, `imported`, and `compacted`; rows become queryable once the job is `compacted`. Use [List bulk import statuses](#operation/GetEnterpriseImport) to follow it. #### Column types For generic Parquet files, a column that isn’t in `column_metadata` is imported as a field, typed from its Parquet type. This applies even when the target table already declares the column as a tag, so map every string tag column (for example, `{"column_mapping": {"host": "tag"}}`). Otherwise the import fails with`invalid column type for column 'host', expected iox::column_type::tag, got iox::column_type::field::string`. A string column with no mapping can also fail with `400` `Unable to infer data type for column`. #### Explicit schema databases In a database created with `schema_mode: explicit`, every column in the file must already be declared on the table. Otherwise the import is rejected and no job is created. Declare columns with`PATCH /api/v3/configure/table`. #### Permissions Requires a token with the `write` action on the target database (for example, `db:DATABASE_NAME:write`), or an admin token. For a token without that permission, including when the database doesn’t exist, the request returns `403`. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag); the compactor node runs the bulk import scheduler that completes the import. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `column_metadata`object Column metadata for a bulk import. Maps column names to InfluxDB column types to declare tag, field, and time columns. Example:`{"column_mapping":{"room":"tag","temp":"f64","time":"time"}}` `column_mapping`requiredobject A map of column name to InfluxDB column type. The JSON key accepts the aliases `column_mapping`, `schema`, and `column_metadata`. `database`requiredstring The target database name. The database must already exist. `source`requiredstring Where to read Parquet files: a prefix in the cluster’s own object store (for example, `staging/exports/`), or an `s3://bucket/prefix` URL. Every Parquet file under the prefix is imported. `table`requiredstring The target table name. The table must already exist. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/import" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "column_metadata": { "column_mapping": { "room": "tag", "temp": "f64", "time": "time" } }, "database": "DATABASE", "source": "SOURCE", "table": "TABLE" }' ``` #### Responses 200Success. The upload has been staged for import. `column_metadata`object Column metadata for a bulk import. Maps column names to InfluxDB column types to declare tag, field, and time columns. Example:`{"column_mapping":{"room":"tag","temp":"f64","time":"time"}}` `column_mapping`requiredobject A map of column name to InfluxDB column type. The JSON key accepts the aliases `column_mapping`, `schema`, and `column_metadata`. `completed_at`integer \ When the import completed, in nanoseconds since the Unix epoch. `created_at`requiredinteger \ When the uploaded file was ready for import, in nanoseconds since the Unix epoch. `db_id`requiredinteger The ID of the database resolved from the supplied name. `filename`requiredstring The filename of the uploaded Parquet data file. `iox_parquet`requiredboolean Whether the file was written by IOx (richer embedded metadata). `last_message`string The last error message if the job has errored. `last_updated_at`integer \ When this info last changed, in nanoseconds since the Unix epoch. `max_timestamp_ns`requiredinteger \ The maximum timestamp in the file (nanoseconds), from the time column’s Parquet statistics. `0` when the file has no statistics the server can read; the imported data isn’t affected. `min_timestamp_ns`requiredinteger \ The minimum timestamp in the file (nanoseconds), from the time column’s Parquet statistics. `0` when the file has no statistics the server can read; the imported data isn’t affected. `row_count`requiredinteger \ The row count from Parquet metadata. `size_bytes`requiredinteger \ The total size of the uploaded Parquet file in bytes. `started_at`integer \ When the import began, in nanoseconds since the Unix epoch. `status`requiredstring The status of a bulk import job. A successful job moves through `queued`, `imported`, and `compacted`; its rows become queryable once it’s `compacted`, not when it’s `imported`. Allowed:`queued`, `dispatched`, `imported`, `compacted`, `failed`, `terminal_failed` `table_id`requiredinteger The ID of the table resolved from the supplied name. `upload_uuid`requiredstring The UUID of the upload. 400Bad request. Possible reasons: missing multipart boundary; missing `file_bytes`, `database`, or `table`; unparseable multipart; invalid or empty Parquet; malformed `column_metadata`; column type or mapping mismatch; or a missing time column. `data`object `error`string 401Unauthorized access. `data`object `error`string 403Forbidden. The token doesn’t have the `write` action on the target database. A token without that permission also gets `403` when the database doesn’t exist (`Not authorized to import into database`). `data`object `error`string 404Not found. The target table does not exist (“Table `cpu` does not exist”). `data`object `error`string 500Internal server error. Object store, status write, or catalog failure. In 3.12, a file whose columns don’t match the table also returns `500` with `Could not modify catalog`: a column that an explicit schema database doesn’t declare, or an unmapped string column that the table declares as a tag. `data`object `error`string #### Related * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Cache distinct values Source: https://docs.influxdata.com/influxdb3/enterprise/api/cache-distinct-values/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) The Distinct Value Cache (DVC) lets you cache distinct values of one or more columns in a table, improving the performance of queries that return distinct tag and field values. The DVC is an in-memory cache that stores distinct values for specific columns in a table. When you create a DVC, you can specify what columns’ distinct values to cache, the maximum number of distinct value combinations to cache, and the maximum age of cached values. A DVC is associated with a table, which can have multiple DVCs. POST`/api/v3/configure/distinct_cache` ### Create distinct cache Creates a distinct cache for a table. #### Request bodyrequired Content-Type:`application/json` `columns`requiredstring[] `db`requiredstring `max_age`integer Optional maximum age in seconds. `max_cardinality`integer Optional maximum cardinality. `name`string Optional cache name. `node_spec`string or object Specifies which nodes in a multi-node InfluxDB 3 Enterprise cluster handle the cache or trigger. Accepts the string `all` or an object that lists specific node IDs. If omitted, defaults to `all`. Default:`all` Example:`"all"` One of:string Create the cache or trigger on all nodes in the cluster. Allowed:`all` Or:object Create the cache or trigger only on the specified nodes. `nodes`requiredstring[] Node IDs (the `--node-id` value each node was started with). Each node must exist and must run in `process` mode. `table`requiredstring Example request body ```json { "columns": [ "tag1", "tag2" ], "db": "mydb", "max_age": 3600, "max_cardinality": 1000, "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/distinct_cache" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"columns":["tag1","tag2"],"db":"mydb","max_age":3600,"max_cardinality":1000,"table":"mytable"}' ``` #### Responses 201Success. The distinct cache has been created. 400 Bad request. The server responds with status `400` if the request would overwrite an existing cache with a different configuration. 409Conflict. A distinct cache with this configuration already exists. DELETE`/api/v3/configure/distinct_cache` ### Delete distinct cache Deletes a distinct cache. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The name of the table containing the distinct cache. `name`requiredstring The name of the distinct cache to delete. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/distinct_cache?db=DB&table=TABLE&name=NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The distinct cache has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Cache not found. #### Related * [Manage the Distinct Value Cache](/influxdb3/enterprise/admin/distinct-value-cache/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Cache last value Source: https://docs.influxdata.com/influxdb3/enterprise/api/cache-last-value/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) The Last Value Cache (LVC) lets you cache the most recent values for specific fields in a table, improving the performance of queries that return the most recent value of a field for specific series or the last N values of a field. The LVC is an in-memory cache that stores the last N number of values for specific fields of series in a table. When you create an LVC, you can specify what fields to cache, what tags to use to identify each series, and the number of values to cache for each unique series. An LVC is associated with a table, which can have multiple LVCs. POST`/api/v3/configure/last_cache` ### Create last cache Creates a last cache for a table. #### Request bodyrequired Content-Type:`application/json` `count`integer Optional count. `db`requiredstring `key_columns`string[] Optional list of key columns. `name`string Optional cache name. `node_spec`string or object Specifies which nodes in a multi-node InfluxDB 3 Enterprise cluster handle the cache or trigger. Accepts the string `all` or an object that lists specific node IDs. If omitted, defaults to `all`. Default:`all` Example:`"all"` One of:string Create the cache or trigger on all nodes in the cluster. Allowed:`all` Or:object Create the cache or trigger only on the specified nodes. `nodes`requiredstring[] Node IDs (the `--node-id` value each node was started with). Each node must exist and must run in `process` mode. `table`requiredstring `ttl`integer Optional time-to-live in seconds. `value_columns`string[] Optional list of value columns. Example request body ```json { "count": 100, "db": "mydb", "key_columns": [ "tag1" ], "table": "mytable", "ttl": 3600, "value_columns": [ "field1" ] } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/last_cache" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"count":100,"db":"mydb","key_columns":["tag1"],"table":"mytable","ttl":3600,"value_columns":["field1"]}' ``` #### Responses 201Success. Last cache created. 400Bad request. A cache with this name already exists or the request is malformed. 401Unauthorized access. `data`object `error`string 404Cache not found. DELETE`/api/v3/configure/last_cache` ### Delete last cache Deletes a last cache. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The name of the table containing the last cache. `name`requiredstring The name of the last cache to delete. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/last_cache?db=DB&table=TABLE&name=NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The last cache has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Cache not found. #### Related * [Manage the Last Value Cache](/influxdb3/enterprise/admin/last-value-cache/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Data deletion Source: https://docs.influxdata.com/influxdb3/enterprise/api/data-deletion/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Manage row-delete requests. These endpoints are only available in InfluxDB 3 Enterprise. POST`/api/v3/row_delete_requests/{seq}/abort` ### Abort a pending row-delete request Aborts a pending row-delete request identified by its sequence number. Returns the updated request with a disposition of `aborted`. Requires a token or user with `delete` permission on the database that the row-delete request targets. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `seq`requiredinteger \ The sequence number of the row-delete request to abort. #### Request bodyrequired Content-Type:`application/json` `reason`requiredstring The reason for aborting the row-delete request. Example request body ```json { "reason": "Predicate was too broad" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/row_delete_requests/{seq}/abort" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"reason":"Predicate was too broad"}' ``` #### Responses 200Success. Returns the aborted row-delete request. `abort_reason`string The reason the request was aborted. Present only for aborted requests. `aborted_at`integer \ The Unix timestamp (seconds) when the request was aborted. Present only for aborted requests. `completed_at`integer \ The Unix timestamp (seconds) when the request completed. Present only for completed requests. `created_at`requiredinteger \ The Unix timestamp (seconds) when the request was created. `db_id`requiredinteger \ The numeric ID of the target database. `db_name`requiredstring The name of the target database. `db_name_stale`requiredboolean Whether the database name no longer matches the current catalog state for the stored database ID. `delete_predicate`requiredstring The predicate that selects rows to delete. `disposition`requiredstring The current disposition of the request: `pending`, `completed`, or `aborted`. `max_time`integer \ The maximum timestamp (nanoseconds) of the delete range. `min_time`integer \ The minimum timestamp (nanoseconds) of the delete range. `progress`object Progress tracking for a single row-delete request. `in_range`requiredobject Per-level progress counts for run sets that overlap the delete criteria. `l1`requiredobject Progress for a single compaction level. `l2`requiredobject Progress for a single compaction level. `l3`requiredobject Progress for a single compaction level. `l4`requiredobject Progress for a single compaction level. `out_of_range`requiredobject Progress counts for run sets outside the delete criteria that require only watermark updates. `processed`requiredinteger \ The number of out-of-range run sets processed. `total`requiredinteger \ The total number of out-of-range run sets. `sequence_id`requiredinteger \ The sequence number of the row-delete request. `sequence`requiredinteger \ The sequence number of the row-delete request. `table_id`requiredinteger \ The numeric ID of the target table. `table_name`requiredstring The name of the target table. `table_name_stale`requiredboolean Whether the table name no longer matches the current catalog state for the stored table ID. Example request body ```json { "abort_reason": "Predicate was too broad", "aborted_at": 1717000100, "created_at": 1717000000, "db_id": 1, "db_name": "sensors", "db_name_stale": false, "delete_predicate": "room = 'kitchen'", "disposition": "aborted", "sequence": 42, "table_id": 3, "table_name": "home", "table_name_stale": false } ``` 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No row-delete request exists with the specified sequence number. `data`object `error`string #### Related * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Database Source: https://docs.influxdata.com/influxdb3/enterprise/api/database/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Create, list, and delete databases in InfluxDB 3 Enterprise. GET`/api/v3/configure/database` ### List databases Retrieves a list of databases. #### Parameters ##### Query parameters `format`requiredstring `show_deleted`boolean Include soft-deleted databases in the response. By default, only active databases are returned. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/configure/database?format=FORMAT" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The response body contains the list of databases. `databases`string[] 400Bad request. 401Unauthorized access. `data`object `error`string 404Database not found. POST`/api/v3/configure/database` ### Create a database Creates a new database in the system. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring The database name. Database names cannot contain underscores (\_). Names must start and end with alphanumeric characters and can contain hyphens (-) in the middle. `retention_period`string The retention period for the database. Specifies how long data should be retained. Use duration format (for example, “1d”, “1h”, “30m”, “7d”). Example:`"7d"` `schema_mode`string How tables in the database get their schemas. Set at creation and can’t be changed. * `implicit` (default): writes create tables and columns. * `explicit`: tables and columns are created only through the configuration API, and writes that reference an undeclared table or column are rejected. Enterprise only; InfluxDB 3 Core returns `400 Bad Request`. Allowed:`implicit`, `explicit` Default:`implicit` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/database" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "db": "DB", "retention_period": "7d", "schema_mode": "implicit" }' ``` #### Responses 200Success. Database created. 400Bad request. 401Unauthorized access. `data`object `error`string 409Database already exists. PUT`/api/v3/configure/database` ### Update a database Updates database configuration, such as retention period. #### Request bodyrequired Content-Type:`application/json` `retention_period`string The retention period for the database. Specifies how long data should be retained. Use duration format (for example, “1d”, “1h”, “30m”, “7d”). Example:`"7d"` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/configure/database" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "retention_period": "7d" }' ``` #### Responses 200Success. The database has been updated. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database not found. DELETE`/api/v3/configure/database` ### Delete a database Soft deletes a database. The database is scheduled for deletion and unavailable for querying. Use the `hard_delete_at` parameter to schedule a hard deletion. Use the `data_only` parameter to delete data while preserving the database schema and resources. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `data_only`boolean Delete only data while preserving the database schema and all associated resources (tokens, triggers, last value caches, distinct value caches, processing engine configurations). When `false` (default), the entire database is deleted. `remove_tables`boolean Used with `data_only=true` to remove table resources (caches) while preserving database-level resources (tokens, triggers, processing engine configurations). Has no effect when `data_only=false`. `hard_delete_at`string \ Schedule the database for hard deletion at the specified time. If not provided, the database will be soft deleted. Use ISO 8601 date-time format (for example, “2025-12-31T23:59:59Z”). #### Deleting a database cannot be undone Deleting a database is a destructive action. Once a database is deleted, data stored in that database cannot be recovered. Also accepts special string values: * `now` — hard delete immediately * `never` — soft delete only (default behavior) * `default` — use the system default hard deletion time Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/database?db=DB" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Database deleted. 401Unauthorized access. `data`object `error`string 404Database not found. DELETE`/api/v3/configure/database/retention_period` ### Remove database retention period Removes the retention period from a database, setting it to infinite retention. #### Parameters ##### Query parameters `db`requiredstring The name of the database. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/database/retention_period?db=DB" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204Success. The database retention period has been removed. 401Unauthorized access. `data`object `error`string 404Database not found. POST`/api/v3/enterprise/configure/file_index` ### Create a file index Creates a file index for a database or table. A file index improves query performance by indexing data files based on specified columns, enabling the query engine to skip irrelevant files during query execution. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `columns`requiredstring[] The columns to use for the file index. `db`requiredstring The database name. `table`string The table name. If omitted, the file index applies to the database. Example request body ```json { "columns": [ "tag1", "tag2" ], "db": "mydb", "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/file_index" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"columns":["tag1","tag2"],"db":"mydb","table":"mytable"}' ``` #### Responses 200Success. The file index has been created. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database or table not found. DELETE`/api/v3/enterprise/configure/file_index` ### Delete a file index Deletes a file index from a database or table. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring The database name. `table`string The table name. If omitted, deletes the database-level file index. Example request body ```json { "db": "mydb", "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/file_index" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"db":"mydb","table":"mytable"}' ``` #### Responses 200Success. The file index has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database, table, or file index not found. #### Related * [Manage databases](/influxdb3/enterprise/admin/databases/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/api/enterprise/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Endpoints available only in InfluxDB 3 Enterprise. Operations carrying this tag also set `x-enterprise-only: true` for programmatic detection. POST`/api/v3/authorize` ### #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/authorize" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success POST`/api/v3/authorize/refresh` ### #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/authorize/refresh" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success POST`/api/v3/authorize/reset-password` ### #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/authorize/reset-password" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success GET`/api/v3/configure/query_concurrency_limit` ### Get the query concurrency limit Returns the current runtime limit on the maximum number of queries that can run concurrently. When the limit is set to the maximum allowed value, `maxConcurrentQueries` is omitted from the response. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/configure/query_concurrency_limit" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the current query concurrency limit. `max_concurrent_queries`integer The maximum number of queries that can run concurrently. Omitted when the limit is set to the maximum allowed value. Example request body ```json { "max_concurrent_queries": 32 } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string PUT`/api/v3/configure/query_concurrency_limit` ### Set the query concurrency limit Sets the runtime limit on the maximum number of queries that can run concurrently. Omit `max_concurrent_queries` (or set it to `null`) to restore the limit configured at server startup. Values below 16 are rejected unless `force=true` is set in the query string. A value of `0` is invalid, and values above the maximum allowed limit are rejected. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Query parameters `force`boolean Set to `true` to allow setting `max_concurrent_queries` to a value below the minimum of 16. #### Request bodyrequired Content-Type:`application/json` `max_concurrent_queries`integer The maximum number of queries that can run concurrently. Set to `null` (or omit) to restore the limit configured at server startup. Must be a positive integer; values below 16 require `force=true`. Example request body ```json { "max_concurrent_queries": 32 } ``` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/configure/query_concurrency_limit" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"max_concurrent_queries":32}' ``` #### Responses 204No content. The query concurrency limit was updated. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string DELETE`/api/v3/configure/query_concurrency_limit` ### Reset the query concurrency limit Resets the runtime query concurrency limit to the value configured at server startup. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/query_concurrency_limit" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204No content. The query concurrency limit was reset. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string PUT`/api/v3/configure/table` ### Update a table Updates a table’s retention period. This endpoint doesn’t add columns. Other properties in the request body, such as `tags` or `fields`, are ignored, and the request still returns`200`. To add columns, use `PATCH /api/v3/configure/table`. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring The name of the database containing the table. `retention_period`string The retention period for the table. Specifies how long data in this table should be retained. Use duration format (for example, “1d”, “1h”, “30m”, “7d”). Example:`"30d"` `table`requiredstring The name of the table to update. Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "db": "DB", "retention_period": "30d", "table": "TABLE" }' ``` #### Responses 200Success. The table has been updated. 400Bad request. 401Unauthorized access. `data`object `error`string 404Table not found. POST`/api/v3/configure/user` ### #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/user" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success GET`/api/v3/enterprise/backup` ### List backups Lists all full backups. Requires an admin (operator) token. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/backup" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of backups. `backups`requiredobject[] All full backups. `completed_at`string \ The completion timestamp. Omitted when the backup is not complete. `created_at`requiredstring \ The creation timestamp. `name`requiredstring The backup name. `status`requiredstring The status of a backup or restore. Allowed:`in_progress`, `completed`, `failed` `total_files`requiredinteger \ The number of files in the backup. `total_size_bytes`requiredinteger \ The total backup size in bytes. `type`requiredstring The backup type. Allowed:`full`, `incremental` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 500Internal server error. Listing backups failed. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string POST`/api/v3/enterprise/backup` ### Start a backup Registers and asynchronously starts a full backup. Returns immediately with the resolved backup name and an in-progress acknowledgement. Backups run asynchronously. Poll [List backups](#operation/GetEnterpriseBackup) or [Get a backup](#operation/GetEnterpriseBackupByName) to track progress. Requires an admin (operator) token. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `force`boolean Overwrite an existing backup of the same name. `name`string The backup name. Defaults to a generated name when omitted. `type`requiredstring The backup type. Only `full` backups are supported. Allowed:`full` Example request body ```json { "force": false, "name": "nightly-backup", "type": "full" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/backup" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"force":false,"name":"nightly-backup","type":"full"}' ``` #### Responses 202Accepted. The backup has started. `backup_name`requiredstring The resolved backup name. `status`requiredstring The status of a newly started backup. Always `in_progress`. Allowed:`in_progress` Example request body ```json { "backup_name": "nightly-backup", "status": "in_progress" } ``` 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 409Conflict. A backup with that name already exists and `force` is `false`, or a duplicate backup job is already in flight. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string DELETE`/api/v3/enterprise/backup` ### Cancel a running backup Cancels an in-progress backup job by name. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `name`requiredstring The name of the backup job to cancel. Example request body ```json { "name": "nightly-backup" } ``` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/backup" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"name":"nightly-backup"}' ``` #### Responses 200Success. The cancellation has been accepted. `backup_name`requiredstring The cancelled backup name. Example request body ```json { "backup_name": "nightly-backup" } ``` 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No backup is running with that name. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string GET`/api/v3/enterprise/backup/{name}` ### Get a backup Returns a single backup’s manifest. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `name`requiredstring The backup name. URL-encoded names are decoded by the server. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/backup/{name}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the backup manifest. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No backup exists with that name. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string DELETE`/api/v3/enterprise/backup/{name}` ### Delete a backup Deletes a backup by name. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `name`requiredstring The backup name. URL-encoded names are decoded by the server. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/backup/{name}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The backup has been deleted. `backup_name`requiredstring The deleted backup name. `type`requiredstring The backup type. Allowed:`full`, `incremental` Example request body ```json { "backup_name": "nightly-backup", "type": "full" } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No backup exists with that name. `data`object `error`string 409Conflict. The backup is in progress and cannot be deleted, or the backup chain is in a conflicting state. `data`object `error`string 503Service unavailable. The backup engine is not present on this node (not a compactor node). `data`object `error`string POST`/api/v3/enterprise/configure/file_index` ### Create a file index Creates a file index for a database or table. A file index improves query performance by indexing data files based on specified columns, enabling the query engine to skip irrelevant files during query execution. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `columns`requiredstring[] The columns to use for the file index. `db`requiredstring The database name. `table`string The table name. If omitted, the file index applies to the database. Example request body ```json { "columns": [ "tag1", "tag2" ], "db": "mydb", "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/file_index" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"columns":["tag1","tag2"],"db":"mydb","table":"mytable"}' ``` #### Responses 200Success. The file index has been created. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database or table not found. DELETE`/api/v3/enterprise/configure/file_index` ### Delete a file index Deletes a file index from a database or table. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring The database name. `table`string The table name. If omitted, deletes the database-level file index. Example request body ```json { "db": "mydb", "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/file_index" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"db":"mydb","table":"mytable"}' ``` #### Responses 200Success. The file index has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database, table, or file index not found. POST`/api/v3/enterprise/configure/node/remove` ### Permanently remove a stopped node Permanently removes a node that has already been marked as stopped, deleting its data from object storage once the compactor finishes reassigning its work. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 remove node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/) * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) #### Request bodyrequired Content-Type:`application/json` `force_finalize`boolean Force-finalize a node stuck in the `Stopping` state (for example, after a crash that never acknowledged the stop). This can delete unsnapshotted WAL data for that node — only use it when the node’s process is confirmed gone and you accept the data-loss risk. `node_id`requiredstring The ID of the node to remove. Example request body ```json { "node_id": "node-1" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/node/remove" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"node_id":"node-1"}' ``` #### Responses 200Success. The node has been removed. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 404Node not found. 409The node is not in a state that allows removal (for example, still running). POST`/api/v3/enterprise/configure/node/stop` ### Mark a node as stopped Marks a node as stopped in the catalog, freeing up the licensed cores it was using for other nodes. Use this endpoint after you have already stopped the physical instance (for example, using `kill` or stopping the container). This endpoint does not shut down the running process — you must stop the instance first. When the node is marked as stopped: 1. Licensed cores from the stopped node are freed for reuse 2. Other nodes in the cluster see the update after their catalog sync interval This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) #### Request bodyrequired Content-Type:`application/json` `node_id`requiredstring The ID of the node to mark as stopped. Example request body ```json { "node_id": "node-1" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/node/stop" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"node_id":"node-1"}' ``` #### Responses 200Success. The node has been marked as stopped. 401Unauthorized access. `data`object `error`string 404Node not found. GET`/api/v3/enterprise/configure/query_groups`Not operational ### List distributed query groups (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 show query\_groups](/influxdb3/enterprise/reference/cli/influxdb3/show/query_groups/) Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/configure/query_groups" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 401Unauthorized access. `data`object `error`string POST`/api/v3/enterprise/configure/query_groups`Not operational ### Create a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 create query\_group](/influxdb3/enterprise/reference/cli/influxdb3/create/query_group/) #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/query_groups" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 201Created. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string GET`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Retrieve a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 401Unauthorized access. `data`object `error`string 404Query group not found. PUT`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Replace a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update query\_group](/influxdb3/enterprise/reference/cli/influxdb3/update/query_group/) #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 404Query group not found. PATCH`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Update a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update query\_group](/influxdb3/enterprise/reference/cli/influxdb3/update/query_group/) #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request PATCH \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 404Query group not found. DELETE`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Delete a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 delete query\_group](/influxdb3/enterprise/reference/cli/influxdb3/delete/query_group/) #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The query group has been deleted. 401Unauthorized access. `data`object `error`string 404Query group not found. POST`/api/v3/enterprise/configure/table/retention_period` ### Set table retention period Sets or updates the retention period for a specific table. Use this endpoint to control how long data in a table is retained independently of the database-level retention period. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) #### Parameters ##### Query parameters `db`requiredstring The database name. `table`requiredstring The table name. `duration`requiredstring The retention period as a human-readable duration (for example, “30d”, “24h”, “1y”). Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/table/retention_period?db=DB&table=TABLE&duration=DURATION" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204Success. The table retention period has been set. 400Bad request. Invalid duration format. 401Unauthorized access. `data`object `error`string 404Database or table not found. DELETE`/api/v3/enterprise/configure/table/retention_period` ### Clear table retention period Removes the retention period from a specific table, reverting to the database-level retention period (or infinite retention if no database-level retention is set). This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) #### Parameters ##### Query parameters `db`requiredstring The database name. `table`requiredstring The table name. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/table/retention_period?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204Success. The table retention period has been cleared. 401Unauthorized access. `data`object `error`string 404Database or table not found. POST`/api/v3/enterprise/configure/token` ### Create a resource token Creates a resource (fine-grained permissions) token. A resource token is a token that has access to specific resources in the system. This endpoint is only available in InfluxDB 3 Enterprise. InfluxDB 3 Cloud supports a narrower set of database-scoped read and write tokens instead of resource tokens—see [Manage tokens in InfluxDB 3 Cloud](/influxdb3/cloud/admin/tokens/). InfluxDB 3 Core doesn’t support resource tokens. #### Request bodyrequired Content-Type:`application/json` `expiry_secs`integer Optional expiration time in seconds. `permissions`requiredobject[] List of permissions to grant to the token. `actions`requiredstring[] List of actions to grant. `resource_names`requiredstring[] List of resource names. Use “\*” for all resources. `resource_type`requiredstring The type of resource. Allowed:`system`, `db` `token_name`requiredstring The name for the resource token. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/token" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "expiry_secs": 0, "permissions": [ { "actions": [], "resource_names": [], "resource_type": "system" } ], "token_name": "TOKEN_NAME" }' ``` #### Responses 201Success. The resource token has been created. The response body contains the token string and metadata. `expiry_secs`integer The expiration time in seconds. `permissions`object[] `actions`string[] `resource_names`string[] List of resource names. Use “\*” for all resources. `resource_type`string Allowed:`system`, `db` `token_name`string Example request body ```json { "expiry_secs": 300000, "permissions": [ { "actions": [ "read" ], "resource_names": [ "*" ], "resource_type": "system" } ], "token_name": "All system information" } ``` 401Unauthorized access. `data`object `error`string GET`/api/v3/enterprise/import` ### List bulk import statuses Lists the status of bulk import jobs, for both uploads and object-store pulls. Requires a token with the `describe` action on at least one database, or an admin token. The list includes only jobs for databases the token can describe. A token that can’t describe any database gets `403`. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag). This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/import" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of bulk import statuses. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 500Internal server error. Object store or read failure. `data`object `error`string POST`/api/v3/enterprise/import` ### Import Parquet data Imports Parquet data into an existing table. Send the data one of two ways: * **Upload** (`multipart/form-data`): send one Parquet file in the request. * **Object-store pull** (`application/json`): name a `source` prefix, and the server imports every Parquet file under it. The source is either a prefix in the cluster’s own object store or an `s3://bucket/prefix`URL. The server reads another bucket with its own object-store credentials; the request carries none, so those credentials must be able to read the bucket. A server whose own object store isn’t S3 can’t read another bucket and returns `400`. The request returns after the import is staged. The compactor node imports the data asynchronously. A job moves through `queued`, `imported`, and `compacted`; rows become queryable once the job is `compacted`. Use [List bulk import statuses](#operation/GetEnterpriseImport) to follow it. #### Column types For generic Parquet files, a column that isn’t in `column_metadata` is imported as a field, typed from its Parquet type. This applies even when the target table already declares the column as a tag, so map every string tag column (for example, `{"column_mapping": {"host": "tag"}}`). Otherwise the import fails with`invalid column type for column 'host', expected iox::column_type::tag, got iox::column_type::field::string`. A string column with no mapping can also fail with `400` `Unable to infer data type for column`. #### Explicit schema databases In a database created with `schema_mode: explicit`, every column in the file must already be declared on the table. Otherwise the import is rejected and no job is created. Declare columns with`PATCH /api/v3/configure/table`. #### Permissions Requires a token with the `write` action on the target database (for example, `db:DATABASE_NAME:write`), or an admin token. For a token without that permission, including when the database doesn’t exist, the request returns `403`. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag); the compactor node runs the bulk import scheduler that completes the import. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `column_metadata`object Column metadata for a bulk import. Maps column names to InfluxDB column types to declare tag, field, and time columns. Example:`{"column_mapping":{"room":"tag","temp":"f64","time":"time"}}` `column_mapping`requiredobject A map of column name to InfluxDB column type. The JSON key accepts the aliases `column_mapping`, `schema`, and `column_metadata`. `database`requiredstring The target database name. The database must already exist. `source`requiredstring Where to read Parquet files: a prefix in the cluster’s own object store (for example, `staging/exports/`), or an `s3://bucket/prefix` URL. Every Parquet file under the prefix is imported. `table`requiredstring The target table name. The table must already exist. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/import" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "column_metadata": { "column_mapping": { "room": "tag", "temp": "f64", "time": "time" } }, "database": "DATABASE", "source": "SOURCE", "table": "TABLE" }' ``` #### Responses 200Success. The upload has been staged for import. `column_metadata`object Column metadata for a bulk import. Maps column names to InfluxDB column types to declare tag, field, and time columns. Example:`{"column_mapping":{"room":"tag","temp":"f64","time":"time"}}` `column_mapping`requiredobject A map of column name to InfluxDB column type. The JSON key accepts the aliases `column_mapping`, `schema`, and `column_metadata`. `completed_at`integer \ When the import completed, in nanoseconds since the Unix epoch. `created_at`requiredinteger \ When the uploaded file was ready for import, in nanoseconds since the Unix epoch. `db_id`requiredinteger The ID of the database resolved from the supplied name. `filename`requiredstring The filename of the uploaded Parquet data file. `iox_parquet`requiredboolean Whether the file was written by IOx (richer embedded metadata). `last_message`string The last error message if the job has errored. `last_updated_at`integer \ When this info last changed, in nanoseconds since the Unix epoch. `max_timestamp_ns`requiredinteger \ The maximum timestamp in the file (nanoseconds), from the time column’s Parquet statistics. `0` when the file has no statistics the server can read; the imported data isn’t affected. `min_timestamp_ns`requiredinteger \ The minimum timestamp in the file (nanoseconds), from the time column’s Parquet statistics. `0` when the file has no statistics the server can read; the imported data isn’t affected. `row_count`requiredinteger \ The row count from Parquet metadata. `size_bytes`requiredinteger \ The total size of the uploaded Parquet file in bytes. `started_at`integer \ When the import began, in nanoseconds since the Unix epoch. `status`requiredstring The status of a bulk import job. A successful job moves through `queued`, `imported`, and `compacted`; its rows become queryable once it’s `compacted`, not when it’s `imported`. Allowed:`queued`, `dispatched`, `imported`, `compacted`, `failed`, `terminal_failed` `table_id`requiredinteger The ID of the table resolved from the supplied name. `upload_uuid`requiredstring The UUID of the upload. 400Bad request. Possible reasons: missing multipart boundary; missing `file_bytes`, `database`, or `table`; unparseable multipart; invalid or empty Parquet; malformed `column_metadata`; column type or mapping mismatch; or a missing time column. `data`object `error`string 401Unauthorized access. `data`object `error`string 403Forbidden. The token doesn’t have the `write` action on the target database. A token without that permission also gets `403` when the database doesn’t exist (`Not authorized to import into database`). `data`object `error`string 404Not found. The target table does not exist (“Table `cpu` does not exist”). `data`object `error`string 500Internal server error. Object store, status write, or catalog failure. In 3.12, a file whose columns don’t match the table also returns `500` with `Could not modify catalog`: a column that an explicit schema database doesn’t declare, or an unmapped string column that the table declares as a tag. `data`object `error`string GET`/api/v3/enterprise/restore` ### List restores Lists every restore for the cluster, newest first, including in-flight and completed restores. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/restore" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of restores. `restores`requiredobject[] All restores, newest first. `backup_name`requiredstring The source backup name. `completed_at`string \ The completion timestamp. Omitted when the restore is not complete. `created_at`requiredstring \ When the restore started. `restore_id`requiredstring The restore handle. `status`requiredstring The status of a backup or restore. Allowed:`in_progress`, `completed`, `failed` `total_files`requiredinteger \ The number of files. `total_size_bytes`requiredinteger \ The total size in bytes. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 500Internal server error. Listing restores failed. `data`object `error`string 503Service unavailable. The restore engine is not present on this node. `data`object `error`string POST`/api/v3/enterprise/restore` ### Start a restore Registers and asynchronously starts a restore from a named backup. Returns a `restore_id` handle for tracking and cancellation. Restores run asynchronously. Poll [Get a restore](#operation/GetEnterpriseRestoreById) to track progress. Only one restore can run cluster-wide at a time; if a restore is already running, this endpoint returns `409 Conflict`. Requires an admin (operator) token. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `backup_name`requiredstring The name of the backup to restore from. Must not be empty. Example request body ```json { "backup_name": "nightly-backup" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/restore" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"backup_name":"nightly-backup"}' ``` #### Responses 202Accepted. The restore has started. `restore_id`requiredstring The handle used to track or cancel the restore. `status`requiredstring The status of a newly started restore. Always `in_progress`. Allowed:`in_progress` Example request body ```json { "restore_id": "01J8X9Z0ABCDEF", "status": "in_progress" } ``` 400Bad request. `backup_name` is empty. `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 409Conflict. A restore is already running. Only one restore can run at a time. `data`object `error`string 503Service unavailable. The restore engine is not present on this node, or the restore lease could not be acquired. `data`object `error`string GET`/api/v3/enterprise/restore/{id}` ### Get a restore Reports a restore’s state by `restore_id`. Reads the persisted manifest, so it works for in-flight and completed restores. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node; otherwise the endpoint returns `503 Service Unavailable`. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `id`requiredstring The `restore_id` returned when the restore started. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/restore/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the restore status. `backup_name`requiredstring The source backup name. `completed_at`string \ The completion timestamp. Null when the restore is not complete. `created_at`requiredstring \ When the restore started. `restore_id`requiredstring The restore handle. `status`requiredstring The status of a backup or restore. Allowed:`in_progress`, `completed`, `failed` `total_files`requiredinteger \ The number of files. `total_size_bytes`requiredinteger \ The total size in bytes. Example request body ```json { "backup_name": "nightly-backup", "completed_at": null, "created_at": "2025-06-16T03:00:00Z", "restore_id": "01J8X9Z0ABCDEF", "status": "in_progress", "total_files": 128, "total_size_bytes": 524288000 } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No restore exists with that ID. `data`object `error`string 503Service unavailable. The restore engine is not present on this node. `data`object `error`string DELETE`/api/v3/enterprise/restore/{id}` ### Cancel a running restore Cancels an in-flight restore by `restore_id`. Requires an admin (operator) token or admin user. Requires the upgraded storage engine (enabled with the `--use-pacha-tree` flag) present on a compactor node. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `id`requiredstring The `restore_id` returned when the restore started. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/restore/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204No content. The cancellation has been accepted. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No restore exists with that ID, the restore is already completed, or the restore engine is not present on this node. `data`object `error`string GET`/api/v3/enterprise/upgrade/parquet_cleanup` ### Get Parquet cleanup status Returns the current or most recent Parquet cleanup state for the cluster. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 manage cleanup-parquet](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/) Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/upgrade/parquet_cleanup" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the current Parquet cleanup state. `bytes_deleted`requiredinteger \ Total size, in bytes, of the deleted files. Always `0` in `dry_run` mode. `bytes_matched`requiredinteger \ Total size, in bytes, of the matched files. `completed_at_ns`integer \ When the cleanup finished, in nanoseconds since the Unix epoch. `files_deleted`requiredinteger \ Number of files deleted so far. Always `0` in `dry_run` mode. `files_failed`requiredinteger \ Number of files the cleanup failed to delete. `files_matched`requiredinteger \ Number of files identified as Parquet-era data to delete. `files_scanned`requiredinteger \ Number of files the cleanup has scanned so far. `last_message`string The most recent human-readable status message, if any. `mode`requiredstring Whether this cleanup deletes data or only reports what it would delete. Allowed:`delete`, `dry_run` `requested_at_ns`integer \ When the cleanup was requested, in nanoseconds since the Unix epoch. `requested_by`string The node that accepted the cleanup request, if recorded. `started_at_ns`integer \ When the cleanup started running, in nanoseconds since the Unix epoch. `status`requiredstring The cleanup’s current status. Allowed:`requested`, `running`, `completed`, `failed` `updated_at_ns`integer \ When this status was last updated, in nanoseconds since the Unix epoch. Example request body ```json { "bytes_deleted": 4294967296, "bytes_matched": 4294967296, "completed_at_ns": 1735689900000000000, "files_deleted": 812, "files_failed": 0, "files_matched": 812, "files_scanned": 1024, "last_message": null, "mode": "delete", "requested_at_ns": 1735689600000000000, "requested_by": "node-1", "started_at_ns": 1735689601000000000, "status": "completed", "updated_at_ns": 1735689900000000000 } ``` 401Unauthorized access. `data`object `error`string 404No Parquet cleanup has ever been requested on this cluster. POST`/api/v3/enterprise/upgrade/parquet_cleanup` ### Request Parquet cleanup after a storage engine upgrade Requests removal of the Parquet-era data left behind by a completed Parquet-to-PachaTree storage engine upgrade. Any node in the cluster accepts the request. The compactor node executes it. Requires the cluster to be fully on the upgraded (PachaTree) storage engine, with the Parquet-to-PachaTree upgrade already complete. Use `dry_run` to count and size the data a real cleanup would delete without deleting anything. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 manage cleanup-parquet](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/) #### Request body Content-Type:`application/json` `dry_run`boolean When `true`, only count and size the Parquet-era data a real cleanup would delete. Delete nothing. Example request body ```json { "dry_run": false } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/upgrade/parquet_cleanup" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"dry_run":false}' ``` #### Responses 202Success. The cleanup started, is already running, or already completed. `state`requiredstring The outcome of this cleanup request. Allowed:`started`, `already_running`, `already_completed` `status`requiredobject Point-in-time view of the durable Parquet cleanup state. Returned by `GET /api/v3/enterprise/upgrade/parquet_cleanup` and embedded in the POST response. Example:`{"bytes_deleted":4294967296,"bytes_matched":4294967296,"completed_at_ns":1735689900000000000,"files_deleted":812,"files_failed":0,"files_matched":812,"files_scanned":1024,"last_message":null,"mode":"delete","requested_at_ns":1735689600000000000,"requested_by":"node-1","started_at_ns":1735689601000000000,"status":"completed","updated_at_ns":1735689900000000000}` `bytes_deleted`requiredinteger \ Total size, in bytes, of the deleted files. Always `0` in `dry_run` mode. `bytes_matched`requiredinteger \ Total size, in bytes, of the matched files. `completed_at_ns`integer \ When the cleanup finished, in nanoseconds since the Unix epoch. `files_deleted`requiredinteger \ Number of files deleted so far. Always `0` in `dry_run` mode. `files_failed`requiredinteger \ Number of files the cleanup failed to delete. `files_matched`requiredinteger \ Number of files identified as Parquet-era data to delete. `files_scanned`requiredinteger \ Number of files the cleanup has scanned so far. `last_message`string The most recent human-readable status message, if any. `mode`requiredstring Whether this cleanup deletes data or only reports what it would delete. Allowed:`delete`, `dry_run` `requested_at_ns`integer \ When the cleanup was requested, in nanoseconds since the Unix epoch. `requested_by`string The node that accepted the cleanup request, if recorded. `started_at_ns`integer \ When the cleanup started running, in nanoseconds since the Unix epoch. `status`requiredstring The cleanup’s current status. Allowed:`requested`, `running`, `completed`, `failed` `updated_at_ns`integer \ When this status was last updated, in nanoseconds since the Unix epoch. 401Unauthorized access. `data`object `error`string 409The cluster isn’t on the upgraded storage engine, no Parquet-to-PachaTree upgrade was ever recorded, the upgrade hasn’t completed, or a cleanup is already active in a different mode (dry run vs. delete). POST`/api/v3/enterprise/upgrade/retry_parquet_to_pacha_tree` ### Retry a failed Parquet-to-PachaTree upgrade Resets a Parquet-to-PachaTree storage engine upgrade that latched to a `failed` state, so it resumes the next time a compactor node starts with `--upgrade-pacha-tree`. Re-checks every previously failed upgrade source against the catalog and object store: sources that are still readable are requeued, sources whose table or database was dropped are recorded as skipped, and sources that can’t be read back are left unresolved for the next retry. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 manage retry-upgrade-to-pacha-tree](/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/) Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/upgrade/retry_parquet_to_pacha_tree" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a summary of what the retry changed. `entries_already_converted`requiredinteger \ Entries whose sequence already held converted data. Recorded as imported instead of skipped. `entries_already_skipped`requiredinteger \ Terminal failures that were already recorded as skips. `entries_inspected`requiredinteger \ Terminal failures the retry re-checked against the catalog and object store. `entries_requeued`requiredinteger \ Entries returned to the queue because their source is still readable. `entries_skipped_source_missing`requiredinteger \ Entries skipped because their source object no longer exists. `entries_skipped_table_dropped`requiredinteger \ Entries skipped because their table was dropped. `entries_unresolved`requiredinteger \ Entries left untouched because the object holding their sequence couldn’t be read back. Run this endpoint again to re-attempt them. `message`requiredstring What to do next. `upgrade_status`requiredstring The status the enterprise upgrade state was left in after the retry. Example request body ```json { "entries_already_converted": 0, "entries_already_skipped": 0, "entries_inspected": 3, "entries_requeued": 2, "entries_skipped_source_missing": 1, "entries_skipped_table_dropped": 0, "entries_unresolved": 0, "message": "Migration state reset. Restart the compactor node with --upgrade-pacha-tree to resume the upgrade.", "upgrade_status": "upgrading" } ``` 401Unauthorized access. `data`object `error`string 409The upgrade isn’t in a state this request can act on (for example, it isn’t `failed`), or another request already reset it. GET`/api/v3/export/databases` ### List databases available for export Returns a list of databases that have compacted data available for Parquet export. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/databases" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a list of database names. 401Unauthorized access. `data`object `error`string GET`/api/v3/export/tables` ### List tables available for export Returns a list of tables in a database that have compacted data available for Parquet export. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > #### Parameters ##### Query parameters `db`requiredstring The name of the database. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/tables?db=DB" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a list of table names. 401Unauthorized access. `data`object `error`string 404Database not found. GET`/api/v3/export/window_data` ### Export window data as Parquet Downloads compacted data for the specified windows as a tar archive containing Parquet files. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The table name. `windows`string Comma-separated list of window dates to export (for example, `2026-01-15,2026-01-16`). If omitted, exports all available windows. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/window_data?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a tar archive containing Parquet files. 401Unauthorized access. `data`object `error`string 404Database, table, or window not found. GET`/api/v3/export/windows` ### List compacted windows for a table Returns a list of compacted 24-hour UTC windows for a table. Each window represents a time range of compacted data that can be exported as Parquet. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The table name. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/windows?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a list of compacted 24-hour windows. 401Unauthorized access. `data`object `error`string 404Database or table not found. POST`/api/v3/isAuthorized` ### Check whether the caller is authorized for an action Reports whether the calling user may perform the single action named in the request body. A denied action returns `200` with `authorized: false`. A `403` response means the caller can’t use this endpoint. #### Request bodyrequired Content-Type:`application/json` `createAdminToken`boolean `createDatabase`boolean `createRole`boolean `createToken`boolean `createUser`boolean `deleteAdminToken`boolean `deleteDatabase`object `identifier`string Resource name. Required when `scope` is `identifier`; not allowed when `scope` is `all`. `scope`requiredstring Allowed:`all`, `identifier` `deleteRole`boolean `deleteToken`boolean `deleteUser`boolean `describeDatabase`object `identifier`string Resource name. Required when `scope` is `identifier`; not allowed when `scope` is `all`. `scope`requiredstring Allowed:`all`, `identifier` `readDatabase`object `identifier`string Resource name. Required when `scope` is `identifier`; not allowed when `scope` is `all`. `scope`requiredstring Allowed:`all`, `identifier` `readRole`boolean `readSystem`object `identifier`string Resource name. Required when `scope` is `identifier`; not allowed when `scope` is `all`. `scope`requiredstring Allowed:`all`, `identifier` `readToken`boolean `readUser`boolean `updateRole`boolean `updateUser`boolean `writeDatabase`object `identifier`string Resource name. Required when `scope` is `identifier`; not allowed when `scope` is `all`. `scope`requiredstring Allowed:`all`, `identifier` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/isAuthorized" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "createAdminToken": false, "createDatabase": false, "createRole": false, "createToken": false, "createUser": false, "deleteAdminToken": false, "deleteDatabase": { "identifier": "IDENTIFIER", "scope": "all" }, "deleteRole": false, "deleteToken": false, "deleteUser": false, "describeDatabase": { "identifier": "IDENTIFIER", "scope": "all" }, "readDatabase": { "identifier": "IDENTIFIER", "scope": "all" }, "readRole": false, "readSystem": { "identifier": "IDENTIFIER", "scope": "all" }, "readToken": false, "readUser": false, "updateRole": false, "updateUser": false, "writeDatabase": { "identifier": "IDENTIFIER", "scope": "all" } }' ``` #### Responses 200The check was evaluated.`authorized` is `false` when the caller isn’t permitted to perform the action. `authorized`requiredboolean 400Bad request. The body must name exactly one action; boolean actions must be `true`. 401Unauthorized access. `data`object `error`string 403Forbidden. Only user subjects can call this endpoint; token subjects receive `403`. GET`/api/v3/loadcap/profiles` ### List load capture profiles Returns load capture profiles stored by a query-capable Enterprise node that has the upgraded storage engine and an explicit `--mode` setting that includes `query`. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Load capture profiles. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404Load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}` ### Retrieve load capture profile details Returns file counts, total size, and an anonymized catalog summary for a load capture profile. The response is the same as the profile preview. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Load capture profile details. `catalog_summary`object Anonymized catalog snapshot taken when the capture started, including databases, tables, columns, and node metadata. Null for `query`-only captures. `file_count`integer Total number of files in the profile archive. `profile_id`string \ Identifier of the load capture profile. `query_count`integer Number of captured queries in the profile. `total_size_bytes`integer Total size of the profile archive contents, in bytes. `wal_file_count`integer Number of anonymized write-ahead log (`.pt`) files in the profile. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. DELETE`/api/v3/loadcap/profiles/{profile_id}` ### Delete a load capture profile Deletes a load capture profile and its stored artifacts from object storage. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200The profile was deleted. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. 409The profile is still capturing. Wait for it to finish (or its status to catch up to `complete`), then retry. GET`/api/v3/loadcap/profiles/{profile_id}/download` ### Download a load capture profile Downloads the load capture profile as a gzipped tar archive containing the anonymized capture artifacts—`catalog.json` and `.pt` write-ahead log files for `write` and `both` captures, and `queries.json.zst` for `query` and `both` captures. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/download" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200The profile archive. The response `Content-Type` is `application/gzip` and `Content-Disposition` provides the `{profile_id}.tar.gz` filename. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}/files` ### List files in a load capture profile Returns the name, size, and type of each file stored in a load capture profile. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/files" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Files in the load capture profile. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}/files/{filename}` ### Download a single file from a load capture profile Downloads one anonymized file from a load capture profile—for example, `catalog.json`, `queries.json.zst`, or a `.pt` write-ahead log file. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/files/{filename}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200The file contents. The response `Content-Type` is `application/octet-stream` and `Content-Disposition` provides the filename. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile or file does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}/preview` ### Preview a load capture profile Returns file counts, total size, and an anonymized catalog summary for a load capture profile without downloading the archive. For `query`-only captures, `catalog_summary` is null. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/preview" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Load capture profile preview. `catalog_summary`object Anonymized catalog snapshot taken when the capture started, including databases, tables, columns, and node metadata. Null for `query`-only captures. `file_count`integer Total number of files in the profile archive. `profile_id`string \ Identifier of the load capture profile. `query_count`integer Number of captured queries in the profile. `total_size_bytes`integer Total size of the profile archive contents, in bytes. `wal_file_count`integer Number of anonymized write-ahead log (`.pt`) files in the profile. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. POST`/api/v3/loadcap/start` ### Start a load capture Starts a timed capture of write requests, query requests, or both on a query-capable Enterprise node that has the upgraded storage engine and an explicit `--mode` setting that includes `query`. Only one capture can run at a time on a node. This endpoint requires an admin token. #### Request bodyrequired Content-Type:`application/json` `duration`requiredstring Capture duration using seconds, minutes, or hours. The maximum duration is one hour. Example:`"5m"` `type`requiredstring The request types to capture. Allowed:`write`, `query`, `both` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/loadcap/start" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "duration": "5m", "type": "write" }' ``` #### Responses 200Load capture started. `profile_id`requiredstring \ Identifier of the new load capture profile. 400Invalid capture type or duration. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404Load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. 409A load capture is already running on this node. GET`/api/v3/roles` ### Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/roles" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success GET`/api/v3/roles_by_name/{roleName}` ### Get a role by name Returns the role with the specified name. Requires a token or user with permission to read roles. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `roleName`requiredstring The name of the role to retrieve. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/roles_by_name/{roleName}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the matching role. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the role was created. `description`string The role description. `id`requiredinteger \ The numeric ID of the role. `isRequiredRole`requiredboolean Whether the role is a required role. `name`requiredstring The role name. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the role was last updated. Example request body ```json { "createdAt": 1717000000, "id": 1, "isRequiredRole": false, "name": "read-write", "updatedAt": 1717000000 } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No role exists with the specified name. `data`object `error`string GET`/api/v3/roles/{id}/permissions` ### Get a role's permissions Returns the permissions granted to the role with the specified ID. Requires a token or user with permission to read roles. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The ID of the role. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/roles/{id}/permissions" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the role’s permissions. `permissions`requiredobject[] The permissions granted to the role. `actions`requiredstring[] The granted actions (snake\_case) for this resource type. `allResources`boolean Database permissions only. `true` when the permission applies to all databases. `identifiers`string[] Database permissions only. Specific database names the permission targets. Empty when `allResources` is `true`. `resourceType`requiredstring The resource category the permission applies to. Allowed:`database`, `token`, `user`, `role`, `admin_token` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No role exists with the specified ID. `data`object `error`string GET`/api/v3/row_delete_requests` ### #### Parameters ##### Query parameters `db`string `table`string `disposition`string `min_sequence`integer `max_sequence`integer `include_progress`boolean Default: (default) Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/row_delete_requests" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success POST`/api/v3/row_delete_requests` ### Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/row_delete_requests" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success POST`/api/v3/row_delete_requests/{seq}/abort` ### Abort a pending row-delete request Aborts a pending row-delete request identified by its sequence number. Returns the updated request with a disposition of `aborted`. Requires a token or user with `delete` permission on the database that the row-delete request targets. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `seq`requiredinteger \ The sequence number of the row-delete request to abort. #### Request bodyrequired Content-Type:`application/json` `reason`requiredstring The reason for aborting the row-delete request. Example request body ```json { "reason": "Predicate was too broad" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/row_delete_requests/{seq}/abort" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"reason":"Predicate was too broad"}' ``` #### Responses 200Success. Returns the aborted row-delete request. `abort_reason`string The reason the request was aborted. Present only for aborted requests. `aborted_at`integer \ The Unix timestamp (seconds) when the request was aborted. Present only for aborted requests. `completed_at`integer \ The Unix timestamp (seconds) when the request completed. Present only for completed requests. `created_at`requiredinteger \ The Unix timestamp (seconds) when the request was created. `db_id`requiredinteger \ The numeric ID of the target database. `db_name`requiredstring The name of the target database. `db_name_stale`requiredboolean Whether the database name no longer matches the current catalog state for the stored database ID. `delete_predicate`requiredstring The predicate that selects rows to delete. `disposition`requiredstring The current disposition of the request: `pending`, `completed`, or `aborted`. `max_time`integer \ The maximum timestamp (nanoseconds) of the delete range. `min_time`integer \ The minimum timestamp (nanoseconds) of the delete range. `progress`object Progress tracking for a single row-delete request. `in_range`requiredobject Per-level progress counts for run sets that overlap the delete criteria. `l1`requiredobject Progress for a single compaction level. `l2`requiredobject Progress for a single compaction level. `l3`requiredobject Progress for a single compaction level. `l4`requiredobject Progress for a single compaction level. `out_of_range`requiredobject Progress counts for run sets outside the delete criteria that require only watermark updates. `processed`requiredinteger \ The number of out-of-range run sets processed. `total`requiredinteger \ The total number of out-of-range run sets. `sequence_id`requiredinteger \ The sequence number of the row-delete request. `sequence`requiredinteger \ The sequence number of the row-delete request. `table_id`requiredinteger \ The numeric ID of the target table. `table_name`requiredstring The name of the target table. `table_name_stale`requiredboolean Whether the table name no longer matches the current catalog state for the stored table ID. Example request body ```json { "abort_reason": "Predicate was too broad", "aborted_at": 1717000100, "created_at": 1717000000, "db_id": 1, "db_name": "sensors", "db_name_stale": false, "delete_predicate": "room = 'kitchen'", "disposition": "aborted", "sequence": 42, "table_id": 3, "table_name": "home", "table_name_stale": false } ``` 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No row-delete request exists with the specified sequence number. `data`object `error`string GET`/api/v3/users` ### Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/users" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success POST`/api/v3/users` ### #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/users" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success GET`/api/v3/users_by_oauth_id/{oauth_id}` ### Get a user by OAuth identity-provider ID Returns the user associated with the specified OAuth identity-provider ID. Requires the server to be configured with an OAuth validator. Requires a token or user with permission to read users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `oauth_id`requiredstring The OAuth identity-provider ID of the user. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/users_by_oauth_id/{oauth_id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the matching user. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the user was created. `deletedAt`integer \ The Unix timestamp (seconds) when the user was deleted. Present only for deleted users. `displayName`string The user’s display name. `oauthId`string The user’s OAuth identity-provider ID. Present when the user has an OAuth login identity. `operatorToken`string The operator token. Returned only when creating the initial user. `requiresPasswordReset`boolean Whether the user must reset their password on next login. Present when the user has a username and password login identity. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the user was last updated. `userId`requiredinteger \ The numeric ID of the user. `username`string The user’s username. Present when the user has a username and password login identity. Example request body ```json { "createdAt": 1717000000, "displayName": "Jane Smith", "requiresPasswordReset": false, "updatedAt": 1717000000, "userId": 1, "username": "jsmith" } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user matches the specified OAuth ID, or OAuth is not configured on the server. `data`object `error`string GET`/api/v3/users/{id}` ### Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/users/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success POST`/api/v3/users/{id}/require-password-reset` ### Require a user to reset their password on next login Marks the specified user as requiring a password reset. The user must set a new password before they are able to authenticate and receive a token. Requires a token or user with permission to update users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The numeric ID of the user. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/users/{id}/require-password-reset" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the user with `requiresPasswordReset` set to `true`. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the user was created. `deletedAt`integer \ The Unix timestamp (seconds) when the user was deleted. Present only for deleted users. `displayName`string The user’s display name. `oauthId`string The user’s OAuth identity-provider ID. Present when the user has an OAuth login identity. `operatorToken`string The operator token. Returned only when creating the initial user. `requiresPasswordReset`boolean Whether the user must reset their password on next login. Present when the user has a username and password login identity. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the user was last updated. `userId`requiredinteger \ The numeric ID of the user. `username`string The user’s username. Present when the user has a username and password login identity. Example request body ```json { "createdAt": 1717000000, "displayName": "Jane Smith", "requiresPasswordReset": false, "updatedAt": 1717000000, "userId": 1, "username": "jsmith" } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user exists with the specified ID. `data`object `error`string GET`/api/v3/users/{id}/roles` ### List all roles assigned to a user Lists all roles assigned to the specified user. Requires a token or user with permission to read users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The numeric ID of the user. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/users/{id}/roles" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of roles assigned to the user. `items`requiredobject[] The roles assigned to the user. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the role was created. `description`string The role description. `id`requiredinteger \ The numeric ID of the role. `isRequiredRole`requiredboolean Whether the role is a required role. `name`requiredstring The role name. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the role was last updated. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user exists with the specified ID. `data`object `error`string PUT`/api/v3/users/{id}/roles` ### Replace all roles assigned to a user Replaces all role assignments for the specified user with the provided list of role IDs. Roles not included in the request are removed from the user. Requires a token or user with permission to update users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The numeric ID of the user. #### Request bodyrequired Content-Type:`application/json` `roleIds`requiredinteger[] The complete set of role IDs to assign to the user. Example request body ```json { "roleIds": [ 1, 2 ] } ``` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/users/{id}/roles" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"roleIds":[1,2]}' ``` #### Responses 200Success. Returns the user’s updated role assignments. `items`requiredobject[] The roles assigned to the user. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the role was created. `description`string The role description. `id`requiredinteger \ The numeric ID of the role. `isRequiredRole`requiredboolean Whether the role is a required role. `name`requiredstring The role name. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the role was last updated. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user exists with the specified ID. `data`object `error`string GET`/ready` ### Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/ready" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success #### Related * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Export data Source: https://docs.influxdata.com/influxdb3/enterprise/api/export-data/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Export data from InfluxDB 3 Enterprise databases. List databases, tables, export windows, and retrieve window data for data migration and backup workflows. GET`/api/v3/export/databases` ### List databases available for export Returns a list of databases that have compacted data available for Parquet export. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/databases" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a list of database names. 401Unauthorized access. `data`object `error`string GET`/api/v3/export/tables` ### List tables available for export Returns a list of tables in a database that have compacted data available for Parquet export. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > #### Parameters ##### Query parameters `db`requiredstring The name of the database. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/tables?db=DB" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a list of table names. 401Unauthorized access. `data`object `error`string 404Database not found. GET`/api/v3/export/window_data` ### Export window data as Parquet Downloads compacted data for the specified windows as a tar archive containing Parquet files. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The table name. `windows`string Comma-separated list of window dates to export (for example, `2026-01-15,2026-01-16`). If omitted, exports all available windows. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/window_data?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a tar archive containing Parquet files. 401Unauthorized access. `data`object `error`string 404Database, table, or window not found. GET`/api/v3/export/windows` ### List compacted windows for a table Returns a list of compacted 24-hour UTC windows for a table. Each window represents a time range of compacted data that can be exported as Parquet. > > > **Requires the upgraded storage engine**: This endpoint requires the[upgraded storage engine](/influxdb3/enterprise/performance-preview/), which is the default for > new clusters. On clusters created with 3.10 or earlier, run the storage engine upgrade with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). > > #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The table name. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/export/windows?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a list of compacted 24-hour windows. 401Unauthorized access. `data`object `error`string 404Database or table not found. #### Related * [Export data](/influxdb3/enterprise/admin/export/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Headers and parameters Source: https://docs.influxdata.com/influxdb3/enterprise/api/headers-and-parameters/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Most InfluxDB API endpoints require parameters in the request–for example, specifying the database to use. ### Common parameters The following table shows common parameters used by many InfluxDB API endpoints. Many endpoints may require other parameters in the query string or in the request body that perform functions specific to those endpoints. |Query parameter|Value type| Description | |---------------|----------|-----------------| | `db` | string |The database name| InfluxDB HTTP API endpoints use standard HTTP request and response headers. The following table shows common headers used by many InfluxDB API endpoints. Some endpoints may use other headers that perform functions more specific to those endpoints–for example, the write endpoints accept the `Content-Encoding` header to indicate that line protocol is compressed in the request body. | Header |Value type| Description | |----------------|----------|------------------------------------------------| | `Accept` | string |The content type that the client can understand.| |`Authorization` | string | The authorization scheme and credential. | |`Content-Length`| integer | The size of the entity-body, in bytes. | | `Content-Type` | string | The format of the data in the request body. | | Query parameter | Value type | Description | | --- | --- | --- | | Query parameter | Value type | Description | | db | string | The database name | | Header | Value type | Description | | --- | --- | --- | | Header | Value type | Description | | Accept | string | The content type that the client can understand. | | Authorization | string | The authorization scheme and credential. | | Content-Length | integer | The size of the entity-body, in bytes. | | Content-Type | string | The format of the data in the request body. | --- # Load capture Source: https://docs.influxdata.com/influxdb3/enterprise/api/load-capture/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Capture an anonymized profile of write requests, query requests, or both on a query-capable node. GET`/api/v3/loadcap/profiles` ### List load capture profiles Returns load capture profiles stored by a query-capable Enterprise node that has the upgraded storage engine and an explicit `--mode` setting that includes `query`. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Load capture profiles. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404Load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}` ### Retrieve load capture profile details Returns file counts, total size, and an anonymized catalog summary for a load capture profile. The response is the same as the profile preview. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Load capture profile details. `catalog_summary`object Anonymized catalog snapshot taken when the capture started, including databases, tables, columns, and node metadata. Null for `query`-only captures. `file_count`integer Total number of files in the profile archive. `profile_id`string \ Identifier of the load capture profile. `query_count`integer Number of captured queries in the profile. `total_size_bytes`integer Total size of the profile archive contents, in bytes. `wal_file_count`integer Number of anonymized write-ahead log (`.pt`) files in the profile. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. DELETE`/api/v3/loadcap/profiles/{profile_id}` ### Delete a load capture profile Deletes a load capture profile and its stored artifacts from object storage. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200The profile was deleted. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. 409The profile is still capturing. Wait for it to finish (or its status to catch up to `complete`), then retry. GET`/api/v3/loadcap/profiles/{profile_id}/download` ### Download a load capture profile Downloads the load capture profile as a gzipped tar archive containing the anonymized capture artifacts—`catalog.json` and `.pt` write-ahead log files for `write` and `both` captures, and `queries.json.zst` for `query` and `both` captures. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/download" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200The profile archive. The response `Content-Type` is `application/gzip` and `Content-Disposition` provides the `{profile_id}.tar.gz` filename. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}/files` ### List files in a load capture profile Returns the name, size, and type of each file stored in a load capture profile. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/files" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Files in the load capture profile. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}/files/{filename}` ### Download a single file from a load capture profile Downloads one anonymized file from a load capture profile—for example, `catalog.json`, `queries.json.zst`, or a `.pt` write-ahead log file. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/files/{filename}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200The file contents. The response `Content-Type` is `application/octet-stream` and `Content-Disposition` provides the filename. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile or file does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. GET`/api/v3/loadcap/profiles/{profile_id}/preview` ### Preview a load capture profile Returns file counts, total size, and an anonymized catalog summary for a load capture profile without downloading the archive. For `query`-only captures, `catalog_summary` is null. This endpoint requires an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/loadcap/profiles/{profile_id}/preview" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Load capture profile preview. `catalog_summary`object Anonymized catalog snapshot taken when the capture started, including databases, tables, columns, and node metadata. Null for `query`-only captures. `file_count`integer Total number of files in the profile archive. `profile_id`string \ Identifier of the load capture profile. `query_count`integer Number of captured queries in the profile. `total_size_bytes`integer Total size of the profile archive contents, in bytes. `wal_file_count`integer Number of anonymized write-ahead log (`.pt`) files in the profile. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404The profile does not exist, or load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. POST`/api/v3/loadcap/start` ### Start a load capture Starts a timed capture of write requests, query requests, or both on a query-capable Enterprise node that has the upgraded storage engine and an explicit `--mode` setting that includes `query`. Only one capture can run at a time on a node. This endpoint requires an admin token. #### Request bodyrequired Content-Type:`application/json` `duration`requiredstring Capture duration using seconds, minutes, or hours. The maximum duration is one hour. Example:`"5m"` `type`requiredstring The request types to capture. Allowed:`write`, `query`, `both` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/loadcap/start" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "duration": "5m", "type": "write" }' ``` #### Responses 200Load capture started. `profile_id`requiredstring \ Identifier of the new load capture profile. 400Invalid capture type or duration. 401Unauthorized access. `data`object `error`string 403Forbidden. An admin token is required. 404Load capture is unavailable because this node does not use the upgraded storage engine with an explicit `--mode` setting that includes `query`. 409A load capture is already running on this node. #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Migrate from InfluxDB v1 or v2 Source: https://docs.influxdata.com/influxdb3/enterprise/api/migrate-from-influxdb-v1-or-v2/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Migrate your existing InfluxDB v1 or v2 workloads to InfluxDB 3 Enterprise. InfluxDB 3 provides compatibility endpoints that work with InfluxDB 1.x and 2.x client libraries and tools. Operations marked with v1 or v2 badges are compatible with the respective InfluxDB version. ### Migration guides * [Migrate from InfluxDB v1 or v2](/influxdb3/enterprise/get-started/migrate-from-influxdb-v1-v2/) - For users migrating from InfluxDB 1.x, 2.x, or Cloud * [Use compatibility APIs to write data](/influxdb3/enterprise/write-data/http-api/compatibility-apis/) - v1 and v2 write endpoints * [Use the v1 HTTP query API](/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/) - InfluxQL queries via HTTP --- # Processing engine Source: https://docs.influxdata.com/influxdb3/enterprise/api/processing-engine/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Manage Processing engine triggers, test plugins, and send requests to trigger On Request plugins. InfluxDB 3 Enterprise provides the InfluxDB 3 processing engine, an embedded Python VM that can dynamically load and trigger Python plugins in response to events in your database. Use Processing engine plugins and triggers to run code and perform tasks for different database events. POST`/api/v3/configure/plugin_environment/install_packages` ### Install plugin packages Installs the specified Python packages into the processing engine plugin environment. This endpoint is synchronous and blocks until the packages are installed. #### Parameters ##### Header parameters `Content-Type`string The format of the data in the request body. Allowed values:`application/json` #### Request bodyrequired Content-Type:`application/json` `packages`requiredstring[] A list of Python package names to install. Can include version specifiers (for example, “scipy==1.9.0”). Example:`["influxdb3-python","scipy","pandas==1.5.0","requests"]` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/plugin_environment/install_packages" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "packages": [ "influxdb3-python", "scipy", "pandas==1.5.0", "requests" ] }' ``` #### Responses 200Success. The packages are installed. In InfluxDB 3 Core 3.12, a failed install can also return `200`; confirm that the package imports before relying on it. 400Bad request. 401Unauthorized access. `data`object `error`string 500The package install failed, for example because pip couldn’t find the package. The body includes the package manager’s error. POST`/api/v3/configure/plugin_environment/install_requirements` ### Install plugin requirements Installs requirements from a requirements file (also known as a “pip requirements file”) into the processing engine plugin environment. This endpoint is synchronous and blocks until the requirements are installed. ### Related * [Processing engine and Python plugins](/influxdb3/enterprise/plugins/) * [Python requirements file format](https://pip.pypa.io/en/stable/reference/requirements-file-format/) #### Parameters ##### Header parameters `Content-Type`string The format of the data in the request body. Allowed values:`application/json` #### Request bodyrequired Content-Type:`application/json` `requirements_location`requiredstring The path to the requirements file containing Python packages to install. Can be a relative path (relative to the plugin directory) or an absolute path. Example:`"requirements.txt"` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/plugin_environment/install_requirements" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "requirements_location": "requirements.txt" }' ``` #### Responses 200Success. The requirements have been installed. 400Bad request. 401Unauthorized access. `data`object `error`string POST`/api/v3/configure/processing_engine_trigger` ### Create processing engine trigger Creates a processing engine trigger with the specified plugin file and trigger specification. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring `disabled`boolean Whether the trigger is disabled. `node_spec`string or object Specifies which nodes in a multi-node InfluxDB 3 Enterprise cluster handle the cache or trigger. Accepts the string `all` or an object that lists specific node IDs. If omitted, defaults to `all`. Default:`all` Example:`"all"` One of:string Create the cache or trigger on all nodes in the cluster. Allowed:`all` Or:object Create the cache or trigger only on the specified nodes. `nodes`requiredstring[] Node IDs (the `--node-id` value each node was started with). Each node must exist and must run in `process` mode. `plugin_filename`requiredstring The path and filename of the plugin to execute–for example,`schedule.py` or `endpoints/report.py`. The path can be absolute or relative to the `--plugin-dir` directory configured when starting InfluxDB 3. The plugin file must implement the trigger interface associated with the trigger’s specification. Prefix the path with `gh:` to fetch the plugin remotely from the configured `--plugin-repo` instead of reading it from the local `--plugin-dir`–for example, `gh:examples/wal_plugin/wal_plugin.py`. Despite the name, `gh:` doesn’t require GitHub–`--plugin-repo` accepts any HTTP/HTTPS URL that serves raw plugin files. For defaults, fetch behavior, and limitations, see [Use example plugins](/influxdb3/enterprise/plugins/#use-example-plugins). `trigger_arguments`object Optional, plugin-specific arguments passed to the plugin entry point as its `args` dictionary. Provide string key-value pairs. Omit this field to pass `None` as `args`. Refer to the plugin documentation for supported argument names and values. Example:`{"target_table":"alerts","threshold":"90"}` `trigger_name`requiredstring `trigger_settings`object `error_behavior`string How the trigger responds when the plugin returns an error: * `log`: Log the error and keep the trigger active. * `retry`: Retry the failed invocation with exponential backoff (up to 5 attempts), then log the failure. Errors identified as permanent aren’t retried. * `disable`: Disable the trigger until you manually re-enable it. Allowed:`log`, `retry`, `disable` Default:`log` `run_async`boolean Whether invocations of the trigger can run concurrently. If `false`, invocations run serially–each invocation must finish before the next one starts. In a cluster with multiple process nodes, the trigger runs on one process node at a time. If `true`, invocations can run concurrently, up to the `--async-trigger-concurrency-limit`server option (default: `8`). In a cluster with multiple process nodes, pending invocations are distributed across all process nodes. `trigger_specification`requiredstring Specifies when and how the processing engine trigger should be invoked. ## Supported trigger specifications: ### Cron-based scheduling Format: `cron:CRON_EXPRESSION` Uses extended (6-field) cron format (second minute hour day\_of\_month month day\_of\_week): ``` ┌───────────── second (0-59) │ ┌───────────── minute (0-59) │ │ ┌───────────── hour (0-23) │ │ │ ┌───────────── day of month (1-31) │ │ │ │ ┌───────────── month (1-12) │ │ │ │ │ ┌───────────── day of week (0-6, Sunday=0) │ │ │ │ │ │ * * * * * * ``` Examples: * `cron:0 0 6 * * 1-5` - Every weekday at 6:00 AM * `cron:0 30 14 * * 5` - Every Friday at 2:30 PM * `cron:0 0 0 1 * *` - First day of every month at midnight ### Interval-based scheduling Format: `every:DURATION` Supported durations: `s` (seconds), `m` (minutes), `h` (hours), `d` (days), `w` (weeks), `M` (months), `y` (years): * `every:30s` - Every 30 seconds * `every:5m` - Every 5 minutes * `every:1h` - Every hour * `every:1d` - Every day * `every:1w` - Every week * `every:1M` - Every month * `every:1y` - Every year **Maximum interval**: 1 year ### Table-based triggers * `all_tables` - Triggers on write events to any table in the database * `table:TABLE_NAME` - Triggers on write events to a specific table ### On-demand triggers Format: `request:REQUEST_PATH` Creates an HTTP endpoint `/api/v3/engine/REQUEST_PATH` for manual invocation: * `request:hello-world` - Creates endpoint `/api/v3/engine/hello-world` * `request:data-export` - Creates endpoint `/api/v3/engine/data-export` Example:`"cron:0 0 6 * * 1-5"` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "db": "DB", "disabled": false, "node_spec": "all", "plugin_filename": "PLUGIN_FILENAME", "trigger_arguments": { "target_table": "alerts", "threshold": "90" }, "trigger_name": "TRIGGER_NAME", "trigger_settings": { "error_behavior": "log", "run_async": false }, "trigger_specification": "cron:0 0 6 * * 1-5" }' ``` #### Responses 200Success. Processing engine trigger created. 400Bad request. 401Unauthorized access. `data`object `error`string 404Trigger not found. DELETE`/api/v3/configure/processing_engine_trigger` ### Delete processing engine trigger Deletes a processing engine trigger. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `trigger_name`requiredstring `force`boolean Force deletion of the trigger even if it has active executions. By default, deletion fails if the trigger is currently executing. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/processing_engine_trigger?db=DB&trigger_name=TRIGGER_NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The processing engine trigger has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Trigger not found. POST`/api/v3/configure/processing_engine_trigger/disable` ### Disable processing engine trigger Disables a processing engine trigger. #### Parameters ##### Query parameters `db`requiredstring The database name. `trigger_name`requiredstring The name of the trigger. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/processing_engine_trigger/disable?db=DB&trigger_name=TRIGGER_NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The processing engine trigger has been disabled. 400Bad request. 401Unauthorized access. `data`object `error`string 404Trigger not found. POST`/api/v3/configure/processing_engine_trigger/enable` ### Enable processing engine trigger Enables a processing engine trigger. #### Parameters ##### Query parameters `db`requiredstring The database name. `trigger_name`requiredstring The name of the trigger. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/processing_engine_trigger/enable?db=DB&trigger_name=TRIGGER_NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The processing engine trigger is enabled. 400Bad request. 401Unauthorized access. `data`object `error`string 404Trigger not found. GET`/api/v3/engine/{request_path}` ### On Request processing engine plugin request Executes the On Request processing engine plugin specified in the trigger’s `plugin_filename`. The request can include request headers, query string parameters, and a request body, which InfluxDB passes to the plugin. An On Request plugin implements the following signature: ```python def process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None) ``` The response depends on the plugin implementation. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/engine/{request_path}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The plugin request has been executed. 400Malformed request. 401Unauthorized access. `data`object `error`string 404Plugin not found. 500Processing failure. POST`/api/v3/engine/{request_path}` ### On Request processing engine plugin request Executes the On Request processing engine plugin specified in the trigger’s `plugin_filename`. The request can include request headers, query string parameters, and a request body, which InfluxDB passes to the plugin. An On Request plugin implements the following signature: ```python def process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None) ``` The response depends on the plugin implementation. #### Parameters ##### Header parameters `Content-Type`string The format of the data in the request body. Allowed values:`application/json` #### Request body Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/engine/{request_path}" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The plugin request has been executed. 400Malformed request. 401Unauthorized access. `data`object `error`string 404Plugin not found. 500Processing failure. POST`/api/v3/plugin_test/schedule` ### Test scheduling plugin Executes a test of a scheduling plugin. #### Request bodyrequired Content-Type:`application/json` `cache_name`string Optional name of the cache to use in the test. `database`requiredstring The database name to use for the test. `filename`requiredstring The path and filename of the plugin to test. `input_arguments`object Optional key-value pairs of arguments to pass to the plugin. `schedule`string Optional schedule specification in cron or interval format. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/plugin_test/schedule" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "cache_name": "CACHE_NAME", "database": "DATABASE", "filename": "FILENAME", "input_arguments": {}, "schedule": "SCHEDULE" }' ``` #### Responses 200Success. The plugin test has been executed. 400Bad request. 401Unauthorized access. `data`object `error`string 404Plugin not enabled. POST`/api/v3/plugin_test/wal` ### Test WAL plugin Executes a test of a write-ahead logging (WAL) plugin. #### Request bodyrequired Content-Type:`application/json` `cache_name`string Optional name of the cache to use in the test. `database`requiredstring The database name to use for the test. `filename`requiredstring The path and filename of the plugin to test. `input_arguments`object Optional key-value pairs of arguments to pass to the plugin. `input_lp`requiredstring Line protocol data to use as input for the test. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/plugin_test/wal" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "cache_name": "CACHE_NAME", "database": "DATABASE", "filename": "FILENAME", "input_arguments": {}, "input_lp": "INPUT_LP" }' ``` #### Responses 200Success. The plugin test has been executed. 400Bad request. 401Unauthorized access. `data`object `error`string 404Plugin not enabled. PUT`/api/v3/plugins/directory` ### Update a multi-file plugin directory Replaces all files in a multi-file plugin directory. The`plugin_name` must match a registered trigger name. Each entry in the `files` array specifies a `relative_path` and `content`—the server writes them into the trigger’s plugin directory. Use this endpoint to update multi-file plugins (directories with`__init__.py` and supporting modules). For single-file plugins, use `PUT /api/v3/plugins/files` instead. #### Request bodyrequired Content-Type:`application/json` `files`requiredobject[] List of plugin files to include in the directory. `content`requiredstring The content of the file. `relative_path`requiredstring The relative path of the file within the plugin directory. `plugin_name`requiredstring The name of the plugin directory to update. Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/plugins/directory" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "files": [ { "content": "CONTENT", "relative_path": "RELATIVE_PATH" } ], "plugin_name": "PLUGIN_NAME" }' ``` #### Responses 200Success. The plugin directory has been updated. 401Unauthorized access. `data`object `error`string 403Forbidden. Admin token required. 500Plugin not found. The `plugin_name` does not match any registered trigger. POST`/api/v3/plugins/files` ### Create a plugin file Creates a single plugin file in the plugin directory. Writes the`content` to a file named after `plugin_name`. Does not require an existing trigger—use this to upload plugin files before creating triggers that reference them. #### Request bodyrequired Content-Type:`application/json` `content`requiredstring The content of the plugin file. `plugin_name`requiredstring The name of the plugin file to update. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/plugins/files" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "content": "CONTENT", "plugin_name": "PLUGIN_NAME" }' ``` #### Responses 200Success. The plugin file has been created. 401Unauthorized access. `data`object `error`string 403Forbidden. Admin token required. PUT`/api/v3/plugins/files` ### Update a plugin file Updates a single plugin file for an existing trigger. The`plugin_name` must match a registered trigger name—the server resolves the trigger’s `plugin_filename` and overwrites that file with the provided `content`. This endpoint does not update triggers whose `plugin_filename` uses the `gh:` prefix. To fetch updated code from the configured plugin source, disable the trigger with`POST /api/v3/configure/processing_engine_trigger/disable`, then re-enable it with`POST /api/v3/configure/processing_engine_trigger/enable`. To upload a new plugin file before creating a trigger, use`POST /api/v3/plugins/files` instead. To update a multi-file plugin directory, use `PUT /api/v3/plugins/directory`. #### Request bodyrequired Content-Type:`application/json` `content`requiredstring The content of the plugin file. `plugin_name`requiredstring The name of the plugin file to update. Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/plugins/files" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "content": "CONTENT", "plugin_name": "PLUGIN_NAME" }' ``` #### Responses 200Success. The plugin file has been updated. 401Unauthorized access. `data`object `error`string 403Forbidden. Admin token required. 500Plugin not found. The `plugin_name` does not match any registered trigger. #### Related * [Processing engine and Python plugins](/influxdb3/enterprise/processing-engine/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Query data Source: https://docs.influxdata.com/influxdb3/enterprise/api/query-data/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Query data stored in InfluxDB 3 Enterprise using SQL or InfluxQL. GET`/api/v3/query_influxql` ### Execute InfluxQL query Executes an InfluxQL query to retrieve data from the specified database. With the [performance upgrade preview](/influxdb3/enterprise/performance-preview/) enabled (`--use-pacha-tree` flag), queries automatically merge results from Parquet and`.pt` files (hybrid query mode). #### Parameters ##### Query parameters `db`string The name of the database. If you provide a query that specifies the database, you can omit the ‘db’ parameter from your request. `q`requiredstring `format`string `params`string JSON-encoded query parameters for parameterized queries. ##### Header parameters `Accept`string The content type that the client can understand. Allowed values:`application/json`, `application/jsonl`, `application/vnd.apache.parquet`, `text/csv` Default:`application/json` #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/query_influxql?q=Q" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The response body contains query results. `results`object[] Example request body ```json { "results": [ { "series": [ { "columns": [ "time", "value" ], "name": "mytable", "values": [ [ "2024-02-02T12:00:00Z", 42 ] ] } ] } ] } ``` 400Bad request. 401Unauthorized access. `data`object `error`string 403Access denied. 404Database not found. 405Method not allowed. 422Unprocessable entity. POST`/api/v3/query_influxql` ### Execute InfluxQL query Executes an InfluxQL query to retrieve data from the specified database. With the [performance upgrade preview](/influxdb3/enterprise/performance-preview/) enabled (`--use-pacha-tree` flag), queries automatically merge results from Parquet and`.pt` files (hybrid query mode). #### Parameters ##### Header parameters `Accept`string The content type that the client can understand. Allowed values:`application/json`, `application/jsonl`, `application/vnd.apache.parquet`, `text/csv` Default:`application/json` `Content-Type`string The format of the data in the request body. Allowed values:`application/json` #### Request body Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/query_influxql" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"db":"mydb","format":"json","params":{},"q":"SELECT * FROM mytable"}' ``` #### Responses 200Success. The response body contains query results. `results`object[] Example request body ```json { "results": [ { "series": [ { "columns": [ "time", "value" ], "name": "mytable", "values": [ [ "2024-02-02T12:00:00Z", 42 ] ] } ] } ] } ``` 400Bad request. 401Unauthorized access. `data`object `error`string 403Access denied. 404Database not found. 405Method not allowed. 422Unprocessable entity. GET`/api/v3/query_sql` ### Execute SQL query Executes an SQL query to retrieve data from the specified database. With the [performance upgrade preview](/influxdb3/enterprise/performance-preview/) enabled (`--use-pacha-tree` flag), queries automatically merge results from Parquet and`.pt` files (hybrid query mode). Use the`/api/v3/query_sql_telemetry` endpoint after executing a query to retrieve detailed execution statistics. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `q`requiredstring \ The query to execute. `format`string `params`string JSON-encoded query parameters for parameterized queries. ##### Header parameters `Accept`string The content type that the client can understand. Allowed values:`application/json`, `application/jsonl`, `application/vnd.apache.parquet`, `text/csv` Default:`application/json` `Content-Type`string The format of the data in the request body. Allowed values:`application/json` #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/query_sql?db=DB&q=Q" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The response body contains query results. `results`object[] Example request body ```json { "results": [ { "series": [ { "columns": [ "time", "value" ], "name": "mytable", "values": [ [ "2024-02-02T12:00:00Z", 42 ] ] } ] } ] } ``` 400Bad request. 401Unauthorized access. `data`object `error`string 403Access denied. 404Database not found. 405Method not allowed. 422Unprocessable entity. POST`/api/v3/query_sql` ### Execute SQL query Executes an SQL query to retrieve data from the specified database. With the [performance upgrade preview](/influxdb3/enterprise/performance-preview/) enabled (`--use-pacha-tree` flag), queries automatically merge results from Parquet and`.pt` files (hybrid query mode). Use the`/api/v3/query_sql_telemetry` endpoint after executing a query to retrieve detailed execution statistics. #### Parameters ##### Header parameters `Accept`string The content type that the client can understand. Allowed values:`application/json`, `application/jsonl`, `application/vnd.apache.parquet`, `text/csv` Default:`application/json` `Content-Type`string The format of the data in the request body. Allowed values:`application/json` #### Request body Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"db":"mydb","format":"json","params":{},"q":"SELECT * FROM mytable"}' ``` #### Responses 200Success. The response body contains query results. `results`object[] Example request body ```json { "results": [ { "series": [ { "columns": [ "time", "value" ], "name": "mytable", "values": [ [ "2024-02-02T12:00:00Z", 42 ] ] } ] } ] } ``` 400Bad request. 401Unauthorized access. `data`object `error`string 403Access denied. 404Database not found. 405Method not allowed. 422Unprocessable entity. GET`/query` ### Execute InfluxQL query (v1-compatible) Executes an InfluxQL query to retrieve data from the specified database. This endpoint is compatible with InfluxDB 1.x client libraries and third-party integrations such as Grafana. Use query parameters to specify the database and the InfluxQL query. #### Related * [Use the InfluxDB v1 HTTP query API and InfluxQL to query data](/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/) #### Parameters ##### Query parameters `chunked`boolean If true, the response is divided into chunks of size `chunk_size`. `chunk_size`integer The number of records that will go into a chunk. This parameter is only used if `chunked=true`. Default:`10000` `db`string \ The database to query. If not provided, the InfluxQL query string must specify the database. `pretty`boolean If true, the JSON response is formatted in a human-readable format. `q`requiredstring The InfluxQL query string. `epoch`string Formats timestamps as [unix (epoch) timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp) with the specified precision instead of [RFC3339 timestamps](/influxdb3/enterprise/reference/glossary/#rfc3339-timestamp) with nanosecond precision. `u`string Username for v1 compatibility authentication. When using Basic authentication or query string authentication, InfluxDB 3 ignores this parameter but allows any arbitrary string for compatibility with InfluxDB 1.x clients. `p`string Password for v1 compatibility authentication. For query string authentication, pass a database token with write permissions as this parameter. InfluxDB 3 checks that the `p` value is an authorized token. `rp`string Retention policy name. Honored but discouraged. InfluxDB 3 doesn’t use retention policies. ##### Header parameters `Accept`string The content type that the client can understand. If `text/csv` is specified, the `Content-type` response header is `application/csv` and the response is formatted as CSV. Returns an error if the format is invalid or non-UTF8. Allowed values:`application/json`, `application/csv`, `text/csv` Default:`application/json` `Authorization`string Authorization header for token-based authentication. Supported schemes: * `Bearer AUTH_TOKEN` - OAuth bearer token scheme * `Token AUTH_TOKEN` - InfluxDB v2 token scheme * `Basic ` - Basic authentication (username is ignored) Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/query?q=Q" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The response body contains query results. `results`object[] Example request body ```json { "results": [ { "series": [ { "columns": [ "time", "value" ], "name": "mytable", "values": [ [ "2024-02-02T12:00:00Z", 42 ] ] } ] } ] } ``` 400Bad request. 401Unauthorized access. `data`object `error`string 403Access denied. 404Database not found. 405Method not allowed. 422Unprocessable entity. POST`/query` ### Execute InfluxQL query (v1-compatible) Executes an InfluxQL query to retrieve data from the specified database. #### Related * [Use the InfluxDB v1 HTTP query API and InfluxQL to query data](/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/) #### Parameters ##### Header parameters `Accept`string The content type that the client can understand. If `text/csv` is specified, the `Content-type` response header is `application/csv` and the response is formatted as CSV. Returns an error if the format is invalid or non-UTF8. Allowed values:`application/json`, `application/csv`, `text/csv` Default:`application/json` #### Request body Content-Type:`application/json` `chunk_size`integer The number of records that will go into a chunk. This parameter is only used if `chunked=true`. Default:`10000` `chunked`boolean If true, the response is divided into chunks of size `chunk_size`. `db`string The database to query. If not provided, the InfluxQL query string must specify the database. `epoch`string A unix timestamp precision. * `h` for hours * `m` for minutes * `s` for seconds * `ms` for milliseconds * `u` or `µ` for microseconds * `ns` for nanoseconds Formats timestamps as [unix (epoch) timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp) with the specified precision instead of [RFC3339 timestamps](/influxdb3/enterprise/reference/glossary/#rfc3339-timestamp) with nanosecond precision. Allowed:`ns`, `u`, `µ`, `ms`, `s`, `m`, `h` `pretty`boolean If true, the JSON response is formatted in a human-readable format. `q`requiredstring The InfluxQL query string. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/query" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "chunk_size": 10000, "chunked": false, "db": "DB", "epoch": "ns", "pretty": false, "q": "Q" }' ``` #### Responses 200Success. The response body contains query results. `results`object[] Example request body ```json { "results": [ { "series": [ { "columns": [ "time", "value" ], "name": "mytable", "values": [ [ "2024-02-02T12:00:00Z", 42 ] ] } ] } ] } ``` 400Bad request. 401Unauthorized access. `data`object `error`string 403Access denied. 404Database not found. 405Method not allowed. 422Unprocessable entity. #### Related * [Query data](/influxdb3/enterprise/query-data/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Query groups Source: https://docs.influxdata.com/influxdb3/enterprise/api/query-groups/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Manage distributed query groups–named sets of query nodes that InfluxDB 3 Enterprise load-balances queries across. > > > **Not yet operational**: These endpoints work and store query group definitions in the catalog, but InfluxDB 3 Enterprise doesn’t yet use those definitions to affect query routing, data placement, or replication. > > GET`/api/v3/enterprise/configure/query_groups`Not operational ### List distributed query groups (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 show query\_groups](/influxdb3/enterprise/reference/cli/influxdb3/show/query_groups/) Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/configure/query_groups" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 401Unauthorized access. `data`object `error`string POST`/api/v3/enterprise/configure/query_groups`Not operational ### Create a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 create query\_group](/influxdb3/enterprise/reference/cli/influxdb3/create/query_group/) #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/query_groups" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 201Created. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string GET`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Retrieve a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 401Unauthorized access. `data`object `error`string 404Query group not found. PUT`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Replace a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update query\_group](/influxdb3/enterprise/reference/cli/influxdb3/update/query_group/) #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 404Query group not found. PATCH`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Update a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update query\_group](/influxdb3/enterprise/reference/cli/influxdb3/update/query_group/) #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request PATCH \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. Not documented. This endpoint is not yet operational; response fields aren’t finalized. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 404Query group not found. DELETE`/api/v3/enterprise/configure/query_groups/{id}`Not operational ### Delete a distributed query group (not yet operational) Reserved for the distributed query group feature. > > > **Not yet operational**: Request and response fields aren’t finalized, and InfluxDB 3 Enterprise doesn’t yet use query group definitions to affect query routing, data placement, or replication. > > This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 delete query\_group](/influxdb3/enterprise/reference/cli/influxdb3/delete/query_group/) #### Parameters ##### Path parameters `id`requiredinteger The ID of the query group. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/query_groups/{id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The query group has been deleted. 401Unauthorized access. `data`object `error`string 404Query group not found. #### Related * [Configure query groups](/influxdb3/enterprise/admin/query-groups/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Quick start Source: https://docs.influxdata.com/influxdb3/enterprise/api/quick-start/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Authenticate, write, and query with the API: 1. Create an admin token to authorize API requests. ``` curl -X POST "http://localhost:8181/api/v3/configure/token/admin" ``` 2. Check the status of the InfluxDB server. ``` curl "http://localhost:8181/health" \ --header "Authorization: Bearer ADMIN_TOKEN" ``` 3. Write data to InfluxDB. ``` curl "http://localhost:8181/api/v3/write_lp?db=sensors&precision=auto" --header "Authorization: Bearer ADMIN_TOKEN" \ --data-raw "home,room=Kitchen temp=72.0 home,room=Living\ room temp=71.5" ``` If all data is written, the response is `204 No Content`. 4. Query data from InfluxDB. ``` curl -G "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer ADMIN_TOKEN" \ --data-urlencode "db=sensors" \ --data-urlencode "q=SELECT * FROM home WHERE room='Living room'" \ --data-urlencode "format=jsonl" ``` Output: ``` {"room":"Living room","temp":71.5,"time":"2025-02-25T20:19:34.984098"} ``` For more information, see the [Get started](/influxdb3/enterprise/get-started/) guide. --- # Server information Source: https://docs.influxdata.com/influxdb3/enterprise/api/server-information/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Retrieve server metrics, health status, and version information for InfluxDB 3 Enterprise. GET`/api/v3/configure/query_concurrency_limit` ### Get the query concurrency limit Returns the current runtime limit on the maximum number of queries that can run concurrently. When the limit is set to the maximum allowed value, `maxConcurrentQueries` is omitted from the response. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/configure/query_concurrency_limit" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the current query concurrency limit. `max_concurrent_queries`integer The maximum number of queries that can run concurrently. Omitted when the limit is set to the maximum allowed value. Example request body ```json { "max_concurrent_queries": 32 } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string PUT`/api/v3/configure/query_concurrency_limit` ### Set the query concurrency limit Sets the runtime limit on the maximum number of queries that can run concurrently. Omit `max_concurrent_queries` (or set it to `null`) to restore the limit configured at server startup. Values below 16 are rejected unless `force=true` is set in the query string. A value of `0` is invalid, and values above the maximum allowed limit are rejected. This endpoint is only available in InfluxDB 3 Enterprise. #### Parameters ##### Query parameters `force`boolean Set to `true` to allow setting `max_concurrent_queries` to a value below the minimum of 16. #### Request bodyrequired Content-Type:`application/json` `max_concurrent_queries`integer The maximum number of queries that can run concurrently. Set to `null` (or omit) to restore the limit configured at server startup. Must be a positive integer; values below 16 require `force=true`. Example request body ```json { "max_concurrent_queries": 32 } ``` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/configure/query_concurrency_limit" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"max_concurrent_queries":32}' ``` #### Responses 204No content. The query concurrency limit was updated. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string DELETE`/api/v3/configure/query_concurrency_limit` ### Reset the query concurrency limit Resets the runtime query concurrency limit to the value configured at server startup. This endpoint is only available in InfluxDB 3 Enterprise. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/query_concurrency_limit" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204No content. The query concurrency limit was reset. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string POST`/api/v3/enterprise/configure/node/remove` ### Permanently remove a stopped node Permanently removes a node that has already been marked as stopped, deleting its data from object storage once the compactor finishes reassigning its work. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 remove node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/) * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) #### Request bodyrequired Content-Type:`application/json` `force_finalize`boolean Force-finalize a node stuck in the `Stopping` state (for example, after a crash that never acknowledged the stop). This can delete unsnapshotted WAL data for that node — only use it when the node’s process is confirmed gone and you accept the data-loss risk. `node_id`requiredstring The ID of the node to remove. Example request body ```json { "node_id": "node-1" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/node/remove" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"node_id":"node-1"}' ``` #### Responses 200Success. The node has been removed. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 404Node not found. 409The node is not in a state that allows removal (for example, still running). POST`/api/v3/enterprise/configure/node/stop` ### Mark a node as stopped Marks a node as stopped in the catalog, freeing up the licensed cores it was using for other nodes. Use this endpoint after you have already stopped the physical instance (for example, using `kill` or stopping the container). This endpoint does not shut down the running process — you must stop the instance first. When the node is marked as stopped: 1. Licensed cores from the stopped node are freed for reuse 2. Other nodes in the cluster see the update after their catalog sync interval This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) #### Request bodyrequired Content-Type:`application/json` `node_id`requiredstring The ID of the node to mark as stopped. Example request body ```json { "node_id": "node-1" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/node/stop" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"node_id":"node-1"}' ``` #### Responses 200Success. The node has been marked as stopped. 401Unauthorized access. `data`object `error`string 404Node not found. GET`/api/v3/enterprise/upgrade/parquet_cleanup` ### Get Parquet cleanup status Returns the current or most recent Parquet cleanup state for the cluster. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 manage cleanup-parquet](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/) Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/enterprise/upgrade/parquet_cleanup" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the current Parquet cleanup state. `bytes_deleted`requiredinteger \ Total size, in bytes, of the deleted files. Always `0` in `dry_run` mode. `bytes_matched`requiredinteger \ Total size, in bytes, of the matched files. `completed_at_ns`integer \ When the cleanup finished, in nanoseconds since the Unix epoch. `files_deleted`requiredinteger \ Number of files deleted so far. Always `0` in `dry_run` mode. `files_failed`requiredinteger \ Number of files the cleanup failed to delete. `files_matched`requiredinteger \ Number of files identified as Parquet-era data to delete. `files_scanned`requiredinteger \ Number of files the cleanup has scanned so far. `last_message`string The most recent human-readable status message, if any. `mode`requiredstring Whether this cleanup deletes data or only reports what it would delete. Allowed:`delete`, `dry_run` `requested_at_ns`integer \ When the cleanup was requested, in nanoseconds since the Unix epoch. `requested_by`string The node that accepted the cleanup request, if recorded. `started_at_ns`integer \ When the cleanup started running, in nanoseconds since the Unix epoch. `status`requiredstring The cleanup’s current status. Allowed:`requested`, `running`, `completed`, `failed` `updated_at_ns`integer \ When this status was last updated, in nanoseconds since the Unix epoch. Example request body ```json { "bytes_deleted": 4294967296, "bytes_matched": 4294967296, "completed_at_ns": 1735689900000000000, "files_deleted": 812, "files_failed": 0, "files_matched": 812, "files_scanned": 1024, "last_message": null, "mode": "delete", "requested_at_ns": 1735689600000000000, "requested_by": "node-1", "started_at_ns": 1735689601000000000, "status": "completed", "updated_at_ns": 1735689900000000000 } ``` 401Unauthorized access. `data`object `error`string 404No Parquet cleanup has ever been requested on this cluster. POST`/api/v3/enterprise/upgrade/parquet_cleanup` ### Request Parquet cleanup after a storage engine upgrade Requests removal of the Parquet-era data left behind by a completed Parquet-to-PachaTree storage engine upgrade. Any node in the cluster accepts the request. The compactor node executes it. Requires the cluster to be fully on the upgraded (PachaTree) storage engine, with the Parquet-to-PachaTree upgrade already complete. Use `dry_run` to count and size the data a real cleanup would delete without deleting anything. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 manage cleanup-parquet](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/) #### Request body Content-Type:`application/json` `dry_run`boolean When `true`, only count and size the Parquet-era data a real cleanup would delete. Delete nothing. Example request body ```json { "dry_run": false } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/upgrade/parquet_cleanup" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"dry_run":false}' ``` #### Responses 202Success. The cleanup started, is already running, or already completed. `state`requiredstring The outcome of this cleanup request. Allowed:`started`, `already_running`, `already_completed` `status`requiredobject Point-in-time view of the durable Parquet cleanup state. Returned by `GET /api/v3/enterprise/upgrade/parquet_cleanup` and embedded in the POST response. Example:`{"bytes_deleted":4294967296,"bytes_matched":4294967296,"completed_at_ns":1735689900000000000,"files_deleted":812,"files_failed":0,"files_matched":812,"files_scanned":1024,"last_message":null,"mode":"delete","requested_at_ns":1735689600000000000,"requested_by":"node-1","started_at_ns":1735689601000000000,"status":"completed","updated_at_ns":1735689900000000000}` `bytes_deleted`requiredinteger \ Total size, in bytes, of the deleted files. Always `0` in `dry_run` mode. `bytes_matched`requiredinteger \ Total size, in bytes, of the matched files. `completed_at_ns`integer \ When the cleanup finished, in nanoseconds since the Unix epoch. `files_deleted`requiredinteger \ Number of files deleted so far. Always `0` in `dry_run` mode. `files_failed`requiredinteger \ Number of files the cleanup failed to delete. `files_matched`requiredinteger \ Number of files identified as Parquet-era data to delete. `files_scanned`requiredinteger \ Number of files the cleanup has scanned so far. `last_message`string The most recent human-readable status message, if any. `mode`requiredstring Whether this cleanup deletes data or only reports what it would delete. Allowed:`delete`, `dry_run` `requested_at_ns`integer \ When the cleanup was requested, in nanoseconds since the Unix epoch. `requested_by`string The node that accepted the cleanup request, if recorded. `started_at_ns`integer \ When the cleanup started running, in nanoseconds since the Unix epoch. `status`requiredstring The cleanup’s current status. Allowed:`requested`, `running`, `completed`, `failed` `updated_at_ns`integer \ When this status was last updated, in nanoseconds since the Unix epoch. 401Unauthorized access. `data`object `error`string 409The cluster isn’t on the upgraded storage engine, no Parquet-to-PachaTree upgrade was ever recorded, the upgrade hasn’t completed, or a cleanup is already active in a different mode (dry run vs. delete). POST`/api/v3/enterprise/upgrade/retry_parquet_to_pacha_tree` ### Retry a failed Parquet-to-PachaTree upgrade Resets a Parquet-to-PachaTree storage engine upgrade that latched to a `failed` state, so it resumes the next time a compactor node starts with `--upgrade-pacha-tree`. Re-checks every previously failed upgrade source against the catalog and object store: sources that are still readable are requeued, sources whose table or database was dropped are recorded as skipped, and sources that can’t be read back are left unresolved for the next retry. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 manage retry-upgrade-to-pacha-tree](/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/) Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/upgrade/retry_parquet_to_pacha_tree" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns a summary of what the retry changed. `entries_already_converted`requiredinteger \ Entries whose sequence already held converted data. Recorded as imported instead of skipped. `entries_already_skipped`requiredinteger \ Terminal failures that were already recorded as skips. `entries_inspected`requiredinteger \ Terminal failures the retry re-checked against the catalog and object store. `entries_requeued`requiredinteger \ Entries returned to the queue because their source is still readable. `entries_skipped_source_missing`requiredinteger \ Entries skipped because their source object no longer exists. `entries_skipped_table_dropped`requiredinteger \ Entries skipped because their table was dropped. `entries_unresolved`requiredinteger \ Entries left untouched because the object holding their sequence couldn’t be read back. Run this endpoint again to re-attempt them. `message`requiredstring What to do next. `upgrade_status`requiredstring The status the enterprise upgrade state was left in after the retry. Example request body ```json { "entries_already_converted": 0, "entries_already_skipped": 0, "entries_inspected": 3, "entries_requeued": 2, "entries_skipped_source_missing": 1, "entries_skipped_table_dropped": 0, "entries_unresolved": 0, "message": "Migration state reset. Restart the compactor node with --upgrade-pacha-tree to resume the upgrade.", "upgrade_status": "upgrading" } ``` 401Unauthorized access. `data`object `error`string 409The upgrade isn’t in a state this request can act on (for example, it isn’t `failed`), or another request already reset it. GET`/api/v1/health` ### Health check (v1) Checks the status of the service. Returns `OK` if the service is running. This endpoint does not return version information. Use the [`/ping`](#operation/GetPing) endpoint to retrieve version details. > > > **Note**: This endpoint requires authentication by default in InfluxDB 3 Enterprise. > > Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v1/health" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Service is running. Returns `OK`. 401Unauthorized. Authentication is required. 500Service is unavailable. GET`/health` ### Health check Checks the status of the service. Returns `OK` if the service is running. This endpoint does not return version information. Use the [`/ping`](#operation/GetPing) endpoint to retrieve version details. > > > **Note**: This endpoint requires authentication by default in InfluxDB 3 Enterprise. > > A database-scoped resource token (for example, `db:DATABASE_NAME:read`) gets `403` from this endpoint. For health checks, use a token with the `system:health:read` permission, or an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/health" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Service is running. Returns `OK`. 401Unauthorized. Authentication is required. 403Forbidden. The token doesn’t have permission to call this endpoint. 500Service is unavailable. GET`/metrics` ### Metrics Retrieves Prometheus-compatible server metrics. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/metrics" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success GET`/ping` ### Ping the server Returns version information for the server. **Important**: Use a GET request. HEAD requests return `404 Not Found`. The response includes version information in both headers and the JSON body: * **Headers**: `x-influxdb-version`, `x-influxdb-build`, and `cluster-uuid`(`cluster-uuid` is also returned on a `401`) * **Body**: JSON object with `version`, `revision`, and `process_id` > > > **Note**: This endpoint requires authentication by default in InfluxDB 3 Enterprise. > > A database-scoped resource token (for example, `db:DATABASE_NAME:read`) gets `403` from this endpoint. For health checks, use a token with the `system:ping:read` permission, or an admin token. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/ping" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The response body contains server information. `process_id`string A unique identifier for the server process. Example:`"b756d9e0-cecd-4f72-b6d0-19e2d4f8cbb7"` `revision`string The git revision hash for the build. Example:`"83b589b883"` `version`string The InfluxDB version number. Example:`"3.8.0"` 401Unauthorized. Authentication is required. 403Forbidden. The token doesn’t have permission to call this endpoint. 404Not Found. Returned for HEAD requests. Use a GET request to retrieve version information. POST`/ping` ### Ping the server Returns version information for the server. Accepts POST in addition to GET. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/ping" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. The response body contains server information. `process_id`string A unique identifier for the server process. Example:`"b756d9e0-cecd-4f72-b6d0-19e2d4f8cbb7"` `revision`string The git revision hash for the build. Example:`"83b589b883"` `version`string The InfluxDB version number. Example:`"3.8.0"` 401Unauthorized. Authentication is required. 403Forbidden. The token doesn’t have permission to call this endpoint. 404Not Found. Returned for HEAD requests. Use a GET request to retrieve version information. #### Related * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Table Source: https://docs.influxdata.com/influxdb3/enterprise/api/table/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Manage table schemas in an InfluxDB 3 Enterprise database. POST`/api/v3/configure/distinct_cache` ### Create distinct cache Creates a distinct cache for a table. #### Request bodyrequired Content-Type:`application/json` `columns`requiredstring[] `db`requiredstring `max_age`integer Optional maximum age in seconds. `max_cardinality`integer Optional maximum cardinality. `name`string Optional cache name. `node_spec`string or object Specifies which nodes in a multi-node InfluxDB 3 Enterprise cluster handle the cache or trigger. Accepts the string `all` or an object that lists specific node IDs. If omitted, defaults to `all`. Default:`all` Example:`"all"` One of:string Create the cache or trigger on all nodes in the cluster. Allowed:`all` Or:object Create the cache or trigger only on the specified nodes. `nodes`requiredstring[] Node IDs (the `--node-id` value each node was started with). Each node must exist and must run in `process` mode. `table`requiredstring Example request body ```json { "columns": [ "tag1", "tag2" ], "db": "mydb", "max_age": 3600, "max_cardinality": 1000, "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/distinct_cache" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"columns":["tag1","tag2"],"db":"mydb","max_age":3600,"max_cardinality":1000,"table":"mytable"}' ``` #### Responses 201Success. The distinct cache has been created. 400 Bad request. The server responds with status `400` if the request would overwrite an existing cache with a different configuration. 409Conflict. A distinct cache with this configuration already exists. DELETE`/api/v3/configure/distinct_cache` ### Delete distinct cache Deletes a distinct cache. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The name of the table containing the distinct cache. `name`requiredstring The name of the distinct cache to delete. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/distinct_cache?db=DB&table=TABLE&name=NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The distinct cache has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Cache not found. POST`/api/v3/configure/last_cache` ### Create last cache Creates a last cache for a table. #### Request bodyrequired Content-Type:`application/json` `count`integer Optional count. `db`requiredstring `key_columns`string[] Optional list of key columns. `name`string Optional cache name. `node_spec`string or object Specifies which nodes in a multi-node InfluxDB 3 Enterprise cluster handle the cache or trigger. Accepts the string `all` or an object that lists specific node IDs. If omitted, defaults to `all`. Default:`all` Example:`"all"` One of:string Create the cache or trigger on all nodes in the cluster. Allowed:`all` Or:object Create the cache or trigger only on the specified nodes. `nodes`requiredstring[] Node IDs (the `--node-id` value each node was started with). Each node must exist and must run in `process` mode. `table`requiredstring `ttl`integer Optional time-to-live in seconds. `value_columns`string[] Optional list of value columns. Example request body ```json { "count": 100, "db": "mydb", "key_columns": [ "tag1" ], "table": "mytable", "ttl": 3600, "value_columns": [ "field1" ] } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/last_cache" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"count":100,"db":"mydb","key_columns":["tag1"],"table":"mytable","ttl":3600,"value_columns":["field1"]}' ``` #### Responses 201Success. Last cache created. 400Bad request. A cache with this name already exists or the request is malformed. 401Unauthorized access. `data`object `error`string 404Cache not found. DELETE`/api/v3/configure/last_cache` ### Delete last cache Deletes a last cache. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring The name of the table containing the last cache. `name`requiredstring The name of the last cache to delete. #### Request bodyrequired Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/last_cache?db=DB&table=TABLE&name=NAME" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" ``` #### Responses 200Success. The last cache has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Cache not found. POST`/api/v3/configure/table` ### Create a table Creates a new table within a database. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring `fields`requiredobject[] `name`requiredstring `type`requiredstring Allowed:`utf8`, `int64`, `uint64`, `float64`, `bool` `retention_period`string The retention period for the table. Specifies how long data in this table should be retained. Use duration format (for example, “1d”, “1h”, “30m”, “7d”). Example:`"30d"` `table`requiredstring `tags`requiredstring[] Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "db": "DB", "fields": [ { "name": "NAME", "type": "utf8" } ], "retention_period": "30d", "table": "TABLE", "tags": [ "TAGS" ] }' ``` #### Responses 200Success. The table has been created. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database not found. PATCH`/api/v3/configure/table` ### Add columns to a table Adds tag and field columns to an existing table. The change is additive: a column that already exists with the requested type is left unchanged. No other table settings are modified. The request must name at least one tag or field. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring `fields`object[] `name`requiredstring `type`requiredstring Allowed:`utf8`, `int64`, `uint64`, `float64`, `bool` `table`requiredstring `tags`string[] Example request[Ask AI about this](#) ```sh curl --request PATCH \ "https://localhost:8181/api/v3/configure/table" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{ "db": "DB", "fields": [ { "name": "NAME", "type": "utf8" } ], "table": "TABLE", "tags": [ "TAGS" ] }' ``` #### Responses 200Success. The columns have been added to the table. The response body is empty. 400Bad request. The request names no tags or fields, or is otherwise invalid. 401Unauthorized access. `data`object `error`string 404Database or table not found. DELETE`/api/v3/configure/table` ### Delete a table Soft deletes a table. The table is scheduled for deletion and unavailable for querying. Use the `hard_delete_at` parameter to schedule a hard deletion. Use the `data_only` parameter to delete data while preserving the table schema and resources. #### Deleting a table cannot be undone Deleting a table is a destructive action. Once a table is deleted, data stored in that table cannot be recovered. #### Soft-deleted tables keep a renamed, visible entry until hard deletion A soft delete renames the table to `-` instead of removing it. Until the table is hard deleted, this renamed entry still appears in `SHOW TABLES` output and in`information_schema.tables` queries. Filter out names that end in a deleted-at timestamp to account for these renamed entries when listing or counting tables. Deleting an already-deleted table returns `409 Conflict`. There’s no API to remove the renamed entry directly; only a scheduled or immediate (`hard_delete_at=now`) hard deletion removes it. #### Parameters ##### Query parameters `db`requiredstring The name of the database. `table`requiredstring `data_only`boolean Delete only data while preserving the table schema and all associated resources (last value caches, distinct value caches). When `false` (default), the entire table is deleted. `hard_delete_at`string \ Schedule the table for hard deletion at the specified time. If not provided, the table will be soft deleted. Use ISO 8601 format (for example, “2025-12-31T23:59:59Z”). Also accepts special string values: * `now` — hard delete immediately * `never` — soft delete only (default behavior) * `default` — use the system default hard deletion time Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/configure/table?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success (no content). The table has been deleted. 401Unauthorized access. `data`object `error`string 404Table not found. 409Conflict. The table is already deleted. POST`/api/v3/enterprise/configure/file_index` ### Create a file index Creates a file index for a database or table. A file index improves query performance by indexing data files based on specified columns, enabling the query engine to skip irrelevant files during query execution. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `columns`requiredstring[] The columns to use for the file index. `db`requiredstring The database name. `table`string The table name. If omitted, the file index applies to the database. Example request body ```json { "columns": [ "tag1", "tag2" ], "db": "mydb", "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/file_index" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"columns":["tag1","tag2"],"db":"mydb","table":"mytable"}' ``` #### Responses 200Success. The file index has been created. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database or table not found. DELETE`/api/v3/enterprise/configure/file_index` ### Delete a file index Deletes a file index from a database or table. This endpoint is only available in InfluxDB 3 Enterprise. #### Request bodyrequired Content-Type:`application/json` `db`requiredstring The database name. `table`string The table name. If omitted, deletes the database-level file index. Example request body ```json { "db": "mydb", "table": "mytable" } ``` Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/file_index" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"db":"mydb","table":"mytable"}' ``` #### Responses 200Success. The file index has been deleted. 400Bad request. 401Unauthorized access. `data`object `error`string 404Database, table, or file index not found. POST`/api/v3/enterprise/configure/table/retention_period` ### Set table retention period Sets or updates the retention period for a specific table. Use this endpoint to control how long data in a table is retained independently of the database-level retention period. This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) #### Parameters ##### Query parameters `db`requiredstring The database name. `table`requiredstring The table name. `duration`requiredstring The retention period as a human-readable duration (for example, “30d”, “24h”, “1y”). Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/enterprise/configure/table/retention_period?db=DB&table=TABLE&duration=DURATION" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204Success. The table retention period has been set. 400Bad request. Invalid duration format. 401Unauthorized access. `data`object `error`string 404Database or table not found. DELETE`/api/v3/enterprise/configure/table/retention_period` ### Clear table retention period Removes the retention period from a specific table, reverting to the database-level retention period (or infinite retention if no database-level retention is set). This endpoint is only available in InfluxDB 3 Enterprise. #### Related * [influxdb3 update table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) #### Parameters ##### Query parameters `db`requiredstring The database name. `table`requiredstring The table name. Example request[Ask AI about this](#) ```sh curl --request DELETE \ "https://localhost:8181/api/v3/enterprise/configure/table/retention_period?db=DB&table=TABLE" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 204Success. The table retention period has been cleared. 401Unauthorized access. `data`object `error`string 404Database or table not found. #### Related * [Manage tables](/influxdb3/enterprise/admin/tables/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # User management Source: https://docs.influxdata.com/influxdb3/enterprise/api/user-management/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Manage users and their role assignments, and look up users and roles. These endpoints are only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by starting the server with `--without-user-auth false`. GET`/api/v3/roles_by_name/{roleName}` ### Get a role by name Returns the role with the specified name. Requires a token or user with permission to read roles. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `roleName`requiredstring The name of the role to retrieve. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/roles_by_name/{roleName}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the matching role. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the role was created. `description`string The role description. `id`requiredinteger \ The numeric ID of the role. `isRequiredRole`requiredboolean Whether the role is a required role. `name`requiredstring The role name. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the role was last updated. Example request body ```json { "createdAt": 1717000000, "id": 1, "isRequiredRole": false, "name": "read-write", "updatedAt": 1717000000 } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No role exists with the specified name. `data`object `error`string GET`/api/v3/roles/{id}/permissions` ### Get a role's permissions Returns the permissions granted to the role with the specified ID. Requires a token or user with permission to read roles. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The ID of the role. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/roles/{id}/permissions" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the role’s permissions. `permissions`requiredobject[] The permissions granted to the role. `actions`requiredstring[] The granted actions (snake\_case) for this resource type. `allResources`boolean Database permissions only. `true` when the permission applies to all databases. `identifiers`string[] Database permissions only. Specific database names the permission targets. Empty when `allResources` is `true`. `resourceType`requiredstring The resource category the permission applies to. Allowed:`database`, `token`, `user`, `role`, `admin_token` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No role exists with the specified ID. `data`object `error`string GET`/api/v3/users_by_oauth_id/{oauth_id}` ### Get a user by OAuth identity-provider ID Returns the user associated with the specified OAuth identity-provider ID. Requires the server to be configured with an OAuth validator. Requires a token or user with permission to read users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `oauth_id`requiredstring The OAuth identity-provider ID of the user. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/users_by_oauth_id/{oauth_id}" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the matching user. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the user was created. `deletedAt`integer \ The Unix timestamp (seconds) when the user was deleted. Present only for deleted users. `displayName`string The user’s display name. `oauthId`string The user’s OAuth identity-provider ID. Present when the user has an OAuth login identity. `operatorToken`string The operator token. Returned only when creating the initial user. `requiresPasswordReset`boolean Whether the user must reset their password on next login. Present when the user has a username and password login identity. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the user was last updated. `userId`requiredinteger \ The numeric ID of the user. `username`string The user’s username. Present when the user has a username and password login identity. Example request body ```json { "createdAt": 1717000000, "displayName": "Jane Smith", "requiresPasswordReset": false, "updatedAt": 1717000000, "userId": 1, "username": "jsmith" } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user matches the specified OAuth ID, or OAuth is not configured on the server. `data`object `error`string POST`/api/v3/users/{id}/require-password-reset` ### Require a user to reset their password on next login Marks the specified user as requiring a password reset. The user must set a new password before they are able to authenticate and receive a token. Requires a token or user with permission to update users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The numeric ID of the user. Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/users/{id}/require-password-reset" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the user with `requiresPasswordReset` set to `true`. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the user was created. `deletedAt`integer \ The Unix timestamp (seconds) when the user was deleted. Present only for deleted users. `displayName`string The user’s display name. `oauthId`string The user’s OAuth identity-provider ID. Present when the user has an OAuth login identity. `operatorToken`string The operator token. Returned only when creating the initial user. `requiresPasswordReset`boolean Whether the user must reset their password on next login. Present when the user has a username and password login identity. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the user was last updated. `userId`requiredinteger \ The numeric ID of the user. `username`string The user’s username. Present when the user has a username and password login identity. Example request body ```json { "createdAt": 1717000000, "displayName": "Jane Smith", "requiresPasswordReset": false, "updatedAt": 1717000000, "userId": 1, "username": "jsmith" } ``` 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user exists with the specified ID. `data`object `error`string GET`/api/v3/users/{id}/roles` ### List all roles assigned to a user Lists all roles assigned to the specified user. Requires a token or user with permission to read users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The numeric ID of the user. Example request[Ask AI about this](#) ```sh curl --request GET \ "https://localhost:8181/api/v3/users/{id}/roles" \ --header "Authorization: Bearer INFLUX_TOKEN" ``` #### Responses 200Success. Returns the list of roles assigned to the user. `items`requiredobject[] The roles assigned to the user. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the role was created. `description`string The role description. `id`requiredinteger \ The numeric ID of the role. `isRequiredRole`requiredboolean Whether the role is a required role. `name`requiredstring The role name. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the role was last updated. 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user exists with the specified ID. `data`object `error`string PUT`/api/v3/users/{id}/roles` ### Replace all roles assigned to a user Replaces all role assignments for the specified user with the provided list of role IDs. Roles not included in the request are removed from the user. Requires a token or user with permission to update users. This endpoint is only available in InfluxDB 3 Enterprise. User authentication is a preview feature enabled by setting `--without-user-auth false`. #### Parameters ##### Path parameters `id`requiredinteger \ The numeric ID of the user. #### Request bodyrequired Content-Type:`application/json` `roleIds`requiredinteger[] The complete set of role IDs to assign to the user. Example request body ```json { "roleIds": [ 1, 2 ] } ``` Example request[Ask AI about this](#) ```sh curl --request PUT \ "https://localhost:8181/api/v3/users/{id}/roles" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: application/json" \ --data-raw '{"roleIds":[1,2]}' ``` #### Responses 200Success. Returns the user’s updated role assignments. `items`requiredobject[] The roles assigned to the user. `createdAt`requiredinteger \ The Unix timestamp (seconds) when the role was created. `description`string The role description. `id`requiredinteger \ The numeric ID of the role. `isRequiredRole`requiredboolean Whether the role is a required role. `name`requiredstring The role name. `updatedAt`requiredinteger \ The Unix timestamp (seconds) when the role was last updated. 400 Request failed. Possible reasons: * Invalid database name * Malformed request body * Invalid timestamp precision `data`object `error`string 401Unauthorized access. `data`object `error`string 403Access denied. `data`object `error`string 404Not found. No user exists with the specified ID. `data`object `error`string #### Related * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) --- # Write data Source: https://docs.influxdata.com/influxdb3/enterprise/api/write-data/ [Download InfluxDB 3 Enterprise API Spec](/openapi/influxdb3-enterprise-openapi.yml) Write data to InfluxDB 3 Enterprise using line protocol format. #### Timestamp precision across write APIs InfluxDB 3 provides multiple write endpoints for compatibility with different InfluxDB versions. The following table compares timestamp precision support across v1, v2, and v3 write APIs: | Precision |v1 (`/write`)|v2 (`/api/v2/write`)|v3 (`/api/v3/write_lp`)| |------------------|-------------|--------------------|-----------------------| |**Auto detection**| ❌ No | ❌ No | ✅ `auto` (default) | | **Seconds** | ✅ `s` | ✅ `s` | ✅ `second` | | **Milliseconds** | ✅ `ms` | ✅ `ms` | ✅ `millisecond` | | **Microseconds** |✅ `u` or `µ` | ✅ `us` | ✅ `microsecond` | | **Nanoseconds** | ✅ `ns` | ✅ `ns` | ✅ `nanosecond` | | **Minutes** | ✅ `m` | ❌ No | ❌ No | | **Hours** | ✅ `h` | ❌ No | ❌ No | | **Default** | Nanosecond | Nanosecond | **Auto** (guessed) | All timestamps are stored internally as nanoseconds. POST`/api/v3/write_lp` ### Write line protocol Writes line protocol to the specified database. This is the native InfluxDB 3 Enterprise write endpoint that provides enhanced control over write behavior with advanced parameters for high-performance and fault-tolerant operations. Use this endpoint to send data in [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) format to InfluxDB. Use query parameters to specify options for writing data. #### Features * **Partial writes**: By default (`accept_partial=true`), InfluxDB writes the valid lines when some lines in a batch fail. Set `accept_partial=false` to reject the whole batch instead. * **Asynchronous writes**: Use `no_sync=true` to skip waiting for WAL synchronization, allowing faster response times but sacrificing durability guarantees * **Flexible precision**: Automatic timestamp precision detection with `precision=auto` (default) #### Column families (performance upgrade preview) With the [performance upgrade preview](/influxdb3/enterprise/performance-preview/) enabled (`--use-pacha-tree` flag), you can assign fields to column families using the `::` delimiter in field names. The portion before `::` is the family name; everything after is the field name. ```txt metrics,host=sA cpu::usage_user=55.2,cpu::usage_sys=12.1,mem::free=2048i 1000000000 ``` Fields in the same family are stored together on disk. For wide tables, this reduces I/O by letting queries read only the families they need. Fields written without `::` are assigned to auto-generated families. For more information, see [Column families](/influxdb3/enterprise/performance-preview/#column-families). #### Auto precision detection When you use `precision=auto` or omit the precision parameter, InfluxDB 3 automatically detects the timestamp precision based on the magnitude of the timestamp value: * Timestamps \< 5e9 → Second precision (multiplied by 1,000,000,000 to convert to nanoseconds) * Timestamps \< 5e12 → Millisecond precision (multiplied by 1,000,000) * Timestamps \< 5e15 → Microsecond precision (multiplied by 1,000) * Larger timestamps → Nanosecond precision (no conversion needed) #### Related * [Use the InfluxDB v3 write\_lp API to write data](/influxdb3/enterprise/write-data/http-api/v3-write-lp/) * [Performance upgrade preview](/influxdb3/enterprise/performance-preview/) #### Parameters ##### Query parameters `db`requiredstring `precision`string `accept_partial`boolean Whether to write the valid lines when some lines in the batch are rejected. Default is `true`: InfluxDB writes the valid lines and returns `400` with`"partial write of line protocol occurred"` and the rejected lines. With `false`, InfluxDB rejects the whole batch and writes nothing. Default:`true` `no_sync`boolean #### Request body Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v3/write_lp?db=DB" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: text/plain" \ --data-raw 'measurement,tag=value field=1 1234567890' ``` #### Responses 204Success (“No Content”). All data in the batch is written and queryable. 400Bad request. Some or all lines were rejected; the body lists each rejected line. With `accept_partial=true` (the default), the valid lines are written and the error is `"partial write of line protocol occurred"`. With `accept_partial=false`, nothing is written. In a database that uses explicit schemas (`schema_mode: explicit`, Enterprise only), a line that writes an undeclared table or column is rejected with a message that names it and points to `/api/v3/configure/table`. 401Unauthorized access. `data`object `error`string 403Access denied. 413Request entity too large. 422Unprocessable entity. The write would exceed a schema limit: the number of databases, tables, columns, tag columns, or fields. A partial write whose rejected lines hit a limit also returns `422` instead of `400`. POST`/api/v2/write` ### Write line protocol (v2-compatible) Writes line protocol to the specified database. This endpoint provides backward compatibility for InfluxDB 2.x write workloads using tools such as InfluxDB 2.x client libraries, the Telegraf `outputs.influxdb_v2` output plugin, or third-party tools. Use this endpoint to send data in [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) format to InfluxDB. Use query parameters to specify options for writing data. #### Related * [Use compatibility APIs to write data](/influxdb3/enterprise/write-data/http-api/compatibility-apis/) #### Parameters ##### Query parameters `bucket`requiredstring A database name. InfluxDB creates the database if it doesn’t already exist, and then writes all points in the batch to the database. This parameter is named `bucket` for compatibility with InfluxDB v2 client libraries. `precision`string The precision for unix timestamps in the line protocol batch. ##### Header parameters `Content-Type`string The content type of the request payload. `Content-Encoding`string The compression applied to the line protocol in the request payload. To send a gzip payload, pass `Content-Encoding: gzip` header. Allowed values:`gzip`, `identity` Default:`identity` `Content-Length`integer The size of the entity-body, in bytes, sent to InfluxDB. `Accept`string The content type that the client can understand. Writes only return a response body if they fail (partially or completely)–for example, due to a syntax problem or type mismatch. Allowed values:`application/json` Default:`application/json` #### Request body Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/api/v2/write?bucket=BUCKET" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: text/plain" \ --data-raw 'measurement,tag=value field=1 1234567890' ``` #### Responses 204Success (“No Content”). All data in the batch is written and queryable. 400Bad request. InfluxDB rejected the whole batch and wrote nothing. This endpoint doesn’t support partial writes: if any line is invalid, no lines are written, and an `accept_partial` parameter is ignored. To write the valid lines of a batch, use `/api/v3/write_lp`. 401Unauthorized access. `data`object `error`string 403Access denied. 413Request entity too large. POST`/write` ### Write line protocol (v1-compatible) Writes line protocol to the specified database. This endpoint provides backward compatibility for InfluxDB 1.x write workloads using tools such as InfluxDB 1.x client libraries, the Telegraf `outputs.influxdb` output plugin, or third-party tools. Use this endpoint to send data in [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) format to InfluxDB. Use query parameters to specify options for writing data. #### Related * [Use compatibility APIs to write data](/influxdb3/enterprise/write-data/http-api/compatibility-apis/) #### Parameters ##### Query parameters `db`requiredstring The name of the database. InfluxDB creates the database if it doesn’t already exist, and then writes all points in the batch to the database. `precision`string The precision for unix timestamps in the line protocol batch. `u`string Username for v1 compatibility authentication. When using Basic authentication or query string authentication, InfluxDB 3 ignores this parameter but allows any arbitrary string for compatibility with InfluxDB 1.x clients. `p`string Password for v1 compatibility authentication. For query string authentication, pass a database token with write permissions as this parameter. InfluxDB 3 checks that the `p` value is an authorized token. `rp`string Retention policy name. Honored but discouraged. InfluxDB 3 doesn’t use retention policies. `consistency`string Write consistency level. Ignored by InfluxDB 3. Provided for compatibility with InfluxDB 1.x clients. ##### Header parameters `Authorization`string Authorization header for token-based authentication. Supported schemes: * `Bearer AUTH_TOKEN` - OAuth bearer token scheme * `Token AUTH_TOKEN` - InfluxDB v2 token scheme * `Basic ` - Basic authentication (username is ignored) `Content-Type`string The content type of the request payload. `Accept`string The content type that the client can understand. Writes only return a response body if they fail (partially or completely)–for example, due to a syntax problem or type mismatch. Allowed values:`application/json` Default:`application/json` `Content-Encoding`string The compression applied to the line protocol in the request payload. To send a gzip payload, pass `Content-Encoding: gzip` header. `Content-Length`string The size of the entity-body, in bytes, sent to InfluxDB. #### Request body Content-Type:`application/json` Example request[Ask AI about this](#) ```sh curl --request POST \ "https://localhost:8181/write?db=DB" \ --header "Authorization: Bearer INFLUX_TOKEN" \ --header "Content-Type: text/plain" \ --data-raw 'measurement,tag=value field=1 1234567890' ``` #### Responses 204Success (“No Content”). All data in the batch is written and queryable. 400Bad request. InfluxDB rejected the whole batch and wrote nothing. This endpoint doesn’t support partial writes: if any line is invalid, no lines are written, and an `accept_partial` parameter is ignored. The body describes the first rejected line. To write the valid lines of a batch, use `/api/v3/write_lp`. 401Unauthorized access. `data`object `error`string 403Access denied. 413Request entity too large. #### Related * [Write data using HTTP APIs](/influxdb3/enterprise/write-data/http-api/) * [Line protocol reference](/influxdb3/enterprise/reference/syntax/line-protocol/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) | Precision | v1 ( /write ) | v2 ( /api/v2/write ) | v3 ( /api/v3/write_lp ) | | --- | --- | --- | --- | | Precision | v1 ( /write ) | v2 ( /api/v2/write ) | v3 ( /api/v3/write_lp ) | | Auto detection | ❌ No | ❌ No | ✅ auto (default) | | Seconds | ✅ s | ✅ s | ✅ second | | Milliseconds | ✅ ms | ✅ ms | ✅ millisecond | | Microseconds | ✅ u or µ | ✅ us | ✅ microsecond | | Nanoseconds | ✅ ns | ✅ ns | ✅ nanosecond | | Minutes | ✅ m | ❌ No | ❌ No | | Hours | ✅ h | ❌ No | ❌ No | | Default | Nanosecond | Nanosecond | Auto (guessed) | --- # Edge Data Replication (EDR) for InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/edr/ * EDR 1.0.0 Edge Data Replication (EDR) moves time series data from edge InfluxDB 3 Enterprise instances up through regional relays to a central hub, over links that might be intermittent or bandwidth-constrained. EDR runs as a separate agent process alongside InfluxDB 3 Enterprise and handles arbitrarily long disconnections—when the link comes back, data catches up automatically. > [!Note] > #### Not the same as replication streams in InfluxDB v2 and InfluxDB Cloud > > This EDR product for InfluxDB 3 Enterprise is unrelated to[Edge Data Replication in InfluxDB v2 and InfluxDB Cloud](/influxdb/v2/write-data/replication/), > which forwards writes from a local OSS bucket to a remote bucket. The two > features share a name and general category (edge-to-cloud replication) but > have different architectures, guarantees, and configuration. If you’re > looking for bucket-level write forwarding in InfluxDB v2 or Cloud, see[Edge Data Replication](/influxdb/v2/write-data/replication/) in the > InfluxDB v2 documentation instead. ## What is EDR? EDR provides durable, observable replication of time series data between InfluxDB 3 Enterprise instances—and from InfluxDB 3 Enterprise to InfluxDB 3 Cloud or AWS Timestream for InfluxDB 3—over connections that might be intermittent, unreliable, or bandwidth-constrained. | Property | Detail | |------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |Delivery guarantee|At-least-once. Data is not lost during disconnection; data that outlives [write-ahead log (WAL)](/influxdb3/enterprise/edr/reference/glossary/#wal-write-ahead-log) retention is recovered from compacted files.| | Ordering | WAL-file order preserved per ingest node (serial pipeline by default). | | Direction | Push from upstream (source) to downstream (destination). The source initiates all transfers. | | Wire format | Line Protocol over HTTP (default) or zstd-compressed [PT](/influxdb3/enterprise/reference/internals/storage-engine/#new-file-format) binary format (agent-to-agent, \~2.5x smaller). | | Topology | Tree-shaped: edges to regionals to central. Each hop is independent. | | Observability | Per-node UI, Prometheus metrics, JSON metrics API, Server-Sent Events (SSE) real-time updates. | EDR is not a synchronous replication layer, a conflict resolution system, a backup tool, or a transformation layer. For the full list of what EDR is and isn’t, see [Limitations](/influxdb3/enterprise/edr/reference/limitations/). ## Who EDR replicates between EDR always replicates **from** InfluxDB 3 Enterprise. It replicates **to**: * InfluxDB 3 Enterprise * [InfluxDB 3 Cloud](/influxdb3/cloud/) * AWS Timestream for InfluxDB 3 InfluxDB 3 Core, InfluxDB Cloud Serverless, InfluxDB Cloud Dedicated, and InfluxDB Clustered are not supported as an EDR source or destination. ## Get started * [Install EDR](/influxdb3/enterprise/edr/install/): Install the agent and CLI binaries. * [Get started with EDR](/influxdb3/enterprise/edr/get-started/): Replicate between two InfluxDB 3 Enterprise instances on one host. * [Run the signals demo](/influxdb3/enterprise/edr/demo/): A fuller 4-node topology. * [Replicate with EDR](/influxdb3/enterprise/edr/replicate/): Configure replication to Enterprise, Cloud, or multiple destinations. * [Monitor EDR](/influxdb3/enterprise/edr/monitor/): Watch replication health, lag, and progress. * [Troubleshoot EDR](/influxdb3/enterprise/edr/troubleshoot/): Diagnose something that looks wrong. * [Reference](/influxdb3/enterprise/edr/reference/): Architecture, configuration schema, CLI, API, and compatibility reference. > [!Note] > #### Getting EDR > > Edge Data Replication requires coordination with InfluxData.[Contact the InfluxData Sales team](https://www.influxdata.com/contact-sales/) to discuss your replication requirements and deployment. [Get started with EDR](/influxdb3/enterprise/edr/get-started/) | Property | Detail | | --- | --- | | Property | Detail | | Delivery guarantee | At-least-once. Data is not lost during disconnection; data that outlives write-ahead log (WAL) retention is recovered from compacted files. | | Ordering | WAL-file order preserved per ingest node (serial pipeline by default). | | Direction | Push from upstream (source) to downstream (destination). The source initiates all transfers. | | Wire format | Line Protocol over HTTP (default) or zstd-compressed PT binary format (agent-to-agent, ~2.5x smaller). | | Topology | Tree-shaped: edges to regionals to central. Each hop is independent. | | Observability | Per-node UI, Prometheus metrics, JSON metrics API, Server-Sent Events (SSE) real-time updates. | --- # Run the EDR signals demo Source: https://docs.influxdata.com/influxdb3/enterprise/edr/demo/ * EDR 1.0.0 The signals demo is the fastest way to see EDR’s store-and-forward, multi-tier relay, and at-least-once delivery in one topology: two edges replicating up through a regional relay to a central hub, entirely in Docker. The demo is a self-contained bundle that you unzip and run with Docker Compose—see [Run it](#run-it). 1. [How EDR sources data](#how-edr-sources-data) 2. [The topology](#the-topology) 3. [The three config shapes](#the-three-config-shapes) 4. [Run it](#run-it) 5. [What this demonstrates](#what-this-demonstrates) 6. [Next](#next) If you haven’t set up EDR yet, start with[Get started with EDR](/influxdb3/enterprise/edr/get-started/), which walks through a simpler two-node case on one host. ## How EDR sources data EDR reads the files that InfluxDB 3 Enterprise’s storage engine writes. For what WAL, PT, gen0, cv2, and `.ptsnap` mean, see[Storage engine terms](/influxdb3/enterprise/edr/reference/architecture/#storage-engine-terms). ### Primary source: WAL files InfluxDB 3 Enterprise, on the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters on 3.11+; on 3.10.x enable it with`--upgrade-pacha-tree`), flushes incoming data to write-ahead log (WAL) files on the object store roughly every second—the flush interval, sooner under heavy ingest as the buffer fills. Snapshots later roll these already-persisted WAL files up into compacted files. EDR’s WAL replicator discovers new files as they appear and ships them downstream. ### Secondary source: compacted PT files The upgraded storage engine promotes WAL data through compaction levels: 1. A write arrives. 2. The storage engine creates a WAL file. 3. A snapshot creates a gen0 PT file from the WAL data and records it in a `.ptsnap` manifest. 4. Compaction progressively promotes gen0 data to cv2 files at level L1 and higher. 5. Retention expiry eventually deletes the data. After a WAL file is snapshotted, it becomes eligible for deletion. If EDR was down or disconnected when that happened, the WAL file might be gone, but the data still exists in gen0 or a higher compaction level (cv2 files). Historic fill and gap fill recover data from these files. For more information, see[Historic fill and gap fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill)and [Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/), which covers sizing the InfluxDB flag that governs this recovery window and the trade-offs of raising it. ## The topology Two edges replicate up through a regional relay to a central hub, entirely in Docker. flowchart LR Mumbai["Mumbai\\nupstream · source\\nInfluxDB 3 + EDR"] -- EDR --\> Singapore Chennai["Chennai\\nupstream · source\\nInfluxDB 3 + EDR"] -- EDR --\> Singapore Singapore["Singapore\\nregional · relay\\nreceives + forwards"] -- EDR --\> London["London\\ndownstream · sink\\nInfluxDB 3 + EDR"] Four nodes, each a pair of containers—an InfluxDB 3 Enterprise instance and an EDR agent beside it. The three roles are the *only* three an EDR agent ever takes: | Node | Role | What its config has | |-------------------|------------------------------------------|--------------------------------------| |`mumbai`, `chennai`| **upstream-only**—a pure source |a `downstream:` block (where it sends)| | `singapore` |**regional relay**—receives *and* forwards| both `upstreams:` and `downstream:` | | `london` | **downstream-only**—a pure sink |an `upstreams:` list (who sends to it)| A fifth container, the **dashboard**, generates the test signals, queries every tier, serves the UI, and runs the channel relay. It is demo scaffolding, not part of EDR. ## The three config shapes Every EDR agent uses one of these three shapes. The `auth_token` on each sender’s `downstream` matches an `upstreams` entry on the receiver—that pairing is how a receiver knows who connected. **Edge (upstream-only)**—`mumbai` (Chennai is identical with its own name and token): ```yaml name: mumbai downstream: name: singapore address: http://singapore-edr:9090 # the next hop's EDR --listen auth_token: mumbai-auth # token-store key historic_fill: { mode: none } # live-only for the demo ``` **Regional relay**—`singapore` receives from both edges, forwards to the hub: ```yaml name: singapore upstreams: - { name: mumbai, auth_token: mumbai-auth, write_token: singapore-token } - { name: chennai, auth_token: chennai-auth, write_token: singapore-token } downstream: name: london address: http://london-edr:9090 auth_token: singapore-auth historic_fill: { mode: none } ``` **Hub (downstream-only)**—`london`, the sink: ```yaml name: london upstreams: - { name: singapore, auth_token: singapore-auth, write_token: london-token } ``` Every EDR deployment composes these three shapes into a tree, however large. For a two-node version, see[Get started with EDR](/influxdb3/enterprise/edr/get-started/). > [!Note] > #### The dashboard relay is demo-only > > Above, each sender’s `downstream.address` is the next hop’s EDR > directly—the production shape. The demo instead points each`downstream.address` at the dashboard’s relay, which transparently > forwards to the same receiver, only so the dashboard can cut and restore a > channel. It changes nothing about how EDR works; in production you point > straight at the next hop. ## Run it The demo ships as its own zip, separate from the agent binaries and container image—see [EDR](/influxdb3/enterprise/edr/) for how to obtain it if you don’t have it yet. Unzip it; the archive extracts to a `signals-demo/`directory. `cd` into that directory and follow its `README.md`—that’s the canonical guide, covering prerequisites, licensing (trial or file-based), driving the dashboard (start and stop signals, cut and restore channels), the `edr-inspect` triage tooling, ports, and layout. In brief, from inside `signals-demo/`: ```bash cp .env.example .env # set the Enterprise image + the four license emails docker compose up --build # then open http://localhost:8090 ``` You’ll watch three signals propagate across the tiers—and, by cutting a channel, watch a downstream node freeze and then catch up with no loss when the link is restored. ## What this demonstrates * **Store-and-forward** across intermittent links—data written while a hop is down is queued and delivered when it returns. * **Multi-tier relay**—a regional node that both receives and forwards (and forwards its own locally-written data too). * **At-least-once delivery** with automatic catch-up; no data lost on disconnection. The demo runs on defaults with no tuning. Before you size a production deployment, see [Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/)for the two settings worth setting deliberately. ## Next * [Replicate with EDR](/influxdb3/enterprise/edr/replicate/)—configure production topologies to Enterprise, Cloud, or multiple destinations. * [Monitor EDR](/influxdb3/enterprise/edr/monitor/)—health, historic fill, and gap fill. * [Troubleshoot EDR](/influxdb3/enterprise/edr/troubleshoot/). [Get started with EDR](/influxdb3/enterprise/edr/get-started/) | Node | Role | What its config has | | --- | --- | --- | | Node | Role | What its config has | | mumbai , chennai | upstream-only —a pure source | a downstream: block (where it sends) | | singapore | regional relay —receives and forwards | both upstreams: and downstream: | | london | downstream-only —a pure sink | an upstreams: list (who sends to it) | --- # Get started with EDR Source: https://docs.influxdata.com/influxdb3/enterprise/edr/get-started/ * EDR 1.0.0 This guide takes you from two running InfluxDB 3 Enterprise instances to a replicated point you can query at the destination. It uses EDR’s native binaries and one host with two instances, so you don’t need a second machine to see EDR work. 1. [Before you begin](#before-you-begin) 2. [Create a token store](#create-a-token-store) 3. [Configure the source node](#configure-the-source-node) 4. [Configure the destination node](#configure-the-destination-node) 5. [Start the agents](#start-the-agents) 6. [Verify replication](#verify-replication) 7. [Test recovery](#test-recovery) 8. [Next](#next) ## Before you begin Start two InfluxDB 3 Enterprise instances on the host, each on the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)and its own data directory and port—one to act as the EDR source, one as the destination. If you don’t have InfluxDB 3 Enterprise running yet, see[Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/). Also [install the EDR binaries](/influxdb3/enterprise/edr/install/) on the host. If you’d rather run EDR in containers, see[Run EDR in Docker](/influxdb3/enterprise/edr/install/docker/) instead of the native steps below—the configuration is the same either way. EDR runs as a separate agent process (`influxdb3-edr`) alongside each InfluxDB 3 Enterprise instance, reading its object store directly. The InfluxDB binary itself is unmodified. For the full component and topology model, see [EDR architecture](/influxdb3/enterprise/edr/reference/architecture/). ## Create a token store Each agent resolves its tokens from a token store: a directory of plain-text files, one file per token. ```bash mkdir -p /etc/edr/secrets # Auth token—identifies the source to the destination. openssl rand -hex 32 > /etc/edr/secrets/source-auth-token # Write token—used by the destination's EDR agent to write into its # local InfluxDB. Typically the destination's InfluxDB admin/operator token. cp /path/to/destination-write-token /etc/edr/secrets/dest-write-token ``` For rotation, ownership, and Vault guidance, see[Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). ## Configure the source node ```yaml # /etc/edr/source-config.yaml name: "source" downstream: name: "destination" address: "http://localhost:9190" # the destination agent's --listen auth_token: "source-auth-token" # name of file in token store historic_fill: mode: none # required—choose: none | full | since ``` `historic_fill` is required on every `downstream` entry—the agent refuses to start without it. This walkthrough uses `mode: none` (live-only): EDR replicates only data written after the agent starts, and doesn’t backfill data already on the source. `full` backfills all pre-existing data before switching to live; `since` backfills data written after a given time. For the complete semantics, see[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/). ## Configure the destination node ```yaml # /etc/edr/destination-config.yaml name: "destination" upstreams: - name: "source" auth_token: "source-auth-token" # verifies the source's identity write_token: "dest-write-token" # used to write into local InfluxDB ``` A node needs at least one of the following: * `downstream` or `downstreams`—where it sends data. * `upstreams`—who sends it data. A node with both is a regional relay—see[Replicate to InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/replicate/to-enterprise/)for multi-hop topologies. ## Start the agents ```bash # Source node: influxdb3-edr \ --config /etc/edr/source-config.yaml \ --data-dir SOURCE_INSTANCE_DATA_DIR \ --token-store /etc/edr/secrets \ --listen 0.0.0.0:9090 \ --state-location /var/lib/edr/source-state # Destination node: EDR_WRITE_ENDPOINT="http://localhost:DESTINATION_INSTANCE_PORT" \ influxdb3-edr \ --config /etc/edr/destination-config.yaml \ --data-dir DESTINATION_INSTANCE_DATA_DIR \ --token-store /etc/edr/secrets \ --listen 0.0.0.0:9190 \ --state-location /var/lib/edr/destination-state ``` * `SOURCE_INSTANCE_DATA_DIR`: the source InfluxDB instance’s data directory * `DESTINATION_INSTANCE_PORT`: the destination InfluxDB instance’s HTTP port * `DESTINATION_INSTANCE_DATA_DIR`: the destination InfluxDB instance’s data directory Each `influxdb3-edr` agent needs its own `--data-dir` (pointing at its InfluxDB instance’s object store), `--listen` address, and`--state-location`, so the two agents on one host don’t collide. See[Install EDR](/influxdb3/enterprise/edr/install/#configure-edr) for what each flag does. ## Verify replication Use each instance’s own InfluxDB admin or database token—not the EDR auth and write tokens from [Create a token store](#create-a-token-store), which authenticate the agents to each other, not you to InfluxDB. 1. Write a point to the source instance: ``` curl -X POST "http://localhost:SOURCE_INSTANCE_PORT/api/v3/write_lp?db=getting_started" \ --header "Authorization: Bearer SOURCE_INSTANCE_ADMIN_TOKEN" \ --data-raw "edr_test,host=source value=1" ``` * `SOURCE_INSTANCE_PORT`: the source InfluxDB instance’s HTTP port * `SOURCE_INSTANCE_ADMIN_TOKEN`: an admin or database token for the source instance 2. Query the destination instance for the same point: ``` curl -G "http://localhost:DESTINATION_INSTANCE_PORT/api/v3/query_sql" \ --header "Authorization: Bearer DESTINATION_INSTANCE_ADMIN_TOKEN" \ --data-urlencode "db=getting_started" \ --data-urlencode "q=SELECT * FROM edr_test" ``` * `DESTINATION_INSTANCE_PORT`: the destination InfluxDB instance’s HTTP port * `DESTINATION_INSTANCE_ADMIN_TOKEN`: an admin or database token for the destination instance The point EDR replicated from the source appears in the result. 3. Open `http://127.0.0.1:9091/ui` on the source host to watch the replication—the embedded UI shows topology, health, and lag. The UI and`/metrics` bind to loopback by default (`--observability-listen`); expose them deliberately if you need off-host access. See[Monitor EDR](/influxdb3/enterprise/edr/monitor/). ## Test recovery EDR delivers at-least-once, even across a destination outage: 1. Stop the destination agent. 2. Write more points to the source. 3. Restart the destination agent. The destination catches up on the backlog it missed while stopped—no data written at the source is lost. See[EDR state and recovery](/influxdb3/enterprise/edr/reference/state-and-recovery/) for what the agent persists to make this possible. If an agent crashes rather than stopping cleanly, see the[log post-mortem playbook](/influxdb3/enterprise/edr/troubleshoot/log-postmortem-playbook/)to reconstruct what happened from its captured log. ## Next * [Run the signals demo](/influxdb3/enterprise/edr/demo/)—a fuller 4-node topology (edge to regional relay to central hub) that also demonstrates an intermittent link recovering. * [Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/)— scope and bandwidth scheduling for production. * [Replicate to multiple destinations](/influxdb3/enterprise/edr/replicate/to-multiple-destinations/)— fan out one source to several destinations at once. * [Monitor EDR](/influxdb3/enterprise/edr/monitor/)—health, historic fill, and gap fill. * [Troubleshoot EDR](/influxdb3/enterprise/edr/troubleshoot/). [Install EDR](/influxdb3/enterprise/edr/install/)[Run the signals demo](/influxdb3/enterprise/edr/demo/) --- # Install EDR Source: https://docs.influxdata.com/influxdb3/enterprise/edr/install/ * EDR 1.0.0 EDR ships as pre-built binaries. You don’t build anything from source. 1. [Binaries](#binaries) 2. [Requirements](#requirements) 3. [Confirm your version](#confirm-your-version) 4. [Configure EDR](#configure-edr) 5. [Run EDR in Docker](#run-edr-in-docker) ## Binaries | Binary | What it is | |---------------|----------------------------------------------------------------------------------------------------------------------------------------| | `influxdb3` | InfluxDB 3 Enterprise server (built and shipped by InfluxData). | |`influxdb3-edr`| The EDR agent. Runs one process per node that participates in replication. | | `edr-inspect` |Read-only triage CLI (`state`, `metrics`, and `topology`). See [Monitor EDR](/influxdb3/enterprise/edr/monitor/#triage-cli-edr-inspect).| Place the binaries on each host—on `PATH`, or referenced by an absolute path in your service definitions. EDR has no other runtime dependencies. > [!Note] > #### EDR version support > > EDR 1.0.0 supports InfluxDB 3 Enterprise 3.11.x and 3.12.x. > InfluxDB 3 Enterprise 3.10.x is unvalidated and unsupported for GA. See[Compatibility](/influxdb3/enterprise/edr/reference/compatibility/) for the > supported storage-format matrix. ## Requirements The `influxdb3` server must run the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters on 3.11+. For an existing 3.10 deployment, follow the[staged upgrade](/influxdb3/enterprise/edr/reference/compatibility/#upgrade-a-310-source-and-a-020-agent) before installing EDR 1.0.0. EDR reads the upgraded engine’s object store directly, so without it,[write-ahead log (WAL)](/influxdb3/enterprise/edr/reference/glossary/#wal-write-ahead-log)files are in a format EDR ignores. Run one `influxdb3-edr` agent per node that sources or receives data. Each agent needs a token store directory and a config file—see[Configure EDR](#configure-edr). ## Confirm your version ```bash influxdb3 --version influxdb3-edr --version ``` ## Configure EDR Every `influxdb3-edr` agent needs a config file, a token store, and the following CLI flags: ```bash influxdb3-edr \ --config /etc/edr/config.yaml \ --data-dir /var/lib/influxdb3 \ --token-store /etc/edr/secrets \ --state-location /var/lib/edr/state \ --listen 0.0.0.0:9090 ``` | Flag | Description | |------------------|------------------------------------------------------------------------------------------------------------------------------| | `--data-dir` | InfluxDB data directory (the shared object store). | | `--token-store` | Path to the token store directory. See [Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). | |`--state-location`|Where the agent persists replication state. See [State and recovery](/influxdb3/enterprise/edr/reference/state-and-recovery/).| | `--listen` | Network listener for the replication protocol. Defaults to `0.0.0.0:9090`. | For the replication topology config (`downstream:` and `upstreams:`), see[Get started](/influxdb3/enterprise/edr/get-started/). For the complete CLI flag reference, see [CLI reference](/influxdb3/enterprise/edr/reference/cli/). For the complete config file schema, see[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/). ## Run EDR in Docker The agent and `edr-inspect` also ship as a container image. For operating that image—ports, mounts, the environment contract, and triage through `docker exec`—see [Run EDR in Docker](/influxdb3/enterprise/edr/install/docker/). ## Run EDR on Kubernetes Use the supported Helm chart to deploy a sender, receiver, or combined relay alongside an existing Enterprise release. See [Install EDR on Kubernetes](/influxdb3/enterprise/edr/install/kubernetes/). [Get started with EDR](/influxdb3/enterprise/edr/get-started/) | Binary | What it is | | --- | --- | | Binary | What it is | | influxdb3 | InfluxDB 3 Enterprise server (built and shipped by InfluxData). | | influxdb3-edr | The EDR agent. Runs one process per node that participates in replication. | | edr-inspect | Read-only triage CLI ( state , metrics , and topology ). See Monitor EDR . | | Flag | Description | | --- | --- | | Flag | Description | | --data-dir | InfluxDB data directory (the shared object store). | | --token-store | Path to the token store directory. See Manage tokens . | | --state-location | Where the agent persists replication state. See State and recovery . | | --listen | Network listener for the replication protocol. Defaults to 0.0.0.0:9090 . | --- # Run EDR in Docker Source: https://docs.influxdata.com/influxdb3/enterprise/edr/install/docker/ * EDR 1.0.0 Running EDR in a container is identical to a native deployment for everything covered in [Install EDR](/influxdb3/enterprise/edr/install/),[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/)(scope, historic and gap fill), [Monitor EDR](/influxdb3/enterprise/edr/monitor/)(the health model), and [Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). This page covers only the**Docker-specific** items: container layout, the environment contract, exposing the operator surfaces, and triage through `docker exec`. For**building** the image, see `docker/README.md` in the source repository. This assumes you already have: * the **EDR agent image** loaded locally (`edr:-pro-`), and * the **InfluxDB 3 Enterprise** image (consumed separately—not built here). A complete worked composite app (Enterprise InfluxDB with an EDR sender and receiver, with configs, tokens, and a `compose.yaml`) lives under `demo/docker-2node-full/` in the source repository. ## Container layout **Ports** | Port | Surface | |------|---------------------------------------------------------------------------------| |`9090`|inbound replication protocol (`/edr/v1/...`) and `/health`—upstreams connect here| |`9091`| operator surfaces (web UI, `/metrics`, JSON API)—**loopback by default** | **Mount points**—the container runs as uid and gid `1500`, matching the InfluxDB 3 Enterprise server image so a shared file object-store volume is readable and writable by both: | Path | Contents | Agent flag | |----------------------|--------------------------------------------------------------|------------------| | `/var/lib/influxdb3` |shared object store (the source’s upgraded-storage-engine WAL)| `--data-dir` | | `/var/lib/edr/state` | EDR state journals (`wal_cursor.json`, …) |`--state-location`| |`/var/lib/edr/secrets`| token store (auth and write tokens) | `--token-store` | | `/etc/edr` | the replication config YAML | `--config` | **Entrypoint**—auto-prepends `influxdb3-edr` (so you can pass just flags) and expands `${VAR}` references in arguments, so config paths and listen addresses can be templated from the environment.`INFLUXDB3_UNSET_VARS="FOO BAR"` scrubs vars before launch. ## Environment contract These are **baked into the image as defaults**, and **both** the agent and`edr-inspect` read them—so a single setting drives both, and `docker exec`triage needs no arguments (see below). Override them at `docker run -e ...`or in compose; the agent’s CLI flags still take precedence over env. | Var | Default | Read by | |------------------------------------|--------------------|---------------------------| | `INFLUXDB3_EDR_LISTEN` | `0.0.0.0:9090` | agent | |`INFLUXDB3_EDR_OBSERVABILITY_LISTEN`| `127.0.0.1:9091` |agent **and** `edr-inspect`| | `INFLUXDB3_EDR_STATE_LOCATION` |`/var/lib/edr/state`|agent **and** `edr-inspect`| | `RUST_LOG` | `info` | agent | > [!Important] > Drive these through env, not the agent’s CLI flags, when you want`edr-inspect` to agree with the agent automatically. If you set, say, the > observability address through a CLI flag only, the env still holds the > default and `edr-inspect` would target the wrong port. ### Exposing the UI and metrics The operator surfaces (`/ui`, `/metrics`, JSON API) bind to **loopback by default** (secure by default). To reach them from the host or another container, set `INFLUXDB3_EDR_OBSERVABILITY_LISTEN=0.0.0.0:9091` and publish the port (`-p 9091:9091`). Triage through `docker exec` does **not**require this—it works on loopback inside the container regardless. ## Triage with `edr-inspect` through `docker exec` `edr-inspect` is a **read-only** triage CLI shipped in the same image. Run it inside a running EDR container—it picks up the state-location and observability address from the container environment (the same vars the agent reads), so it needs **no arguments**: ```bash docker exec edr-inspect state # state journals: live repl / gap fill / historic docker exec edr-inspect metrics # live /metrics, scrolling watch (--once for one block) docker exec edr-inspect topology # upstream/downstream diagram + per-edge health ``` * **`state`** reads `$INFLUXDB3_EDR_STATE_LOCATION`. It is **offline**—it works even if the agent is wedged or stopped, as long as the journals are on the mounted state volume. Reports three sections: live replication (WAL cursor), live gap fill (gap ledger), historic replication (manifest progress). * **`metrics`** and **`topology`** read`$INFLUXDB3_EDR_OBSERVABILITY_LISTEN` and reach the agent over loopback**inside the same container**—so they work with the default loopback-only binding (no need to expose `9091`). * Override per-invocation if you need a different target:`edr-inspect state ` (a path, or an `s3://`, `gs://`, or `az://` URL), `edr-inspect metrics --once`. `edr-inspect` needs no `--data-dir`—it reads only the state journals and the agent’s `/metrics`, never the object store. ## Docker-specific troubleshooting | Symptom | Cause and fix | |--------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | EDR can’t read the source WAL, or `state` shows nothing |The agent’s `--data-dir` and `--state-location` volumes must be the **same** the InfluxDB server (and any prior EDR run) wrote to. With a shared file object store, mount the **same named volume** into both containers.| | Permission denied on a mounted volume | The container runs as uid `1500`. On bind mounts from a Linux host, `chown -R 1500:1500 ` (Docker Desktop/macOS does not enforce host uid, so it usually “just works” there). | | Container exits immediately | The default `CMD` is `--help`. Pass real flags (`--config ...`) or an env-templated command; the entrypoint prepends `influxdb3-edr`. Check `docker logs `. | | `edr-inspect` reports “no EDR agent reachable” | The observability address it derived doesn’t match where the agent bound. Set `INFLUXDB3_EDR_OBSERVABILITY_LISTEN` (env) so both agree, or pass the address positionally. | | `edr-inspect state` errors “required argument” | Neither a positional nor `$INFLUXDB3_EDR_STATE_LOCATION` was set—set the env (it is by default in this image) or pass the path. | |Container connects to host InfluxDB (EDR in Docker, InfluxDB native)| Reach the host as `host.docker.internal` (add `--add-host host.docker.internal:host-gateway` on Linux); set `EDR_WRITE_ENDPOINT` or the upstream address accordingly. | | Replication uses a stale image | `docker pull` or `docker load` the new tag and recreate the container—a running container keeps its old image layers. | For non-Docker symptoms (config errors, data not flowing, replication lag, the halted state, log messages), see[Troubleshoot EDR](/influxdb3/enterprise/edr/troubleshoot/). | Port | Surface | | --- | --- | | Port | Surface | | 9090 | inbound replication protocol ( /edr/v1/... ) and /health —upstreams connect here | | 9091 | operator surfaces (web UI, /metrics , JSON API)— loopback by default | | Path | Contents | Agent flag | | --- | --- | --- | | Path | Contents | Agent flag | | /var/lib/influxdb3 | shared object store (the source’s upgraded-storage-engine WAL) | --data-dir | | /var/lib/edr/state | EDR state journals ( wal_cursor.json , …) | --state-location | | /var/lib/edr/secrets | token store (auth and write tokens) | --token-store | | /etc/edr | the replication config YAML | --config | | Var | Default | Read by | | --- | --- | --- | | Var | Default | Read by | | INFLUXDB3_EDR_LISTEN | 0.0.0.0:9090 | agent | | INFLUXDB3_EDR_OBSERVABILITY_LISTEN | 127.0.0.1:9091 | agent and edr-inspect | | INFLUXDB3_EDR_STATE_LOCATION | /var/lib/edr/state | agent and edr-inspect | | RUST_LOG | info | agent | | Symptom | Cause and fix | | --- | --- | | Symptom | Cause and fix | | EDR can’t read the source WAL, or state shows nothing | The agent’s --data-dir and --state-location volumes must be the same the InfluxDB server (and any prior EDR run) wrote to. With a shared file object store, mount the same named volume into both containers. | | Permission denied on a mounted volume | The container runs as uid 1500 . On bind mounts from a Linux host, chown -R 1500:1500 (Docker Desktop/macOS does not enforce host uid, so it usually “just works” there). | | Container exits immediately | The default CMD is --help . Pass real flags ( --config ... ) or an env-templated command; the entrypoint prepends influxdb3-edr . Check docker logs . | | edr-inspect reports “no EDR agent reachable” | The observability address it derived doesn’t match where the agent bound. Set INFLUXDB3_EDR_OBSERVABILITY_LISTEN (env) so both agree, or pass the address positionally. | | edr-inspect state errors “required argument” | Neither a positional nor $INFLUXDB3_EDR_STATE_LOCATION was set—set the env (it is by default in this image) or pass the path. | | Container connects to host InfluxDB (EDR in Docker, InfluxDB native) | Reach the host as host.docker.internal (add --add-host host.docker.internal:host-gateway on Linux); set EDR_WRITE_ENDPOINT or the upstream address accordingly. | | Replication uses a stale image | docker pull or docker load the new tag and recreate the container—a running container keeps its old image layers. | --- # Install EDR on Kubernetes Source: https://docs.influxdata.com/influxdb3/enterprise/edr/install/kubernetes/ * EDR 1.0.0 The `influxdb3-edr-adjunct` Helm chart deploys a separate EDR agent alongside an unmodified [InfluxDB 3 Enterprise Helm release](https://github.com/influxdata/helm-charts/tree/master/charts/influxdb3-enterprise). The sender reads the source’s object store directly. It does not inject a sidecar or change the server release. The agent stores its cursor and journals in its own object-store location, separate from source data. It needs no state PersistentVolumeClaim (PVC). ## Prerequisites * An InfluxDB 3 Enterprise source on a [supported server line](/influxdb3/enterprise/edr/reference/compatibility/), using the upgraded storage engine and S3, Google Cloud Storage, or Azure Blob Storage. * The EDR 1.0.0 image in a registry your Kubernetes nodes can pull from. Select the `amd64` or `arm64` image for your nodes. * The EDR Helm bundle `edr-helm-v1.0.0.tgz` and its example values files.[Contact the InfluxData Sales team](https://www.influxdata.com/contact-sales/) for artifacts and deployment coordination. * Durable agent state storage and credentials that can access it. Senders also need source object-store access. * A destination write token and a shared authentication token for each sender/receiver link. Load the delivered image archive before pushing the required architecture’s image to your registry: ```bash docker load --input edr-images-v1.0.0.tar.gz ``` Use image tag `1.0.0-pro7dd361197e-amd64` or `1.0.0-pro7dd361197e-arm64`. Set `image.repository` to your registry’s image path in each values file. ## Choose a role | `role` | Operation | Required configuration | |----------|----------------------------------------------------------------------|-----------------------------------------------| | `sender` | Reads a source store and sends data downstream. | `source`, `downstream`, and `state` | |`receiver`| Accepts replication and writes to a destination server. | `receiver` and `state` | |`combined`|Receives data into a local server and sends that server’s data onward.|`source`, `downstream`, `receiver`, and `state`| A sender-to-receiver pipeline uses two releases of the EDR chart. Copy the corresponding example from the delivered chart into your own values file. Keep the chart’s `values.yaml` as a reference. Extract the archive to access its examples: ```bash tar --extract --gzip --file edr-helm-v1.0.0.tgz ``` The S3 examples below assume the Enterprise releases are named `idb-src` and `idb-dst` in the same namespace. Replace bucket names, registry paths, and Secret names for your deployment. > [!Important] > Chart defaults assert `idempotentWrites: true` and set `concurrentSends: 8`. > Use these defaults only for duplicate-safe workloads without overwrites. > For workloads with overwrites, set `idempotentWrites: false`, `concurrentSends: 1`, and `historicFill: none`. > See [Performance vs. correctness](/influxdb3/enterprise/edr/reference/config-file/#performance-vs-correctness). ## Deploy a sender Create `my-sender.yaml`: ```yaml role: sender image: repository: REGISTRY/edr tag: "1.0.0-pro7dd361197e-amd64" state: location: "s3://edr-state/idb-src" s3: existingSecret: idb-src-influxdb3-enterprise-s3 source: releaseName: idb-src objectStore: s3 s3: bucket: influxdb3-enterprise-src tokenStore: existingSecret: edrs-tokens downstream: address: "http://edrr-edr:9090" authToken: downstream-auth idempotentWrites: false concurrentSends: 1 historicFill: none ``` Create the `edrs-tokens` Kubernetes Secret with a `downstream-auth` key containing the shared link token. The `s3.existingSecret` must contain `access-key-id` and `secret-access-key` keys. Its credentials must access both the source bucket and agent state location. Install from the delivered chart archive: ```bash helm install edrs ./edr-helm-v1.0.0.tgz --values my-sender.yaml ``` The agent’s node name is `edrs-edr`. The receiver must use this name in its upstream list. ## Deploy a receiver Create `my-receiver.yaml`: ```yaml role: receiver image: repository: REGISTRY/edr tag: "1.0.0-pro7dd361197e-amd64" state: location: "s3://edr-state/idb-dst" s3: existingSecret: idb-dst-influxdb3-enterprise-s3 tokenStore: existingSecret: edrr-link-tokens extraSecrets: - idb-dst-admin-token receiver: writeEndpoint: "http://idb-dst-influxdb3-enterprise-ingester:8181" upstreams: - name: edrs-edr authToken: link-auth writeToken: admin-token.json ``` Create `edrr-link-tokens` with a `link-auth` key whose value equals the sender’s `downstream-auth` token. Use the actual Secret configured by the destination Enterprise release’s `security.auth.adminToken.existingSecret` in place of `idb-dst-admin-token`. Its `admin-token.json` key mounts unchanged into the token directory. Alternatively, supply a dedicated write token as its own file in a projected Secret and use that file name as `writeToken`. The receiver needs no source store, but it needs credentials for its own state location. ```bash helm install edrr ./edr-helm-v1.0.0.tgz --values my-receiver.yaml ``` ## Deploy a combined relay Start from the chart’s `examples/values-combined.yaml`. Set `role: combined` and configure both facets in one values file: * Point `source` at the relay’s own Enterprise store. * Point `downstream.address` at the next receiver. * Point `receiver.writeEndpoint` at the relay’s own Enterprise write endpoint. * Add each incoming sender to `receiver.upstreams` with its node name and token references. * Set `state.location` to a separate durable location for the relay agent. * Project all required incoming, outgoing, and write-token files through `tokenStore.existingSecret` and `tokenStore.extraSecrets`. ```bash helm install edrm ./edr-helm-v1.0.0.tgz --values my-combined.yaml ``` ## Object stores and token Secrets S3, Google Cloud Storage, and Azure Blob Storage support both source reads and agent state. Use `s3://`, `gs://`, or `az://` for `state.location`. For sources, the chart uses `source.objectStore: s3`, `gcs`, or `azure`; this differs from the binary’s `--object-store-type google` spelling. | Provider | Source values | Credential values | |--------------------|------------------------|----------------------------------------------------------------------------------| | S3 | `source.s3.bucket` | `s3.existingSecret` with `access-key-id` and `secret-access-key` | |Google Cloud Storage| `source.gcs.bucket` | `gcs.existingSecret` with `service-account.json` | | Azure Blob Storage |`source.azure.container`|`azure.storageAccount` and `azure.existingSecret` with `azure-storage-account-key`| Where configured, platform identity can supply credentials instead of an explicit credential Secret. See the delivered chart’s values reference for provider settings and endpoints. The chart projects token Secrets as files at `/etc/influxdb3-edr/tokens` and sets `INFLUXDB3_EDR_TOKEN_STORE`. File names must match the configuration’s token references. The volume uses no `subPath` mount. Once Kubernetes updates the projected files, the agent reads rotated tokens on the next use without a restart. A Helm upgrade that changes chart-rendered config or token values rolls the pod through checksum annotations. See [Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). ## Operations and limitations * Run exactly one replica per agent. The chart fixes `replicas: 1` and uses `strategy: Recreate` to avoid concurrent agents sharing state. Do not scale the Deployment. * Keep the same `state.location` when replacing or restarting a Deployment. Its cursor and journals survive pod deletion in object storage. * Apply configuration changes with `helm upgrade`; changes such as a downstream address require a restart. * The `-edr` ClusterIP Service exposes replication on port 9090 and observability on port 9091 by default. Scrape the observability port for [metrics](/influxdb3/enterprise/edr/monitor/). * Source and state use one credential set per process. Keep both in the same provider and account. * S3 source buckets are bucket-rooted; source bucket prefixes are not supported. Agent state can use its own prefix. * File-backed source storage with a ReadWriteMany PVC is for development only. Use object storage for production. | role | Operation | Required configuration | | --- | --- | --- | | role | Operation | Required configuration | | sender | Reads a source store and sends data downstream. | source , downstream , and state | | receiver | Accepts replication and writes to a destination server. | receiver and state | | combined | Receives data into a local server and sends that server’s data onward. | source , downstream , receiver , and state | | Provider | Source values | Credential values | | --- | --- | --- | | Provider | Source values | Credential values | | S3 | source.s3.bucket | s3.existingSecret with access-key-id and secret-access-key | | Google Cloud Storage | source.gcs.bucket | gcs.existingSecret with service-account.json | | Azure Blob Storage | source.azure.container | azure.storageAccount and azure.existingSecret with azure-storage-account-key | --- # Manage tokens Source: https://docs.influxdata.com/influxdb3/enterprise/edr/manage-tokens/ * EDR 1.0.0 Tokens are resolved from the token store **on every use—never cached**. To rotate a token, overwrite the file; the next operation uses the new value. No restart required. If both ends rotate, update the destination first. ## Token store Set the directory with `--token-store` or `INFLUXDB3_EDR_TOKEN_STORE`. The file name is the token reference in the configuration. Each file can contain either a whitespace-trimmed raw token or an InfluxDB server token JSON object: ```json {"token": "DESTINATION_TOKEN", "name": "destination-write"} ``` The store reads the `token` field. If `expiry_millis` is present and has passed, the agent warns once; the server still enforces token validity. Other fields, such as `name`, do not affect token resolution. For example, copy a server `admin-token.json` unchanged into the store and reference it with `write_token: admin-token.json`. The store does not select entries from a `permission-tokens.json` collection; provision the required token as its own file. Keep local token files `chmod 600`. The agent warns once per file if group or world permissions are present. Kubernetes Secret volumes can use `0440` with the container’s group. Token names are validated against path traversal. Tokens are never logged. Each node’s config file references two kinds of tokens by name: * **Auth token**—identifies a source to its destination. Each upstream needs a unique auth token so the downstream can identify who connected (`upstreams[].auth_token` on the receiving node; `auth_token` on the sending node’s `downstream`). * **Write token**—the destination’s EDR agent uses it to write the source’s data into local InfluxDB (`upstreams[].write_token`). Typically the destination InfluxDB instance’s admin or operator token. ```bash mkdir -p /etc/edr/secrets # Auth token -- generate a unique token per source node. openssl rand -hex 32 > /etc/edr/secrets/my-auth-token # Write token -- typically the destination's InfluxDB admin/operator token. cp /path/to/destination-write-token /etc/edr/secrets/dest-write-token ``` ## Rotate a token Overwrite the token file at its existing path with the new secret. The agent resolves tokens fresh on every use, so no restart is required. When rotating a token shared between a source and a destination, update the**destination** first, then the source. ## Retire an upstream There’s no dedicated decommission API. Permanently retire an upstream by removing it from the downstream’s config (its `upstreams` list) and revoking its auth token. Previously replicated data at the destination is preserved. ## Vault-backed token stores Sync Vault secrets into the file store, for example, with Vault Agent templates. --- # Monitor EDR Source: https://docs.influxdata.com/influxdb3/enterprise/edr/monitor/ * EDR 1.0.0 1. [Embedded UI](#embedded-ui) 2. [Endpoints](#endpoints) 3. [The halted state](#the-halted-state) 4. [Health model](#health-model) 5. [Triage CLI (edr-inspect)](#triage-cli-edr-inspect) For backfilling pre-existing or gapped data, see[Historic fill and gap fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/). For a Grafana dashboard built on the metrics below, see[Grafana dashboard](/influxdb3/enterprise/edr/monitor/grafana-dashboard/). ## Embedded UI Every EDR agent serves a web UI at `/ui` on its **observability listener**(`--observability-listen`, loopback `127.0.0.1:9091` by default): topology tree with health badges, per-channel metrics (bytes, lag, retry state), historic fill progress, and an activity log. Status is server-computed only—all thresholds come from config. ## Endpoints The observability endpoints below are served on `--observability-listen`(loopback by default—expose with `--observability-listen 0.0.0.0:`).`/health` and the replication protocol are on the network `--listen`. | Endpoint | Purpose | |--------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `GET /metrics` | Prometheus exposition format (for example, `edr_downstream_bytes_received_total{upstream=...}`, `edr_downstream_write_failures_total{...}`). | |`GET /edr/v1/metrics_json`|Structured JSON metrics. Key fields under `self_upstream_metrics`: `pending_bytes`, `in_flight_sends`, `priority_queue_depth`, `batches_sent`, `bytes_sent`, `consecutive_send_failures`, `gaps_pending`, `historic_fill_active`, `gapfill_files_done`/`_total`, `tier_depths`, `cursor_wal_id`.| | `GET /edr/v1/status` | Channel and connection status. | | `GET /edr/v1/node_info` | Node identity and health summary. | | `GET /edr/v1/events` | Server-Sent Events stream of real-time state changes. | | `GET /health` | Liveness. | ## Outstanding work and row counts The Prometheus gauge `edr_owed_work` reports work still owed to each destination. A value of `0` means the agent has no outstanding replication work for that destination at that observation. Use it with channel health when waiting for a backlog to drain before maintenance. New source writes can create more work after the observation. GA fixes a transient false zero while gaps move from pending to in progress. `rows_sent` counts physical rows, not SQL `count(*)` results. Tables wider than 100 fields split a logical row across field families, so the counter can exceed the logical row count. For data validation, compare field-level values as well as counts. See [ordering and wide tables](/influxdb3/enterprise/edr/reference/config-file/#performance-vs-correctness). ## The halted state EDR never voluntarily skips deliverable data. When the downstream persistently **refuses** data that exists and could be delivered, the agent halts replication rather than replicate around the problem—a downstream that is stale-but-consistent as of the halt point beats one that looks live but silently misrepresents some tables. Halting triggers: * **Schema conflict**—immediately. Fix the downstream schema. * **Authentication failure**—immediately. Fix the token or the config. * **Malformed batch** (the receiver cannot parse what this agent sent)— immediately. Usually a version mismatch: upgrade the downstream agent. * **Persistent write errors or unclassified rejections**—after the patience window (`retry.halt_patience_secs`, default 180). Within the window they are treated as transient and retried quietly. What Halted means: * Dispatch stops on **every** tier—no partial replication past the halt point. Upstream data keeps accumulating safely on the object store; the cursor does not move; nothing is lost. * Underneath, the refused batch keeps retrying (every 60s for schema conflicts and malformed batches)—it is the probe. **The moment the downstream accepts it, replication auto-resumes in order.** No restart, no acknowledgement step. * The JSON observability API reports `replication_halted` greater than `0` and sets `halted_since_ns`. The upstream agent includes the flag in its status report. The downstream agent exposes the reported value. See the [halt field values](/influxdb3/enterprise/edr/reference/api/#halt-fields). * The agent writes the following: * An `ERROR` log entry that names the database and the reason. * A `WARN` heartbeat on every probe, with the halt duration. * An audit event. Network outages and downstream restarts do **not** halt—they retry with backoff and show as degraded health; they self-correct. Escape hatches (deliberately the only ones): fix the downstream (auto-resume), or exclude the offending table from the replication scope and restart the agent. There is intentionally no “skip after N hours” option. ## Health model Two layers, combined for display: * **Network health** (silence duration vs the comms config): Healthy, Degraded (missed `degraded_after` intervals), Unreachable (missed`unhealthy_after`), or Unknown (never connected). * **Application health** (last write result): Active, Unauthorised, SchemaConflict, Stalled (network unreachable), or Unknown. Health status is the worst of (retry count vs config thresholds, silence duration vs config thresholds). Heartbeats and reports do not count as successful data sends—`last_successful_send_ts` only advances on real data. Schema and auth issues are local to the two directly connected nodes; they are not propagated up the chain. ## Triage CLI (`edr-inspect`) `edr-inspect` is a **read-only** triage tool shipped alongside the agent. Three independent subcommands, each reading one source: | Command | Reads | Output | |------------------------------------|--------------------------------------------------|---------------------------------------------------------------------------------------------------| |`edr-inspect state `|the state journals only (offline—no running agent)|live replication (WAL cursor), live gap fill (gap ledger), historic replication (manifest progress)| | `edr-inspect metrics [addr]` | the live `/metrics` (observability listener) | a scrolling table watch; `--once` for one detailed block | | `edr-inspect topology [addr]` | `/edr/v1/topology` | a one-shot upstream and downstream ASCII diagram with per-edge health | `state` works even against a **stopped or wedged** agent (it reads the journals directly); it also accepts a cloud `state-location`(`s3://`, `gs://`, or `az://`, credentials from the environment).`metrics` and `topology` default to the loopback observability address (`$INFLUXDB3_EDR_OBSERVABILITY_LISTEN`). In Docker, run these through`docker exec` with no arguments—see[Run EDR in Docker](/influxdb3/enterprise/edr/install/docker/#triage-with-edr-inspect-through-docker-exec). | Endpoint | Purpose | | --- | --- | | Endpoint | Purpose | | GET /metrics | Prometheus exposition format (for example, edr_downstream_bytes_received_total{upstream=...} , edr_downstream_write_failures_total{...} ). | | GET /edr/v1/metrics_json | Structured JSON metrics. Key fields under self_upstream_metrics : pending_bytes , in_flight_sends , priority_queue_depth , batches_sent , bytes_sent , consecutive_send_failures , gaps_pending , historic_fill_active , gapfill_files_done / _total , tier_depths , cursor_wal_id . | | GET /edr/v1/status | Channel and connection status. | | GET /edr/v1/node_info | Node identity and health summary. | | GET /edr/v1/events | Server-Sent Events stream of real-time state changes. | | GET /health | Liveness. | | Command | Reads | Output | | --- | --- | --- | | Command | Reads | Output | | edr-inspect state | the state journals only (offline—no running agent) | live replication (WAL cursor), live gap fill (gap ledger), historic replication (manifest progress) | | edr-inspect metrics [addr] | the live /metrics (observability listener) | a scrolling table watch; --once for one detailed block | | edr-inspect topology [addr] | /edr/v1/topology | a one-shot upstream and downstream ASCII diagram with per-edge health | --- # Grafana dashboard Source: https://docs.influxdata.com/influxdb3/enterprise/edr/monitor/grafana-dashboard/ * EDR 1.0.0 1. [Exclude the metrics database from replication](#1-exclude-the-metrics-database-from-replication) 2. [Create the metrics database](#2-create-the-metrics-database) 3. [Configure Telegraf](#3-configure-telegraf) 4. [Configure Grafana](#4-configure-grafana) 5. [Verify](#5-verify) A ready-made Grafana dashboard turns EDR’s `/metrics` endpoint (see[Endpoints](/influxdb3/enterprise/edr/monitor/#endpoints)) into a fleet-level headline, a per-channel status table, and time-series detail. The pipeline is Telegraf (scrapes `/metrics`) to a local InfluxDB database to Grafana. The config and dashboard files ship in two places—the `observability/`folder at the top of the EDR documentation bundle, and (the same files)`signals-demo/observability/` inside the demo bundle, where the demo’s optional observability overlay mounts them. They’re generic: nothing in them is demo-specific, so use either copy for a production deployment. | File | Purpose | |---------------------------------------------------|----------------------------------------------------------------| | `telegraf/telegraf.conf` |Scrapes this node’s `/metrics` and writes it to a local database| |`grafana/provisioning/datasources/internal_edr.yml`| Points Grafana at that database | | `grafana/provisioning/dashboards/dashboards.yml` | Loads the dashboard below automatically on startup | | `grafana/dashboards/edr-overview.json` | The dashboard itself | **One Grafana per node, not centralized.** Each node’s dashboard reads only that node’s own metrics, from a database dedicated to them (`internal_edr`by default) on that node’s own InfluxDB—it is deliberately not part of the data being replicated between nodes, so there is nothing to aggregate centrally. ## 1. Exclude the metrics database from replication This only applies to a node that has its own `downstream:`—a source or a relay, that is, anything that forwards data onward. A pure destination node has nothing to exclude and can skip straight to step 2. `internal_edr` is local operational data, not something to ship downstream. The default scope (`type: instance`, replicate everything) would otherwise pick it up. Add a scope exclusion in the `scope` block under `downstream:`in this node’s EDR agent config file: ```yaml downstream: # ... name, address, auth_token ... scope: type: instance # everything else still replicates, exclude: # including databases created later databases: [internal_edr] ``` If you’re already scoping to specific databases (`type: databases`), leave `internal_edr` off the list—an allowlist that doesn’t name it already keeps it local. ## 2. Create the metrics database On this node’s InfluxDB, create a database named `internal_edr` (or your own name—keep `telegraf.conf`’s `bucket` and the datasource’s `dbName` in sync if you rename it) to hold Telegraf’s writes. ## 3. Configure Telegraf Point Telegraf at the shipped `observability/telegraf/telegraf.conf`unmodified—don’t write your own—and set the environment variables it expects before starting it: | Variable | Example | Purpose | |---------------------|-------------------------------|-------------------------------------------------| | `EDR_METRICS_URL` |`http://localhost:9091/metrics`| This node’s EDR agent, observability listener | | `INFLUX_WRITE_URL` | `http://localhost:8181` | This node’s InfluxDB | |`INFLUX_ADMIN_TOKEN` | — | This InfluxDB instance’s admin token | |`EDR_SCRAPE_INTERVAL`| `10s` (default) |Optional. Scrape **and** flush cadence—see below.| ```bash EDR_METRICS_URL=http://localhost:9091/metrics \ INFLUX_WRITE_URL=http://localhost:8181 \ INFLUX_ADMIN_TOKEN=ADMIN_TOKEN \ telegraf --config observability/telegraf/telegraf.conf ``` `EDR_SCRAPE_INTERVAL` is the only knob that governs dashboard freshness: the status panels read the latest sample per channel with no averaging window, so the dashboard reflects a channel’s real health within one scrape interval of the agent deciding it. The default `10s` suits a production fleet; drop it to `1s`-`2s` for a near-live view (`/metrics` is cheap to generate). It does **not** change how quickly the agent itself declares a channel degraded or down—that is the `comms` anti-flap window on the sending node (`comms.interval_secs`, `degraded_after`, and`unhealthy_after` in[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/)). > [!Important] > If you adapt the file, keep `name_override = "edr"` in the`[[inputs.prometheus]]` block—every dashboard query is `FROM "edr"`; > without it Telegraf writes to its default measurement name (`prometheus`) > and every panel silently shows no data. ## 4. Configure Grafana Mount `observability/grafana/provisioning/` and`observability/grafana/dashboards/` at the paths Grafana expects (`/etc/grafana/provisioning/datasources/`,`/etc/grafana/provisioning/dashboards/`, and the folder `dashboards.yml`’s`path` points at), and set two environment variables for the Grafana process before starting it. They’re read from `${INFLUX_QUERY_URL}` and`${INFLUX_ADMIN_TOKEN}` placeholders inside`grafana/provisioning/datasources/internal_edr.yml`, which Grafana expands from its own environment at startup—the same admin token as step 3, not a separate one: | Variable | Example | Purpose | |--------------------|-----------------------|------------------------------------| | `INFLUX_QUERY_URL` |`http://localhost:8181`| This node’s InfluxDB | |`INFLUX_ADMIN_TOKEN`| — |This InfluxDB instance’s admin token| ```bash INFLUX_QUERY_URL=http://localhost:8181 \ INFLUX_ADMIN_TOKEN=ADMIN_TOKEN \ grafana-server --homepath /usr/share/grafana ``` The dashboard itself loads automatically on startup—`dashboards.yml`points Grafana at the mounted `grafana/dashboards/edr-overview.json`, no manual import needed. A worked example ships with the[signals demo](/influxdb3/enterprise/edr/demo/)—its optional`compose.observability.yaml` overlay gives every one of the four nodes this same Telegraf and Grafana pipeline (`demo/signals-demo/` in the source repository). ## 5. Verify Open Grafana and confirm the fleet headline row shows a healthy count. If every panel is empty: * Check Telegraf’s logs for scrape or write errors, and confirm`name_override = "edr"` is set. * Confirm the datasource’s token has read access to `internal_edr`. * Confirm the EDR agent’s observability listener is reachable from wherever Telegraf runs (loopback by default—see[Endpoints](/influxdb3/enterprise/edr/monitor/#endpoints)). ### What you’ll see | Tier | Shows | |----------------------------------------------------|--------------------------------------------------------------------------------| | Fleet headline | Healthy and unhealthy counts, upstream and downstream | | Channel status table | Exactly one row per channel—network and application health, status | | Detail — upstream (selectable per upstream) | Bytes received/sec, write failures & columns dropped | |Detail — downstream (totals across all destinations)|Bytes sent/sec, pending backlog, WAL backlog, gaps, halted state per destination| Terminology matches the rest of this guide: **upstream** channels are nodes sending data to this node; **downstream** is where this node sends data to. The headline and status-table panels show **current** state—each reads the latest sample per channel with no averaging window—so they track the agent’s own health within one `EDR_SCRAPE_INTERVAL`, never show a channel in two states at once, and clear the moment a recovery is scraped. The time-series detail panels are historical, as usual. | File | Purpose | | --- | --- | | File | Purpose | | telegraf/telegraf.conf | Scrapes this node’s /metrics and writes it to a local database | | grafana/provisioning/datasources/internal_edr.yml | Points Grafana at that database | | grafana/provisioning/dashboards/dashboards.yml | Loads the dashboard below automatically on startup | | grafana/dashboards/edr-overview.json | The dashboard itself | | Variable | Example | Purpose | | --- | --- | --- | | Variable | Example | Purpose | | EDR_METRICS_URL | http://localhost:9091/metrics | This node’s EDR agent, observability listener | | INFLUX_WRITE_URL | http://localhost:8181 | This node’s InfluxDB | | INFLUX_ADMIN_TOKEN | — | This InfluxDB instance’s admin token | | EDR_SCRAPE_INTERVAL | 10s (default) | Optional. Scrape and flush cadence—see below. | | Variable | Example | Purpose | | --- | --- | --- | | Variable | Example | Purpose | | INFLUX_QUERY_URL | http://localhost:8181 | This node’s InfluxDB | | INFLUX_ADMIN_TOKEN | — | This InfluxDB instance’s admin token | | Tier | Shows | | --- | --- | | Tier | Shows | | Fleet headline | Healthy and unhealthy counts, upstream and downstream | | Channel status table | Exactly one row per channel—network and application health, status | | Detail — upstream (selectable per upstream) | Bytes received/sec, write failures & columns dropped | | Detail — downstream (totals across all destinations) | Bytes sent/sec, pending backlog, WAL backlog, gaps, halted state per destination | --- # Historic fill and gap fill Source: https://docs.influxdata.com/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/ * EDR 1.0.0 1. [Historic fill](#historic-fill) 2. [Gap fill](#gap-fill) ## Historic fill Historic fill replicates data that existed on the source *before* EDR was deployed, or before this destination was added. `historic_fill` is required—there is no default. This is deliberate: you must declare your intent for existing data before EDR attaches to a source. If `historic_fill` is absent, the agent **refuses to start** and prints the choices below. ```yaml downstream: # ... name, address, auth_token ... historic_fill: mode: none # required -- choose: none | full | since # mode: full # mode: since # since: "7d" # only with mode: since -- whole days (Nd) or ISO date (no sub-day units) ``` | Mode | Behavior | |-------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`none` | **Live only.** Start at the most recent [write-ahead log (WAL)](/influxdb3/enterprise/edr/reference/glossary/#wal-write-ahead-log) file (the leading edge) and replicate new data as it lands; no pre-existing data is backfilled. Data that evicts before EDR reaches it is *not* recovered. | |`full` | Backfill **all** pre-existing data (from snapshots and [cv2](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor)), then live. | |`since`|Backfill pre-existing data whose **data timestamp** is after the `since` threshold, then live. Threshold is a whole-day duration (`7d`, `30d`—floored to the start of that UTC day) or an ISO date (`2026-06-01`). Sub-day units (`30m`, `6h`) are **rejected**—cv2 is partitioned by data date, so day is the only honest resolution. An unparseable value fails at startup.| **Start point.** On a fresh start, *every* mode (including `none`) seeds the live cursor to the **snapshot boundary**—the frontier WAL ID near the leading edge—so live replication begins from the most recent WAL, never from WAL 1. The difference between the modes is only what (if anything) gets backfilled *below* that boundary. `none` does no backfill; `full`and `since` hand the below-boundary data to historic fill. The boundary that divides historic fill from live replication is in WAL-ID (ingest) space and is computed once at startup. The `since` threshold filters by *data timestamp* (event time). ### How historic fill works On first startup (no prior cursor), the agent computes a snapshot boundary—the frontier WAL ID. The live WAL replicator handles everything after the frontier; historic fill handles everything before it: 1. **Snapshot manifests (`.ptsnap`)** are read to build a work list of[gen0](/influxdb3/enterprise/reference/storage-engine-config-options/#gen0)files, each carrying its WAL range and time range. 2. **Gen0 files** on the work list are read and replicated at lower priority than live data unless a priority rule claims historic fill explicitly. Historic-fill concurrency follows `concurrent_sends`. With `concurrent_sends: 1`, fill processes one file at a time. Higher values allow parallel backfill and require `idempotent_writes: true`. The pipeline continues across snapshot boundaries. 3. **cv2 files (compaction levels L1 and above)** are the primary medium for genuinely old data—data whose gen0 files and snapshot manifests have already been compacted away. cv2 candidates are enumerated by time window and processed as first-class work list entries. Progress is persisted in a manifest file, so historic fill survives agent restarts and resumes where it left off. When the work list is exhausted the fill is marked complete and never runs again (the manifest records completion). ### What to expect in the logs Historic fill is deliberately loud. These messages are normal: * `historic fill: starting — N snapshots, M files`—work list built. * `historic fill: MANIFEST GAP — snapshot manifests ... deleted by compactor`—**expected** whenever the compactor has cleaned up older snapshot manifests. The agent recovers the gap from surviving WAL files where possible, and from cv2 files for the evicted head. * `historic fill: MANIFEST GAP only partially covered by WAL files ...`—the mixed three-tier situation: a suffix of the gap comes from WAL files, the evicted head from cv2. cv2 is the primary medium for that data. * `historic fill: added N cv2 work list entries ...`—cv2 recovery in progress. * Warnings about over-replication when sending cv2 files—**expected**whenever EDR recovers compacted data. cv2 files are compacted, so they contain data beyond the exact recovery window. Block-level time filters cut this down, but some over-replication is inherent. The destination’s idempotent writes absorb the duplicates, so correctness is unaffected. The warnings report the bandwidth cost; they don’t mean something is wrong. * `historic fill: completed` or `historic fill: already completed — skipping`—done; restart-safe. One historic fill message is **not** normal: `POTENTIAL DATA LOSS`. See[Common log messages](/influxdb3/enterprise/edr/troubleshoot/common-issues/#common-log-messages). During a fill, the metrics API reports `historic_fill_active`,`gapfill_files_done`, and `gapfill_files_total` so you can track progress. ## Gap fill Gap fill recovers WAL ranges missed during live replication, including delivery failures, transient object-store reads, and server WAL eviction. The agent records each range in the gap ledger. Recovery first re-reads the exact raw WAL files if they still exist. A temporarily unreadable file causes recovery to wait briefly and retry the read. Only an evicted range falls back to snapshot manifests, gen0, and compacted cv2 files.`gapfill_files_from_wal_total` counts files recovered from raw WAL. An agent behind the retention floor records the whole evicted span as one gap and resumes live replication above that floor. Gap fill recovers the span in the background. If a channel is halted, `gapfill_blocked_on_halt` and the log identify the blocked recovery. Operationally: * Gaps appear in metrics as `gaps_pending` and in the gap ledger file next to the state file. After recovery they are marked resolved. * **A recovering agent is visibly recovering.** Gaps a round is actively working move from `gaps_pending` to `gaps_in_progress` (pending counts only untouched gaps), and the round’s phases publish progress:`gapfill_discovery_candidates` and `gapfill_discovery_scanned` while it searches the store for covering files, then `gapfill_files_done`and `gapfill_files_total` while it replicates them. If those gauges are moving, recovery is working—even when `gaps_resolved_total` hasn’t ticked yet (gaps settle when their round completes). * `gapfill_files_vanished_total` counts covering files the server’s compactor removed mid-recovery (the agent re-discovers their replacements automatically). A high rate means recovery is racing compaction—expect rounds to take longer. * Recovery from gen0 is precise. Recovery from cv2 over-replicates (see above)—expected, absorbed by idempotent writes. * If you see gaps regularly, increase `--wal-snapshots-to-keep` on the source InfluxDB. See[Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/); recovery from compacted files is a safety net, not the intended steady state. | Mode | Behavior | | --- | --- | | Mode | Behavior | | none | Live only. Start at the most recent write-ahead log (WAL) file (the leading edge) and replicate new data as it lands; no pre-existing data is backfilled. Data that evicts before EDR reaches it is not recovered. | | full | Backfill all pre-existing data (from snapshots and cv2 ), then live. | | since | Backfill pre-existing data whose data timestamp is after the since threshold, then live. Threshold is a whole-day duration ( 7d , 30d —floored to the start of that UTC day) or an ISO date ( 2026-06-01 ). Sub-day units ( 30m , 6h ) are rejected —cv2 is partitioned by data date, so day is the only honest resolution. An unparseable value fails at startup. | --- # EDR reference Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/ * EDR 1.0.0 ### [EDR architecture](/influxdb3/enterprise/edr/reference/architecture/) The EDR sidecar agent model, its components, deployment topology shapes, and the replication pipeline that moves data from source to destination. ### [EDR configuration file reference](/influxdb3/enterprise/edr/reference/config-file/) Complete field reference for the EDR replication config YAML file— top-level fields, downstream and upstream shapes, scope, priorities, and bandwidth scheduling. ### [EDR CLI reference](/influxdb3/enterprise/edr/reference/cli/) Complete flag and environment variable reference for the influxdb3-edr agent and the edr-inspect triage CLI. ### [EDR replication protocol reference](/influxdb3/enterprise/edr/reference/api/) The EDR agent-to-agent replication protocol—endpoints, protocol version negotiation, capabilities, and the observability API. ### [EDR state and recovery](/influxdb3/enterprise/edr/reference/state-and-recovery/) What EDR remembers between restarts, where it keeps it, what each piece means, and what happens if you delete it. ### [EDR compatibility](/influxdb3/enterprise/edr/reference/compatibility/) EDR’s agent-to-agent protocol compatibility and agent-to-InfluxDB storage format compatibility matrices, and supported destinations. ### [EDR limitations](/influxdb3/enterprise/edr/reference/limitations/) What EDR is not, and known implementation limitations tracked for future releases. ### [Glossary](/influxdb3/enterprise/edr/reference/glossary/) Terms related to InfluxData products and platforms. --- # EDR replication protocol reference Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/api/ * EDR 1.0.0 > [!Note] > `v1` in `/edr/v1/*` is the EDR replication protocol’s own version. It has > no relationship to InfluxDB v1. EDR sources from InfluxDB 3 Enterprise and > replicates to InfluxDB 3 Enterprise, InfluxDB 3 Cloud, or AWS Timestream > for InfluxDB 3. Agent-to-agent communication is HTTP-based. The downstream facet serves the endpoints below. The upstream agent connects (`/connect`), then sends data batches (`/data`) and periodic reports (`/report`, default every 10 seconds). The report carries the upstream’s health and, when`share_topology` is enabled, the tree of upstreams behind it—this is how a central node assembles a full network picture without direct connectivity to every edge. The HTTP response acknowledges each data batch synchronously. The upstream’s replication cursor advances only on confirmed delivery. ## Replication protocol endpoints These endpoints carry the EDR agent-to-agent replication protocol (`/edr/v1/*`), not an InfluxDB 3 API. | Endpoint |Method| Purpose | |-----------------|------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| |`/edr/v1/connect`| POST |Handshake: upstream declares identity, scope, comms expectations, topology. Sent only between agents negotiating protocol v2 or below; not sent between v3 peers.| | `/edr/v1/data` | POST | Data batches in Line Protocol (LP) or zstd-compressed [PT](/influxdb3/enterprise/reference/internals/storage-engine/#new-file-format) format. | |`/edr/v1/report` | POST | Periodic status report: health, progress, upstream topology. Default interval 10 seconds. | | `/health` | GET | Liveness. | ## Observability API endpoints These endpoints expose an EDR agent’s own replication metrics and status, not InfluxDB 3 data. | Endpoint |Method| Purpose | |----------------------|------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `/metrics` | GET |Prometheus-format metrics for both facets, including send/receive byte and row counters, per-tier queue depths (`edr_upstream_tier_queue_depth{tier=...}`), cursor positions, retry counts, gap and historic fill progress, and over-replication counters.| |`/edr/v1/metrics_json`| GET | The same data as structured JSON, including the `tier_depths` array, for programmatic consumption and the UI. | | `/edr/v1/status` | GET | This node’s replication state: facets, channels, health, cursors, backlog. | | `/edr/v1/topology` | GET | The replication network as known to this node (its upstream tree, assembled from reports, plus its downstream link). | | `/edr/v1/logs` | GET | Recent structured log entries. | | `/edr/v1/node_info` | GET | Node identity, role, location, outbound health, and outbound metric snapshots. | | `/edr/v1/events` | GET | Server-Sent Events (SSE) stream that fires when state changes, used by the UI for live updates without polling. | | `/ui` | GET | Embedded web UI. | ### Halt fields `GET /edr/v1/node_info` returns the outbound metric snapshot under`upstream_facet` when the agent has one destination. For multiple destinations, read each snapshot under`upstream_facet_by_destination[]`. Use these values to interpret the halt fields in either snapshot: | Field | Value | Meaning | |--------------------|----------------|------------------------------------| |`replication_halted`| `0` | Replication isn’t halted. | |`replication_halted`|Positive integer| The value counts halted batches. | |`last_write_result` | `0` |The agent hasn’t attempted a write. | |`last_write_result` | `1` | The last write succeeded. | |`last_write_result` | `2` | Authentication failed. | |`last_write_result` | `3` |A schema conflict blocked the write.| `edr-inspect metrics` translates `last_write_result` into`last-write=unknown`, `last-write=success`, `last-write=auth_failure`, or`last-write=schema_conflict`. When `replication_halted` is greater than `0`, the command also prints the `REPLICATION HALTED: N batch(es)` message. The replication protocol endpoints and `/health` are served on the network listener (`--listen`, default `0.0.0.0:9090`). The observability endpoints are served on the observability listener (`--observability-listen`, loopback `127.0.0.1:9091` by default). See[CLI reference](/influxdb3/enterprise/edr/reference/cli/). Logging is structured JSON (tracing) covering replication progress, connection state changes, errors and retries, configuration changes, gap advisories, and historic fill recovery decisions (including every loud over-replication warning). Significant state changes and exceptions are additionally emitted through the audit service provider interface (SPI) (stdout implementation initially). ## Protocol version negotiation Directly connected agents negotiate a **protocol version** and a**capability set**, per hop—never propagated along a relay chain (a bad hop reports at that hop; the rest of the chain is unaffected). * **Protocol v1** is the name for the wire behavior of agents released before versioning existed (0.1.x and 0.2.x). Those agents send no version fields. EDR interprets missing version fields as protocol v1. The protocol v1 behavior hasn’t changed. * **Protocol v2** (interim development builds; never in a released version) added explicit protocol negotiation. Protocol v2 agents negotiate at `/connect`. The `/connect` request gates `/data`. The `/data` endpoint returns HTTP 428 until the agents connect. * **Protocol v3** (current) uses the **stateless data path**. Every `/report` includes the upstream agent’s full declaration: identity, location, communication settings, agent version, and protocol declaration. The downstream agent returns its declaration in the `ReportAccept` response. Both agents negotiate the protocol on every report. A response without a downstream declaration identifies a protocol v2 or earlier agent. Every `/data` request includes a compact declaration in the `X-EDR-Name`,`X-EDR-Protocol: ,`, and `X-EDR-Caps` headers. Protocol v3 peers never send `/connect`. The `/data` endpoint requires no prior contact. Any receiver instance can serve any request. This stateless behavior supports receiver fault tolerance and load balancing. * Both agents calculate the same agreement. The effective version is the lower of the two declared versions. An agent refuses the connection if the effective version is lower than either agent’s minimum. An effective version below protocol v3 selects the v2 connect-gated flow. Mixed-version fleets require no additional configuration. |Upstream \\ Downstream|0.1.x / 0.2.x (v1)|interim dev builds (v2)|1.0.0-0.rc.1+ (v3)| |----------------------|------------------|-----------------------|------------------| | **0.1.x / 0.2.x** | v1 (implicit) | v1—accepted | v1—accepted | |**interim dev builds**| v1—accepted | v2, connect-gated |v2, connect-gated | | **1.0.0-0.rc.1+** | v1—accepted | v2, connect-gated |**v3, stateless** | With the current minimums, **no released pairing can be refused**. The first possible refusal requires a future release that deliberately raises its minimum—a compatibility break that will be called out in[release notes](/influxdb3/enterprise/edr/release-notes/). ### Refusal Refusal is explicit and terminal. The refusing side answers **HTTP 426**(on `/connect`, `/report`, or `/data`) with a diagnosis naming both versions and both minimums; the upstream latches a **protocol halt**: dispatch stops (nothing pops from any tier), health reads `Halted`, the reason appears in the status API, the UI, and the logs, and a`PROTOCOL-HALTED` WARN repeats once a minute while halted. Negotiation re-runs on **every report** (default 10s), so a peer upgrade clears the halt within one report interval—there’s no separate probe task. See[Troubleshoot EDR](/influxdb3/enterprise/edr/troubleshoot/common-issues/#protocol-incompatible-agents-of-different-versions). ### Capabilities Capabilities gate optional features (the version gates wire-breaking changes). Initial registry: `pt-wire-1`—the PT wire format (`encoding: pt`). A downstream that negotiates successfully but lacks a capability the local config requires produces the same protocol halt, with a message naming the config key. ## Security and tokens * **TLS** is required for all upstream-downstream communication in production deployments. * **Token-by-name model**: the YAML config references tokens by name; values are resolved at runtime from the token store (file-based directory, one token per named entry, validated at startup). Config files are therefore safe to version and share. * **Authentication**: each upstream presents its auth token when connecting; the downstream maps the token to the expected upstream identity (`upstreams[].auth_token`). * **Authorization by the destination**: the downstream writes each upstream’s data using that upstream’s `write_token` against the local InfluxDB 3 instance. The token’s permissions on the destination determine what the upstream can effectively write—EDR implements no authorization layer of its own. Restricting or revoking an upstream is done by scoping or revoking its tokens. See [Manage tokens](/influxdb3/enterprise/edr/manage-tokens/) for token store operations. | Endpoint | Method | Purpose | | --- | --- | --- | | Endpoint | Method | Purpose | | /edr/v1/connect | POST | Handshake: upstream declares identity, scope, comms expectations, topology. Sent only between agents negotiating protocol v2 or below; not sent between v3 peers. | | /edr/v1/data | POST | Data batches in Line Protocol (LP) or zstd-compressed PT format. | | /edr/v1/report | POST | Periodic status report: health, progress, upstream topology. Default interval 10 seconds. | | /health | GET | Liveness. | | Endpoint | Method | Purpose | | --- | --- | --- | | Endpoint | Method | Purpose | | /metrics | GET | Prometheus-format metrics for both facets, including send/receive byte and row counters, per-tier queue depths ( edr_upstream_tier_queue_depth{tier=...} ), cursor positions, retry counts, gap and historic fill progress, and over-replication counters. | | /edr/v1/metrics_json | GET | The same data as structured JSON, including the tier_depths array, for programmatic consumption and the UI. | | /edr/v1/status | GET | This node’s replication state: facets, channels, health, cursors, backlog. | | /edr/v1/topology | GET | The replication network as known to this node (its upstream tree, assembled from reports, plus its downstream link). | | /edr/v1/logs | GET | Recent structured log entries. | | /edr/v1/node_info | GET | Node identity, role, location, outbound health, and outbound metric snapshots. | | /edr/v1/events | GET | Server-Sent Events (SSE) stream that fires when state changes, used by the UI for live updates without polling. | | /ui | GET | Embedded web UI. | | Field | Value | Meaning | | --- | --- | --- | | Field | Value | Meaning | | replication_halted | 0 | Replication isn’t halted. | | replication_halted | Positive integer | The value counts halted batches. | | last_write_result | 0 | The agent hasn’t attempted a write. | | last_write_result | 1 | The last write succeeded. | | last_write_result | 2 | Authentication failed. | | last_write_result | 3 | A schema conflict blocked the write. | | Upstream \ Downstream | 0.1.x / 0.2.x (v1) | interim dev builds (v2) | 1.0.0-0.rc.1+ (v3) | | --- | --- | --- | --- | | Upstream \ Downstream | 0.1.x / 0.2.x (v1) | interim dev builds (v2) | 1.0.0-0.rc.1+ (v3) | | 0.1.x / 0.2.x | v1 (implicit) | v1—accepted | v1—accepted | | interim dev builds | v1—accepted | v2, connect-gated | v2, connect-gated | | 1.0.0-0.rc.1+ | v1—accepted | v2, connect-gated | v3, stateless | --- # EDR architecture Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/architecture/ * EDR 1.0.0 1. [External sidecar agent](#external-sidecar-agent) 2. [Storage engine terms](#storage-engine-terms) 3. [Two timestamps](#two-timestamps) 4. [The replication pipeline](#the-replication-pipeline) ## External sidecar agent EDR runs as a separate process (`influxdb3-edr`) alongside InfluxDB 3 Enterprise, reading the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)’s object store directly. The InfluxDB binary itself is unmodified. flowchart TB subgraph SRC["Source Node"] direction LR S1["InfluxDB 3 Enterprise"] -- "shared object store" --\> S2["EDR Agent"] end subgraph DST["Destination Node"] direction LR D1["EDR Agent"] --\> D2["InfluxDB 3"] end SRC -- "HTTP POST (EDRP)" --\> DST ### Components * **EDR Agent** (`influxdb3-edr`)—the replication process. One per node. Depending on configuration, one agent process acts as: * **Upstream facet** (source role)—reads the local InfluxDB instance’s upgraded-storage-engine object store (WAL files, snapshots, compacted files), maintains replication state, and pushes data to a downstream destination. * **Downstream facet** (sink role)—receives data from one or more configured upstream agents and writes it to the local InfluxDB instance using a per-upstream write token. * **Combined**—both facets in one process, for hierarchical topologies where a regional node receives from edges and forwards to a central node. * **Token Store**—a directory of secret files for authentication. See[Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). * **Config File**—YAML defining topology, scope, and behavior. See[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/). * **Object Store**—the InfluxDB data directory (local filesystem, or S3, GCS, or Azure through `--object-store-type`). The agent reads it through the standard `ObjectStore` abstraction, so local filesystem, S3, GCS, and Azure backends are all supported. Multi-node InfluxDB sources are supported—each ingest node’s WAL stream is tracked independently; multi-node sources require `idempotent_writes: true`. The memory store is not supported as a source—EDR requires durable storage to guarantee replication state persistence and at-least-once delivery. ### Deployment shapes | Deployment | Data flow | |-----------------------|--------------------------------------------------------------------------------------| |Hierarchical, any depth| Edge nodes send to regional nodes, which forward to a central node. | | Direct | Edge nodes send directly to a central node. | | Peer-to-peer | Two nodes send data to each other. | | Direct mode |An edge agent writes to InfluxDB 3 through the v3 write API without a receiving agent.| * **Hierarchical**—a combined agent at the regional node receives from upstream edges and forwards onward. Each hop has its own independent configuration (scope, priorities, bandwidth schedule, credentials). There’s no architectural depth limit; practical deployments are 2-3 levels. * **Peer-to-peer**—two nodes can each configure the other as their downstream, giving bidirectional replication. Scope filtering on each sender prevents circular amplification—each side replicates only its locally-owned databases and tables, so data received from the peer is not echoed back. * **Direct mode**—the upstream agent can write straight to a plain InfluxDB 3 instance using the InfluxDB v3 write API, with no downstream agent. This sacrifices the agent-to-agent protocol’s PT wire format, topology, and health reporting, but requires nothing at the destination beyond a write token. See[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/#direct-mode). ## Storage engine terms EDR reads the files that InfluxDB 3 Enterprise’s[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)writes. These terms appear throughout the EDR documentation: * **WAL (write-ahead log)**—the durable record of recent writes, flushed to the object store roughly every second. EDR’s live path replicates WAL files. See[WAL](/influxdb3/enterprise/edr/reference/glossary/#wal-write-ahead-log). * **PT**—the upgraded storage engine’s columnar file format (`.pt` files). EDR can send PT blocks on the wire without decoding rows. See[New file format](/influxdb3/enterprise/reference/internals/storage-engine/#new-file-format). * **Gen0**—the first compaction generation, produced from WAL data. See[Gen0](/influxdb3/enterprise/reference/storage-engine-config-options/#gen0). * **cv2**—compacted files at levels L1 and above. The compactor organizes them into fixed 24-hour UTC windows, so they’re partitioned by data date. They’re the primary medium for data whose gen0 files and snapshot manifests have already been compacted away. See[Compactor](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor). * **`.ptsnap`**—snapshot manifests. Each records the `last_wal` ID its snapshot covers, which is how EDR computes the snapshot boundary. ## Two timestamps EDR distinguishes two timestamps throughout, and they must never be confused: | Term | Meaning | Monotonic? | |------------------|--------------------------------------------|---------------------------------------------------| |**Data timestamp**|When the event occurred (the `time` column).|No—writers can write any timestamp, past or future.| | **Ingest time** | When InfluxDB received the write. | Yes, within a node. | The boundary that divides historic fill from live replication is defined in WAL-ID space, which is ingest time. The `historic_fill.since` threshold is a data-timestamp filter. See[Historic fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill). ## The replication pipeline The upstream facet runs a pipeline of cooperating tasks. Three producers—the WAL Replicator, Historic Fill, and Gap Fill—feed a Tiered File Queue: one first-in, first-out (FIFO) queue per tier, where a tier is one distinct priority value (see [Priorities](#priorities)). The queue applies backpressure, so a producer slows when its tier fills. A Tiered Dispatcher drains that queue with deficit round-robin (DRR) scheduling and file-atomic dispatch, then encodes and sends each bundle downstream. ### WAL replicator (live path) #### Starting point: the snapshot boundary * At startup, the agent computes the snapshot boundary once: for each ingest node, the `last_wal` ID recorded in the most recent `.ptsnap`manifest. * The WAL replicator reads only from this boundary upward, not from the leading edge and not from independent discovery. Everything below the boundary belongs to[historic fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill). Whether historic fill actually backfills that range depends on the configured `historic_fill` mode. * Historic fill and the WAL replicator run concurrently, each on its own side of the boundary. Because the agent computes the boundary once in its main startup path and shares it with both producers, the division of work is deterministic—no race between them introduces a gap or an overlap. * On subsequent startups, the replicator resumes from its persisted cursor. #### Polling and per-file processing The WAL replicator polls the object store (default every 1000 ms, configurable through `poll_interval_ms`) for new WAL files per ingest node. For each file it: 1. Extracts block references (database, table, data-time range, byte range) from the WAL file’s block headers—no row decoding at this stage. 2. Evaluates each block against the replication scope, dropping out-of-scope blocks, and against the priority rules, assigning each in-scope block a priority and so a tier. 3. Groups the file’s in-scope blocks by tier and emits one FileBundle per tier. ### File-atomic dispatch A **FileBundle** is the unit of dispatch: all in-scope blocks of one source file destined for one tier. The dispatcher never interleaves blocks from different files within a send. A file’s replication cursor entry completes only when all of its bundles, across all tiers, have completed. A WAL file whose blocks match different priority rules splits into one bundle per tier. High-priority blocks do not promote the whole file, and a full low-priority tier does not block the high-priority bundle: bundles enter the queue highest-tier first, and each tier has independent capacity and backpressure. The **TieredFileQueue** holds one FIFO queue per priority tier. The **TieredDispatcher** takes bundles from those queues using deficit round-robin (DRR) across tiers. Each tier has a configurable share weight. An empty tier’s deficit resets, and the dispatcher redistributes its capacity to non-empty tiers (work-stealing). With no priority configuration, the queue degenerates to a single plain FIFO with no quality-of-service (QoS) overhead. For each bundle it takes, the dispatcher acquires a concurrency permit, encodes the bundle’s blocks, and sends one HTTP POST. The default permit count is 1, which gives strict-order delivery.`concurrent_sends > 1` requires `idempotent_writes: true`, because out-of-order arrival silently reverts overwrites under last-write-wins. Completion accounting is per file. A counter tracks completed bundles against the file’s expected bundle count. When all bundles complete, the replication cursor advances contiguously, high-water-mark style, even when concurrent files complete out of order. Failures are recorded: a file with a failed bundle still completes its accounting, with the failure flagged. Sends are retried with exponential backoff, per the `retry` configuration. Delivery confirmation is optional: bundles can carry a confirmation channel. The channel signals historic fill and gap fill that the downstream accepted a specific file’s data. They mark the corresponding work-list entry done only after that signal. For files that split into multiple bundles, the confirmation fires when the last bundle completes. ### Priorities Priority rules form an ordered, first-match-wins list. Each rule matches a scope (all, a database, or specific tables) and, optionally, a data-time recency window (for example, “within the last 1h”). Each rule assigns a numeric priority (1-255, lower is more urgent). Unique priority values become dispatch tiers. An optional per-rule `share` weight sets the tier’s DRR share. One rule can claim historic fill traffic for its tier with`historic_fill: true`; otherwise historic fill uses the lowest-priority tier. The dispatcher schedules across tiers by weighted sharing with DRR, not by strict ordering that can starve lower tiers. Lower tiers make progress in proportion to their share, even while high-priority traffic flows. Reordering across files means a restart can re-send data that the file-granular cursor can’t account for. That’s why priority routing requires `idempotent_writes: true`. See[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/#priorities)for the field schema. ### Wire format Set the `encoding` config field to one of two wire encodings: * **`lp`**—Line Protocol text, gzip-compressed in transport. Universal; required for direct mode and any non-agent destination. * **`pt`**—PT blocks with schema front matter, zstd-compressed. Agent-to-agent only. Approximately 2.5x bandwidth reduction over LP. Block data flows from the source object store to the wire without decoding and re-encoding rows. With either encoding, the destination applies writes idempotently: re-delivering the same rows converges to the same state (last-writer-wins on an identical series key and timestamp). | Deployment | Data flow | | --- | --- | | Deployment | Data flow | | Hierarchical, any depth | Edge nodes send to regional nodes, which forward to a central node. | | Direct | Edge nodes send directly to a central node. | | Peer-to-peer | Two nodes send data to each other. | | Direct mode | An edge agent writes to InfluxDB 3 through the v3 write API without a receiving agent. | | Term | Meaning | Monotonic? | | --- | --- | --- | | Term | Meaning | Monotonic? | | Data timestamp | When the event occurred (the time column). | No—writers can write any timestamp, past or future. | | Ingest time | When InfluxDB received the write. | Yes, within a node. | --- # EDR CLI reference Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/cli/ * EDR 1.0.0 ## `influxdb3-edr` ``` influxdb3-edr [OPTIONS] ``` | Flag | Required | Default | |----------------------------------------|---------------------------------------|----------------| | `--config ` | Yes | — | | `--data-dir ` | Yes | — | | `--token-store ` |Yes, or set `INFLUXDB3_EDR_TOKEN_STORE`| — | | `--listen ` | No | `0.0.0.0:9090` | | `--observability-listen ` | No |`127.0.0.1:9091`| | `--listener-file-path ` | No | — | | `--state-location ` | No | `edr-state` | | `--poll-interval-ms ` | No | `1000` | | `--object-store-type ` | No | — | | `--wal-cleanup-enabled` | No | `false` | | `--wal-cleanup-interval-secs ` | No | `300` | | `--wal-cleanup-snapshot-margin ` | No | `100` | |`--wal-cleanup-max-destination-hold `| No | unbounded | | `--benchmark ` | No | — | * **`--config `**—path to the replication config YAML file. * **`--data-dir `**—InfluxDB data directory (shared object store). * **`--token-store `**—path to the token store directory. Accepts raw token files or server token JSON files containing a `token` field. Env: `INFLUXDB3_EDR_TOKEN_STORE`. See [Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). * **`--listen `**—network listener for the replication protocol (`/edr/v1/connect|data|report`) and `/health`. Must be reachable by upstreams. `0.0.0.0:0` for an OS-assigned port. Env:`INFLUXDB3_EDR_LISTEN`. * **`--observability-listen `**—**loopback by default**: the UI, `/metrics`, and the JSON observability API. Set to`0.0.0.0:` (or a specific interface) to expose for off-host Prometheus or a remote UI. Env: `INFLUXDB3_EDR_OBSERVABILITY_LISTEN`. * **`--listener-file-path `**—file to write the actual network listener address to (for testing with port 0). * **`--state-location `**—state persistence location,**independent of `--object-store-type`**. A path or `file://` URL is local; `s3://bucket/prefix`, `gs://...`, `az://...` are object storage (credentials and region from the environment). Holds`wal_cursor.json`, `gap_ledger.json`, `historic_manifest.json`. Env:`INFLUXDB3_EDR_STATE_LOCATION`. * **`--poll-interval-ms `**—object store polling interval in milliseconds. * **`--object-store-type `**—override the object store type (`s3`, `google`, `azure`). Uses environment variables for credentials. * **`--wal-cleanup-enabled`**—**opt-in.** With high server WAL retention, this is the supported production pattern. Enables agent-side deletion of already-replicated WAL files from the source object store, so they don’t build up past the replicated journal. Off by default (deleting source WAL is destructive). See[Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/). Env:`INFLUXDB3_EDR_WAL_CLEANUP_ENABLED`. * **`--wal-cleanup-interval-secs `**—how often the cleanup sweep runs (only when enabled). Env:`INFLUXDB3_EDR_WAL_CLEANUP_INTERVAL_SECS`. * **`--wal-cleanup-snapshot-margin `**—WAL files to keep below the last snapshotted WAL id, the safety margin behind the snapshot boundary (only when enabled). Env:`INFLUXDB3_EDR_WAL_CLEANUP_SNAPSHOT_MARGIN`. * **`--wal-cleanup-max-destination-hold `**—fan-out only: a destination whose cursor hasn’t advanced for this long (`"7d"`,`"12h"`, `"30m"`) stops holding the cleanup floor; it recovers the evicted range through gap fill when it returns (over-replication, never loss). Unset means a down destination pins WAL indefinitely. Env:`INFLUXDB3_EDR_WAL_CLEANUP_MAX_DESTINATION_HOLD`. * **`--benchmark `**—run a bandwidth benchmark against up to N WAL files and exit. State and index files live wherever you point them. Use durable local storage or object storage for agent state. The state file holds the WAL cursor (what has been replicated); losing it forces re-replication from the snapshot boundary, which is safe but wasteful. The index file is an append-only change-tracking log. ### Environment variables | Variable | Purpose | |----------------------------------------------------------------------|-------------------------------------------------------------------------------------------| | `INFLUXDB3_EDR_TOKEN_STORE` | Token store directory, as an alternative to `--token-store`. | | `EDR_WRITE_ENDPOINT` |Local InfluxDB write endpoint for the destination facet (default: `http://localhost:8181`).| | `RUST_LOG` | Log level filter (default: `info`). Example: `info,influxdb3_catalog=warn` | |`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_DEFAULT_REGION`| S3 credentials (with `--object-store-type s3`). | | `INFLUXDB3_BUCKET` | S3, GCS, or Azure bucket name. | | `INFLUXDB3_EDR_WAL_CLEANUP_ENABLED` | Enable agent-side WAL cleanup (`--wal-cleanup-enabled`). | | `INFLUXDB3_EDR_WAL_CLEANUP_INTERVAL_SECS` | WAL cleanup sweep interval (`--wal-cleanup-interval-secs`). | | `INFLUXDB3_EDR_WAL_CLEANUP_SNAPSHOT_MARGIN` | WAL files kept below the snapshot boundary (`--wal-cleanup-snapshot-margin`). | | `INFLUXDB3_EDR_WAL_CLEANUP_MAX_DESTINATION_HOLD` | Fan-out cleanup-floor hold bound (`--wal-cleanup-max-destination-hold`). | ## `edr-inspect` `edr-inspect` is a **read-only** triage CLI shipped alongside the agent. | Command | Reads | Output | |------------------------------------|--------------------------------------------------|---------------------------------------------------------------------------------------------------| |`edr-inspect state `|the state journals only (offline—no running agent)|live replication (WAL cursor), live gap fill (gap ledger), historic replication (manifest progress)| | `edr-inspect metrics [addr]` | the live `/metrics` (observability listener) | a scrolling table watch; `--once` for one detailed block | | `edr-inspect topology [addr]` | `/edr/v1/topology` | a one-shot upstream and downstream ASCII diagram with per-edge health | `state` works even against a stopped or wedged agent (it reads the journals directly); it also accepts a cloud `state-location` (`s3://`, `gs://`, or`az://`, credentials from the environment). `metrics` and `topology`default to the loopback observability address (`$INFLUXDB3_EDR_OBSERVABILITY_LISTEN`). `edr-inspect` needs no`--data-dir`—it reads only the state journals and the agent’s `/metrics`, never the object store. For usage in Docker through `docker exec`, see[Run EDR in Docker](/influxdb3/enterprise/edr/install/docker/#triage-with-edr-inspect-through-docker-exec). | Flag | Required | Default | | --- | --- | --- | | Flag | Required | Default | | --config | Yes | — | | --data-dir | Yes | — | | --token-store | Yes, or set INFLUXDB3_EDR_TOKEN_STORE | — | | --listen | No | 0.0.0.0:9090 | | --observability-listen | No | 127.0.0.1:9091 | | --listener-file-path | No | — | | --state-location | No | edr-state | | --poll-interval-ms | No | 1000 | | --object-store-type | No | — | | --wal-cleanup-enabled | No | false | | --wal-cleanup-interval-secs | No | 300 | | --wal-cleanup-snapshot-margin | No | 100 | | --wal-cleanup-max-destination-hold | No | unbounded | | --benchmark | No | — | | Variable | Purpose | | --- | --- | | Variable | Purpose | | INFLUXDB3_EDR_TOKEN_STORE | Token store directory, as an alternative to --token-store . | | EDR_WRITE_ENDPOINT | Local InfluxDB write endpoint for the destination facet (default: http://localhost:8181 ). | | RUST_LOG | Log level filter (default: info ). Example: info,influxdb3_catalog=warn | | AWS_ACCESS_KEY_ID , AWS_SECRET_ACCESS_KEY , and AWS_DEFAULT_REGION | S3 credentials (with --object-store-type s3 ). | | INFLUXDB3_BUCKET | S3, GCS, or Azure bucket name. | | INFLUXDB3_EDR_WAL_CLEANUP_ENABLED | Enable agent-side WAL cleanup ( --wal-cleanup-enabled ). | | INFLUXDB3_EDR_WAL_CLEANUP_INTERVAL_SECS | WAL cleanup sweep interval ( --wal-cleanup-interval-secs ). | | INFLUXDB3_EDR_WAL_CLEANUP_SNAPSHOT_MARGIN | WAL files kept below the snapshot boundary ( --wal-cleanup-snapshot-margin ). | | INFLUXDB3_EDR_WAL_CLEANUP_MAX_DESTINATION_HOLD | Fan-out cleanup-floor hold bound ( --wal-cleanup-max-destination-hold ). | | Command | Reads | Output | | --- | --- | --- | | Command | Reads | Output | | edr-inspect state | the state journals only (offline—no running agent) | live replication (WAL cursor), live gap fill (gap ledger), historic replication (manifest progress) | | edr-inspect metrics [addr] | the live /metrics (observability listener) | a scrolling table watch; --once for one detailed block | | edr-inspect topology [addr] | /edr/v1/topology | a one-shot upstream and downstream ASCII diagram with per-edge health | --- # EDR compatibility Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/compatibility/ * EDR 1.0.0 EDR has two compatibility dimensions. > [!Important] > #### Compatibility is binary > > Neither dimension has a partial mode. In a compatible pairing, live > replication, historic fill, and gap recovery all work. An incompatible > pairing refuses to start or halts loudly, and names the side that must > change. |Compatibility dimension| Compared components | Compatibility check | Incompatible behavior | |-----------------------|--------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------| | Agent protocol | Two EDR agents on one replication hop. |Protocol v3 agents negotiate on every `/report`. Protocol v1 and v2 agents negotiate at `/connect`.| The upstream agent halts the replication hop. Both agents report their protocol versions. | | Storage format |An EDR agent and the source InfluxDB 3 Enterprise store.| The agent checks the storage format at startup and while it runs. |The agent refuses to start with an incompatible store. If a running store becomes incompatible, the agent halts replication.| **Where to read compatibility information**: Run `influxdb3-edr --version`to find the agent version and build SHA. Check the startup log for the storage compatibility verdict and build matrix. Check connection logs or`/edr/v1/status` for the protocol negotiated with each upstream agent. Check `/edr/v1/node_info` for the supported InfluxDB server lines and active compatibility halt reasons. Use `edr-inspect topology` and`edr-inspect metrics` to assess the resulting channel health and replication state. ## Supported destinations EDR always replicates **from** InfluxDB 3 Enterprise. It replicates **to**: * InfluxDB 3 Enterprise (agent-to-agent, full protocol and storage compatibility as described below) * [InfluxDB 3 Cloud](/influxdb3/cloud/) (see[Replicate to InfluxDB 3 Cloud](/influxdb3/enterprise/edr/replicate/to-cloud/)) * AWS Timestream for InfluxDB 3 > [!Note] > #### AWS Timestream for InfluxDB 3 > > AWS Timestream for InfluxDB 3 is a supported EDR destination. AWS > maintains this destination’s own documentation; see AWS’s documentation > for AWS Timestream for InfluxDB 3 for setup and compatibility details > specific to that destination. InfluxDB 3 Core is **not** supported as an EDR source or destination. InfluxDB Cloud Serverless, InfluxDB Cloud Dedicated, and InfluxDB Clustered are also not supported as an EDR source or destination. ## Compatibility between EDR agents (wire protocol) Directly connected agents negotiate a **protocol version** and a**capability set** on each hop. Negotiation never propagates along a relay chain, so a bad hop reports at that hop and the rest of the chain is unaffected. With the current minimums, **no released pairing can be refused**. The first possible refusal requires a future release that deliberately raises its minimum, and [release notes](/influxdb3/enterprise/edr/release-notes/) will call out that break. For the protocol version matrix, what each version changed, the capability registry, and what a refusal looks like on the wire, see[Protocol version negotiation](/influxdb3/enterprise/edr/reference/api/#protocol-version-negotiation). ## Compatibility between EDR and InfluxDB 3 Enterprise (storage formats) The sender agent reads the source server’s object store directly—WAL files, snapshot manifests, compaction checkpoints, catalog. Compatibility is therefore about **on-disk formats**, and the store itself is the evidence: at startup the agent sniffs the newest artifacts and either starts, logging `storage compatibility: COMPATIBLE`, or refuses with the exact format mismatch and logs `storage compatibility: INCOMPATIBLE`. ### Supported source server versions The compatibility table lists the InfluxDB 3 Enterprise source server versions supported by each EDR release. | EDR release | InfluxDB 3 Enterprise 3.10.x | 3.11.x | 3.12.x | Newer | |-------------|--------------------------------|--------------------------------|--------------------------------|-----------| |0.1.0 / 0.2.0|Supported (built against 3.10.0)| Unsupported | Unsupported |Unsupported| |1.0.0-0.rc.1 | Supported |Supported (built against 3.11.0)| Unsupported |Unsupported| |1.0.0-0.rc.2 | Supported |Supported (built against 3.11.0)| Unsupported |Unsupported| |1.0.0-0.rc.3 | Supported |Supported (built against 3.11.0)| Unsupported |Unsupported| | 1.0.0 GA | Unvalidated and unsupported | Supported |Supported (built against 3.12.0)|Unsupported| > [!Warning] > The GA binary’s displayed server-line list still includes 3.10.x. > That display does not establish support: EDR 1.0.0 is validated and supported only with 3.11.x and 3.12.x sources. > EDR rc.3 is the last release validated with 3.10.x sources. The release validation covers 3.10.5 and 3.11.0 stores for rc.1 through rc.3. GA validation covers 3.11.x and 3.12.x, including historic fill through compacted data written by 3.11. An out-of-matrix pairing might *appear* to partly function—for example, 0.2.0 live-replicating from a 3.11 store—but it is not a supported mode and recovery paths will fail. **If the source server is upgraded underneath a running agent** past the agent’s matrix, the agent **halts everything, loudly**—same discipline as a protocol halt: ERROR with the observed vs supported format versions, a per-minute WARN, health `Halted`, and an hourly re-probe (covers a server rollback; an EDR upgrade restarts the agent anyway). Replication backlog during the halt is **owed work, not loss**: after upgrading EDR, the WAL tail drains normally and anything the source evicted meanwhile is recovered through the compacted files. ## Upgrade to EDR 1.0.0 ### Replace a release candidate For rc.1, rc.2, or rc.3 on a 3.11 source: 1. Stop the agent. 2. Replace the binary or container image with EDR 1.0.0. 3. Start it with the same configuration, state location, and source data location. 4. Check channel health and [outstanding work](/influxdb3/enterprise/edr/monitor/#outstanding-work-and-row-counts). No configuration change is required for rc-to-GA replacement. Keep the existing cursor and journals so the new agent resumes replication. If the source still runs 3.10, first follow [Upgrade a 3.10 source and a 0.2.0 agent](#upgrade-a-310-source-and-a-020-agent). ### Upgrade a 3.10 source and a 0.2.0 agent Keep every intermediate agent/server pairing within the support matrix. See the [compatibility table](#supported-source-server-versions). 1. Upgrade EDR to `1.0.0-0.rc.3`, preserving its configuration, state, and source data locations. State migrates automatically from 0.2.0. Review the rc.1 change to UTC bandwidth schedules in the [release notes](/influxdb3/enterprise/edr/release-notes/#v100-0rc1). 2. Upgrade the source server from 3.10 to 3.11. The rc.3 agent supports both lines. 3. Replace rc.3 with EDR 1.0.0 using [Replace a release candidate](#replace-a-release-candidate). 4. If required, upgrade the source from 3.11 to 3.12. The GA agent supports both lines. Do not upgrade the source directly from 3.10 to 3.12 while rc.3 is reading it. Rc.3 does not support a 3.12 source. Likewise, do not start the GA agent against a 3.10 source. > [!Warning] > Earlier builds can mislabel field values in tables wider than 100 fields. > After upgrading from 0.2.0, rc.1, or rc.2, verify field-level values downstream or re-replicate affected tables. > Row counts cannot detect this defect. > See the [rc.3 release notes](/influxdb3/enterprise/edr/release-notes/#v100-0rc3). ## Storage format details For details about the storage formats in the compatibility matrix, see[Storage engine terms](/influxdb3/enterprise/edr/reference/architecture/#storage-engine-terms)for what each one holds: | Format | What EDR uses it for | Stability | |--------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| | `.pt` WAL files | live replication (block shipping) | unchanged across 3.10, 3.11, and 3.12 | | `.ptsnap` snapshot manifests | snapshot boundary, historic planning | v3 across 3.10, 3.11, and 3.12 | |[Compaction](/influxdb3/enterprise/reference/internals/durability/#upgraded-storage-engine-compaction) checkpoints (`.ptv2chk`)—the cv2 tier|historic fill and gap recovery read compacted data once EDR’s WAL replicator can no longer see it|Versioned state stamps: v9 on 3.10; 3.11 and 3.12 can use v12 or v13. Checkpoint versions alone do not distinguish 3.11 from 3.12.| | Run-set indexes (`.idx`) | Tracked by the compatibility matrix | 3.12 writes v3, which the 3.11 reader rejects; rc.3 does not support 3.12 sources. | | Catalog | schema for encoding | versioned; compatibility tracked per release | ## Release compatibility summary Each entry in [Release notes](/influxdb3/enterprise/edr/release-notes/) lists the EDR protocol version and supported InfluxDB 3 Enterprise server versions for that release. It also identifies the Enterprise version and build revision used to build EDR. | Compatibility dimension | Compared components | Compatibility check | Incompatible behavior | | --- | --- | --- | --- | | Compatibility dimension | Compared components | Compatibility check | Incompatible behavior | | Agent protocol | Two EDR agents on one replication hop. | Protocol v3 agents negotiate on every /report . Protocol v1 and v2 agents negotiate at /connect . | The upstream agent halts the replication hop. Both agents report their protocol versions. | | Storage format | An EDR agent and the source InfluxDB 3 Enterprise store. | The agent checks the storage format at startup and while it runs. | The agent refuses to start with an incompatible store. If a running store becomes incompatible, the agent halts replication. | | EDR release | InfluxDB 3 Enterprise 3.10.x | 3.11.x | 3.12.x | Newer | | --- | --- | --- | --- | --- | | EDR release | InfluxDB 3 Enterprise 3.10.x | 3.11.x | 3.12.x | Newer | | 0.1.0 / 0.2.0 | Supported (built against 3.10.0) | Unsupported | Unsupported | Unsupported | | 1.0.0-0.rc.1 | Supported | Supported (built against 3.11.0) | Unsupported | Unsupported | | 1.0.0-0.rc.2 | Supported | Supported (built against 3.11.0) | Unsupported | Unsupported | | 1.0.0-0.rc.3 | Supported | Supported (built against 3.11.0) | Unsupported | Unsupported | | 1.0.0 GA | Unvalidated and unsupported | Supported | Supported (built against 3.12.0) | Unsupported | | Format | What EDR uses it for | Stability | | --- | --- | --- | | Format | What EDR uses it for | Stability | | .pt WAL files | live replication (block shipping) | unchanged across 3.10, 3.11, and 3.12 | | .ptsnap snapshot manifests | snapshot boundary, historic planning | v3 across 3.10, 3.11, and 3.12 | | Compaction checkpoints ( .ptv2chk )—the cv2 tier | historic fill and gap recovery read compacted data once EDR’s WAL replicator can no longer see it | Versioned state stamps: v9 on 3.10; 3.11 and 3.12 can use v12 or v13. Checkpoint versions alone do not distinguish 3.11 from 3.12. | | Run-set indexes ( .idx ) | Tracked by the compatibility matrix | 3.12 writes v3, which the 3.11 reader rejects; rc.3 does not support 3.12 sources. | | Catalog | schema for encoding | versioned; compatibility tracked per release | --- # EDR configuration file reference Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/config-file/ * EDR 1.0.0 Configuration is a YAML file passed to the agent with `--config`. Tokens are referenced **by name** and resolved at runtime from a token store directory (`--token-store`)—secrets never live in the config file. For a task-oriented walkthrough, see [Get started with EDR](/influxdb3/enterprise/edr/get-started/). ## Top-level fields | Field | Type |Required| |-------------|------|--------| | `name` |String| Yes | | `location` |Object| No | |`downstream` |Object| No\* | |`downstreams`| List | No\* | | `upstreams` | List | No\* | * **`name`**—node identity name. Used in topology, logs, and UI. * **`location`**—geographic location (`label`, `lat`, `lng`). Shown on map UI. * **`downstream`**—where this node sends data to. Shorthand for a one-entry `downstreams:` list. * **`downstreams`**—multiple downstream destinations (fan-out). See[Replicate to multiple destinations](/influxdb3/enterprise/edr/replicate/to-multiple-destinations/). Declaring both forms at once is a config error. * **`upstreams`**—who sends data to this node. \* At least one of `downstream`, `downstreams`, or `upstreams` must be present. A node with both is a regional hub: it receives from edges and forwards on. ## Downstream (source to destination) | Field | Type | Default | |----------------------------|------------------|--------------| | `name` | String | — | | `address` | String | — | | `auth_token` | String | — | | `mode` | `edr` | `direct` | `edr` | | `scope` | Object |Instance (all)| | `comms.interval_secs` | Integer | 10 | | `comms.degraded_after` | Integer | 3 | | `comms.unhealthy_after` | Integer | 6 | |`retry.initial_backoff_secs`| Integer | 1 | | `retry.max_backoff_secs` | Integer | 30 | | `retry.multiplier` | Integer | 2 | | `retry.halt_patience_secs` | Integer | 180 | | `share_topology` | Boolean | true | | `idempotent_writes` | Boolean | false | | `historic_fill` | Object | **required** | | `priorities` | List | absent | | `on_state_loss` |`recover` | `halt`| derived | | `encoding` | `lp` | `pt` | `lp` | | `concurrent_sends` | Integer | 1 | | `poll_interval_ms` | Integer | 1000 | | `bandwidth_schedule` | List | absent | | `bandwidth_timezone` | String | `"utc"` | * **`name`**—name of the destination node. * **`address`**—URL of the destination EDR agent (for example,`http://hub:9090`). * **`auth_token`**—token store key for authenticating to the destination. * **`mode`**—`edr`: agent-to-agent. `direct`: write directly to an InfluxDB v3 write API. * **`scope`**—which databases and tables to replicate. See[Scope](#scope). * **`comms.interval_secs`**—how often (seconds) to send status reports when idle. * **`comms.degraded_after`**—missed reports before the destination considers this channel degraded. * **`comms.unhealthy_after`**—missed reports before the destination considers this channel unhealthy. * **`retry.initial_backoff_secs`**—initial retry backoff after send failure. * **`retry.max_backoff_secs`**—maximum retry backoff. * **`retry.multiplier`**—exponential backoff multiplier. * **`retry.halt_patience_secs`**—patience window before persistent write errors or unclassified rejections trigger the halted state. See [The halted state](/influxdb3/enterprise/edr/monitor/#the-halted-state). * **`share_topology`**—include this node’s upstream tree in reports to the destination. When false, this node appears as a leaf. * **`idempotent_writes`**—user asserts no `(series_key, timestamp)`pair is ever written with differing field values. Unlocks`concurrent_sends > 1`, multi-ingest replication, priority reordering, and historic fill. See[Performance vs. correctness](#performance-vs-correctness) below. * **`historic_fill`**—declares the historic start point (`mode: none`| `full` | `since`). No default—absent means the agent refuses to start. Modes `full` and `since` require `idempotent_writes: true`. See [Historic fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill). * **`priorities`**—priority routing rules (first match wins). Requires`idempotent_writes: true`. See [Priorities](#priorities). * **`on_state_loss`**—what to do when a state journal and its previous-good mirror are both corrupt. Default: `recover` when`idempotent_writes: true`, else `halt`. Explicit `recover` without the idempotency assertion is rejected. See[State and recovery](/influxdb3/enterprise/edr/reference/state-and-recovery/). * **`encoding`**—wire encoding. `pt` (zstd-compressed PT) is agent-to-agent only, approximately 2.5x bandwidth saving. * **`concurrent_sends`**—maximum concurrent send tasks. `1` = strict-order delivery (the default). `> 1` requires`idempotent_writes: true`; `0` is rejected. * **`poll_interval_ms`**—WAL discovery poll interval. * **`bandwidth_schedule`**—time-of-day send rate control. See[Bandwidth scheduling](#bandwidth-scheduling). * **`bandwidth_timezone`**—timezone the whole `bandwidth_schedule` is interpreted in. IANA name (for example, `"Asia/Kolkata"`) or`"utc"`. ## Performance vs. correctness EDR’s default is correctness-first: strict-order delivery. One batch is in flight at a time (`concurrent_sends: 1`), held through retries, so data arrives at the destination in exactly the order it was written at the source. When the same point (identical series key and timestamp) is written more than once—an overwrite—InfluxDB resolves it last-write-wins by arrival order. Any concurrency that lets batches race can deliver an overwrite before the original it replaces. The original then arrives second and wins, so the destination reverts to the earlier value. Nothing is lost in transit, and every count matches, which makes this harder to detect than data loss. `idempotent_writes: true` is your assertion about the workload: no (series key, timestamp) pair is ever written with differing field values. Under that assertion, re-ordering and re-delivery cannot change the final stored values, which is what makes the throughput features below safe. The agent enforces the pairing at startup and on config reload—these are rejected without the assertion: | Feature | Why it needs the assertion | |--------------------------------|-------------------------------------------------------------| | `concurrent_sends > 1` |batches race to the destination; arrival order != write order| | `priorities` | deliberately reorders blocks within and across WAL files | |`historic_fill: full` or `since`| backfill overlaps live replication and re-sends data | | multiple ingest nodes | independent per-node WAL streams interleave | **Choosing a mode:** * **Workload has overwrites, upserts, or corrections** (or you can’t rule them out): leave the defaults—`idempotent_writes: false`,`concurrent_sends: 1`, `historic_fill: { mode: none }`. Throughput is bounded by round-trip latency per batch; size batches up rather than adding concurrency. * **Workload is append-only** (each point written once—typical sensor and metrics ingest): set `idempotent_writes: true` and raise`concurrent_sends` (for example, 4-16) for parallel delivery; historic fill and priorities become available. The assertion is about the *writers*, not about EDR: if any producer can ever rewrite a point with a different value, it is not idempotent— regardless of how rarely it happens. ### Wide tables and overwritten points A table wider than 100 fields stores one logical row as several field-family rows. With strict-order delivery, the destination merges the families in order. If delivery reorders writes to an overwritten point, each family slice resolves last-write-wins separately. One family can retain the newer write while another retains the older write, producing a row that never existed at the source. A query can also temporarily see a partially delivered row before all families arrive. Append-only workloads avoid this overwrite problem.`idempotent_writes: true` asserts duplicate-safe writes; it does not make updates safe under reordering. This caveat is separate from the field-name defect fixed in rc.3, which affected earlier builds even without overwrites. See the [rc.3 warning](/influxdb3/enterprise/edr/release-notes/#v100-0rc3). ## Upstreams (destination receiving sources) Each entry in the `upstreams` list: * **`name`**—expected source node name (must match the source’s`name`). * **`auth_token`**—token store key for verifying the source’s identity. Each upstream needs a unique auth token so the downstream can identify who connected. * **`write_token`**—token store key for writing the source’s data into local InfluxDB. ## Scope ```yaml # Replicate everything (default): scope: type: instance # Specific databases: scope: type: databases names: [sensors, metrics] # Specific tables: scope: type: tables refs: - database: sensors table: temperature # Everything EXCEPT named databases/tables (exclude wins; exact names only): scope: type: instance exclude: databases: [internal_edr] ``` |`scope.type`| Meaning | |------------|--------------------------------------------------------------------------| | `instance` |All databases on the source instance (the default when `scope` is absent).| |`databases` | A named list of databases (all tables within each). | | `tables` | Specific `{database, table}` pairs. | Scope filtering happens at block evaluation: out-of-scope blocks are dropped before queueing, so they consume no bandwidth. `exclude:` subtracts from whatever the scope includes—the natural way to say “replicate everything except local ops noise” without inverting the config into an allowlist. Two trade-offs to understand before choosing this shape: * **It is fail-open.** Under `type: instance` with exclusions, a database created tomorrow replicates automatically. That is the point—but if you scope for compliance or data-jurisdiction reasons, prefer the fail-closed allowlist (`type: databases`), where new databases never leave the node until you name them. When a new database enters scope this way, the agent logs it once (`new database entered replication scope`) so automatic growth is visible, not silent. * **Exclusions are validated at load.** Excluding something the include can’t reach, excluding under `type: tables`, duplicate entries, or excluding every included database are all rejected with the offender named—a config that would silently do nothing (or everything) fails fast instead. Config hot-reload picks up scope changes. Narrowing scope (adding an exclusion) takes effect immediately. Widening scope (removing an exclusion, adding a database) replicates new data going forward only— historic data for the newly added entity is not backfilled; re-run historic fill if the older data is owed. ## Priorities Ordered, first-match-wins list. Requires `idempotent_writes: true`. ```yaml priorities: - match: { type: tables, database: sensors, tables: [temperature, humidity] } recency: "1h" # data-time recency window: matches only priority: 1 # blocks whose max data time is within 1h of now share: 8 # DRR weight for this tier (optional) - match: { type: database, name: sensors } recency: "24h" priority: 2 - match: { type: all } priority: 3 historic_fill: true # historic fill traffic uses this tier ``` * **`match.type`** (`all` | `database` | `tables`)—what the rule matches. * **`recency`** (Duration)—optional data-time recency window; the rule matches only blocks whose max data time is within the window of now. * **`priority`** (Integer, 1-255)—lower is more urgent. Unique values become dispatch tiers. * **`share`** (Integer)—optional deficit round-robin (DRR) weight for this tier. * **`historic_fill`** (Boolean)—one rule can claim historic fill traffic for its tier. Otherwise historic fill rides the lowest-priority tier. See [Priorities](/influxdb3/enterprise/edr/reference/architecture/#priorities) for how tiers are scheduled. ## Bandwidth scheduling ```yaml downstream: # ... name, address, auth_token ... bandwidth_timezone: "Europe/London" # optional; default "utc" bandwidth_schedule: - name: "Factory shutdown" dates: ["2026-08-10", "2026-08-11"] # YYYY-MM-DD = this exact date only mode: silent - name: "Christmas" dates: ["12-25"] # MM-DD = recurring every year mode: unlimited - name: "Maintenance window" hours: "02:00-03:00" mode: silent silent_reports: true - name: "Weekday business hours" hours: "08:00-18:00" days: [mon, tue, wed, thu, fri] # optional; absent = every day mode: limited max_bytes_per_sec: 100000 - name: "Weekday overnight" hours: "18:00-08:00" # wraps midnight days: [mon, tue, wed, thu, fri] mode: unlimited - name: "Weekend" hours: "00:00-24:00" # whole day, written as an explicit range days: [sat, sun] mode: unlimited ``` * **`name`**—optional label, shown in the UI and logs. * **`hours`**—`"HH:MM-HH:MM"`, 24-hour, in the schedule’s timezone (see `bandwidth_timezone`). Wraps midnight (`"22:00-06:00"`).`"24:00"` is accepted as an end-of-day sentinel only, so a whole-day window can be written as `"00:00-24:00"` instead of the off-by-one `"00:00-23:59"`—as a *start* time it’s a config error. Mutually exclusive with `dates`. * **`days`**—optional list of `mon`, `tue`, `wed`, `thu`, `fri`,`sat`, and `sun` restricting which weekdays `hours` applies to. Absent = every day. A wrapping window (for example, `"22:00-06:00", days: [fri]`) is anchored to its**start day**—the post-midnight tail on Saturday still counts as Friday’s window. Combining `days` with `dates` is a config error. * **`dates`**—optional list of calendar-date overrides, matching the**whole day**: `"YYYY-MM-DD"` (one-time—this exact date only) or`"MM-DD"` (no year—recurs every year, for example `"12-25"`). An entry has either `hours` (with optional `days`) or `dates`, never both—combining them, or setting neither, is a config error. * **`mode`**—`unlimited` (full speed), `limited` (throttled to`max_bytes_per_sec`), or `silent` (no data sends). * **`max_bytes_per_sec`**—required for `limited`. * **`silent_reports`**—for `silent` mode: keep sending status reports (default true). When false, no traffic at all—the downstream marks the channel unhealthy for the duration. Entries are evaluated in order; the first matching entry wins, so more specific rules (a weekend override, a calendar exception) must come before the general rule they’re meant to carve an exception out of. When`bandwidth_schedule` is absent, or nothing matches at a given moment, the pipeline runs unlimited. Data is never dropped by scheduling—it queues during limited or silent windows and drains when the window changes. `bandwidth_timezone` sets the zone the whole schedule is interpreted in:`"utc"` (the default) or an IANA name like `"Asia/Kolkata"`, which also handles daylight-saving correctly—unlike hand-offsetting a UTC window, which drifts an hour for half the year. > [!Important] > #### Bandwidth schedules changed to UTC by default at the 1.0.0 release candidate > > Schedules previously ran in process-local time—which was already UTC > inside containers, and whatever the host zone happened to be on bare > metal. If you relied on host-local hours, set `bandwidth_timezone` to that > zone. Schedule entries also now reject unknown keys, `"24:00"` as a start > time, two-digit years in `dates`, and an empty `days:` list—each of these > previously mis-parsed silently. The agent validates the schedule at load time (and on hot reload) and logs advisory warnings—never load errors—for rules that are fully or partially shadowed by an earlier rule, weekly coverage gaps that silently default to unlimited, and duplicate or conflicting calendar dates. These also appear in `/edr/v1/metrics` (`bandwidth_warnings`) and the UI. ## Direct mode ```yaml downstream: name: "cloud-influx" address: "https://cloud.influxdata.com" auth_token: "cloud-api-token" mode: direct ``` When the destination does not run an EDR agent, replicate directly to a plain InfluxDB v3 write endpoint. In direct mode there is no `/connect`handshake, health checks use the destination’s `GET /health`, and no topology is propagated. Auth failures and schema conflicts are handled identically to agent-to-agent mode. The `pt` wire encoding is not available in direct mode. ## Configuration reload The config file is polled for changes. Reloadable without restart: scope, comms expectations, retry parameters. Requiring a restart: node name, upstream list, destination address, destination mode, `idempotent_writes`, priority rules, token store path, data directory, listen address. The agent logs the reason when a change is rejected for hot-reload. ## Multi-ingest node support When InfluxDB 3 Enterprise runs as a cluster with multiple ingest nodes sharing an object store, each node produces WAL files under its own path prefix. Set `idempotent_writes: true` and the agent discovers all node prefixes automatically, tracking an independent WAL cursor per node. The user guarantee behind `idempotent_writes`: no `(series_key, timestamp)`pair is ever written with differing field values across any ingest node. Duplicate writes with identical values are fine. If your load balancer hashes by series key, the guarantee holds automatically. With `idempotent_writes: false` (default), the agent **refuses to start**if multiple ingest nodes are detected—this prevents silent data corruption. Cursors are stored per node in the state file; an upgrade from a single-node deployment migrates the existing cursor automatically. | Field | Type | Required | | --- | --- | --- | | Field | Type | Required | | name | String | Yes | | location | Object | No | | downstream | Object | No* | | downstreams | List | No* | | upstreams | List | No* | | Field | Type | Default | | --- | --- | --- | | Field | Type | Default | | name | String | — | | address | String | — | | auth_token | String | — | | mode | edr | direct | edr | | scope | Object | Instance (all) | | comms.interval_secs | Integer | 10 | | comms.degraded_after | Integer | 3 | | comms.unhealthy_after | Integer | 6 | | retry.initial_backoff_secs | Integer | 1 | | retry.max_backoff_secs | Integer | 30 | | retry.multiplier | Integer | 2 | | retry.halt_patience_secs | Integer | 180 | | share_topology | Boolean | true | | idempotent_writes | Boolean | false | | historic_fill | Object | required | | priorities | List | absent | | on_state_loss | recover | halt | derived | | encoding | lp | pt | lp | | concurrent_sends | Integer | 1 | | poll_interval_ms | Integer | 1000 | | bandwidth_schedule | List | absent | | bandwidth_timezone | String | "utc" | | Feature | Why it needs the assertion | | --- | --- | | Feature | Why it needs the assertion | | concurrent_sends > 1 | batches race to the destination; arrival order != write order | | priorities | deliberately reorders blocks within and across WAL files | | historic_fill: full or since | backfill overlaps live replication and re-sends data | | multiple ingest nodes | independent per-node WAL streams interleave | | scope.type | Meaning | | --- | --- | | scope.type | Meaning | | instance | All databases on the source instance (the default when scope is absent). | | databases | A named list of databases (all tables within each). | | tables | Specific {database, table} pairs. | --- # Glossary Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/glossary/ * EDR 1.0.0 [A](#a) | [B](#b) | [C](#c) | [D](#d) | [E](#e) | [F](#f) | [G](#g) | [H](#h) | [I](#i) | [J](#j) | [K](#k) | [L](#l) | [M](#m) | [N](#n) | [O](#o) | [P](#p) | [Q](#q) | [R](#r) | [S](#s) | [T](#t) | [U](#u) | [V](#v) | [W](#w) | X | Y | Z A ---------- ### abstract syntax tree (AST) Tree representation of source code that shows the structure, content, and rules of programming statements and discards additional syntax elements. The tree is hierarchical, with elements of program statements broken down into their parts. For more information about AST design, see [Abstract Syntax Tree on Wikipedia](https://en.wikipedia.org/wiki/Abstract_syntax_tree). ### agent A background process started by (or on behalf of) a user that typically requires user input. [Telegraf](/telegraf/v1/) is an agent that requires user input (a configuration file) to gather metrics from declared input plugins and sends metrics to declared output plugins, based on the plugins enabled for a configuration. Related entries:[input plugin](#input-plugin),[output plugin](#output-plugin),[daemon](#daemon) ### aggregator plugin Receives metrics from input plugins, creates aggregate metrics, and then passes aggregate metrics to configured output plugins. Related entries:[input plugin](#input-plugin),[output plugin](#output-plugin),[processor plugin](#processor-plugin) ### aggregate A function that returns an aggregated value across a set of points. For a list of available aggregation functions, see [SQL aggregate functions](/influxdb3/enterprise/reference/sql/functions/aggregate/). Related entries:[function](#function),[selector](#selector) ### API Application programming interface that facilitates and standardizes communication between two or more computer programs. ### argument A value passed to a function or command that determines how the process operates. Related entries:[parameter](#parameter) B ---------- ### batch A collection of points in line protocol format, separated by newlines (`0x0A`). Submitting a batch of points using a single HTTP request to the write endpoints drastically increases performance by reducing the HTTP overhead. InfluxData typically recommends batch sizes of 5,000-10,000 points. In some use cases, performance may improve with significantly smaller or larger batches. Related entries:[line protocol](#line-protocol-lp),[point](#point) ### batch size The number of lines or individual data points in a line protocol batch. The Telegraf agent sends metrics to output plugins in batches rather than individually. Batch size controls the size of each write batch that Telegraf sends to the output plugins. Related entries:[output plugin](#output-plugin) ### bin In a cumulative histogram, a bin includes all data points less than or equal to a specified upper bound. In a normal histogram, a bin includes all data points between the upper and lower bounds. Histogram bins are also sometimes referred to as “buckets.” ### boolean A data type with two possible values: true or false. By convention, you can express `true` as the integer `1` and false as the integer `0` (zero). ### bucket “Bucket” is the term used in InfluxDB 2.x and *InfluxDB Cloud Serverless* to refer to a named location where time series data is stored. Bucket is synonymous with “database” when using InfluxDB 3 Enterprise. Related entries:[database](#database) C ---------- ### CSV Comma-separated values (CSV) delimits text between commas to separate values. A CSV file stores tabular data (numbers and text) in plain text. Each line of the file is a data row. Each row consists of one or more columns, separated by commas. CSV file format is not fully standardized. ### cardinality Cardinality is the number of unique values in a set. Series cardinality is the number of unique [series](#series) in a database as a whole. With the InfluxDB 3 storage engine, high series cardinality *does not* affect performance. ### cluster A collection of servers or processes that work together as a single unit. An InfluxDB cluster is a collection of InfluxDB servers dedicated to the workload of a single customer. ### collect Collect and write time series data to InfluxDB using line protocol and any of the following tools: * Telegraf * the InfluxDB v1, v2, or v3 HTTP APIs * `influxdb3` command line interface (CLI) for InfluxDB 3 Core and Enterprise * InfluxDB 3 Processing engine with the `LineBuilder` and `influxdb3_local` shared API. * v1 or v2 `influx` command line interface (CLI) * InfluxDB v1, v2, or v3 client libraries ### collection interval The default global interval for collecting data from each Telegraf input plugin. The collection interval can be overridden by each individual input plugin’s configuration. Related entries:[input plugin](#input-plugin) ### collection jitter Collection jitter prevents every input plugin from collecting metrics simultaneously, which can have a measurable effect on the system. For each collection interval, every Telegraf input plugin sleeps for a random time between zero and the collection jitter before collecting the metrics. Related entries:[collection interval](#collection-interval),[input plugin](#input-plugin) ### column InfluxDB data is stored in tables within rows and columns. Columns store tag sets and fields sets, and time values. The only required column is *time*, which stores timestamps and is included in all InfluxDB tables. ### common log format (CLF) A standardized text file format used by the InfluxDB server to create log entries when generating server log files. ### compaction Compressing time series data to optimize disk usage. ### continuous query (CQ) Continuous queries are a feature of InfluxDB 1.x used to regularly downsample or process time series data. D ---------- ### daemon A background process that runs without user input. ### dashboard A collection of data visualizations, charts, and panels organized in a single view to monitor and analyze time series data. Dashboards provide at-a-glance visualization of metrics and allow users to track trends, spot anomalies, and understand relationships between different data points over time. Some versions of InfluxDB include built-in dashboarding features. InfluxDB can integrate with third-party visualization and dashboard tools, such as[Grafana](https://grafana.com), [Apache Superset](https://superset.apache.org/),[Tableau](https://www.tableau.com/). ### data model A data model organizes elements of data and standardizes how they relate to one another and to properties of the real world entities. For more information, see the [InfluxDB 3 Enterprise data model](/influxdb3/enterprise/get-started/#data-model) ### data service Stores time series data and handles writes and queries. ### data source A source of data that InfluxDB collects or queries data from. Related entries:[database](#database) ### data type A data type is defined by the values it can take, the programming language used, or the operations that can be performed on it. InfluxDB supports the following data types: * string * boolean * float (64-bit) * integer (64-bit) * unsigned integer (64-bit) * time For more information about different data types, see: * [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/#data-types-and-format) * [InfluxQL](/influxdb3/enterprise/reference/influxql/#literals) * [SQL](/influxdb3/enterprise/reference/sql/data-types/#sql-and-arrow-data-types) #### database A named location where time series data is stored. In InfluxDB 1.x, a database represented a logical container for users, retention policies, continuous queries, and time series data. In InfluxDB 2.x and InfluxDB Cloud Serverless, the equivalent of this concept is an InfluxDB [bucket](#bucket). Related entries:[bucket](#bucket),[retention policy](#retention-policy-rp) ### date-time InfluxDB stores the date-time format for each data point in a timestamp with nanosecond-precision Unix time. Specifying a timestamp is optional. If a timestamp isn’t specified for a data point, InfluxDB uses the server’s local nanosecond timestamp in UTC. ### downsample Aggregating high resolution data into lower resolution data to preserve disk space. ### downstream (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the node an [upstream](#upstream-edr) node sends replicated data to. A node can be downstream-only (a sink), or both upstream and downstream at once (a regional relay). Related entries:[upstream (EDR)](#upstream-edr),[sidecar agent](#sidecar-agent) ### duration A data type that represents a duration of time–for example, `1s`, `1m`, `1h`, `1d`. Retention periods are set using durations. Related entries:[retention period](#retention-period) E ---------- ### event Metrics gathered at irregular time intervals. ### expression A combination of one or more constants, variables, operators, and functions. In the following SQL example, `now() - INTERVAL '7 days'` is an expression that calculates the difference between the `now()` function expression and the duration represented by `INTERVAL '7 days`: ```sql SELECT * FROM home WHERE time >= now() - INTERVAL '7 days' ``` F ---------- ### field A key-value pair in InfluxDB’s data structure that records a data value. Generally, field values change over time. Fields are required in InfluxDB’s data structure. Related entries:[field key](#field-key),[field set](#field-set),[field value](#field-value),[tag](#tag) ### field key The key of the key-value pair. Field keys are strings. Related entries:[field](#field),[field set](#field-set),[field value](#field-value),[tag key](#tag-key) ### field set The collection of field key-value pairs. Related entries:[field](#field),[field key](#field-key),[field value](#field-value),[point](#point) ### field value The value of a key-value pair. Field values are the actual data; they can be strings, floats, integers, unsigned integers or booleans. A field value is always associated with a timestamp. Related entries:[field](#field),[field key](#field-key),[field set](#field-set),[tag set](#tag-set),[tag value](#tag-value),[timestamp](#timestamp) ### file block A file block is a fixed-length chunk of data read into memory when requested by an application. ### float A real number written with a decimal point dividing the integer and fractional parts (`1.0`, `3.14`, `-20.1`). InfluxDB supports 64-bit float values. ### flush interval The global interval for flushing data from each Telegraf output plugin to its destination. This value should not be set lower than the collection interval. Related entries:[collection interval](#collection-interval),[flush jitter](#flush-jitter),[output plugin](#output-plugin) ### flush jitter Flush jitter prevents every Telegraf output plugin from sending writes simultaneously, which can overwhelm some data sinks. Each flush interval, every Telegraf output plugin sleeps for a random time between zero and the flush jitter before emitting metrics. Flush jitter smooths out write spikes when running a large number of Telegraf instances. Related entries:[flush interval](#flush-interval),[output plugin](#output-plugin) ### function A function is an operation that performs a specific task. Functions take input, operate on that input, and then return output. For complete lists of available query language functions, see: * [InfluxQL functions](/influxdb3/enterprise/reference/influxql/functions/) * [SQL functions](/influxdb3/enterprise/reference/sql/functions/). Related entries:[aggregate](#aggregate),[selector](#selector) G ---------- ### gap fill (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the process that recovers a WAL file the replicator expected but that was already deleted (snapshotted and evicted) before it could be sent–typically after a long outage with short WAL retention. The agent detects the missing WAL ID range, records it in a gap ledger, and recovers the data from later-generation files. Related entries:[historic fill (EDR)](#historic-fill-edr),[WAL replication (EDR)](#wal-replication-edr) ### gzip gzip is a type of data compression that compress chunks of data, which is restored by unzipping compressed gzip files. The gzip file extension is `.gz`. H ---------- ### halted state (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the state an agent enters when the downstream persistently refuses data that exists and could be delivered–for example, a schema conflict, an authentication failure, or a malformed batch. Dispatch stops on every tier until the cause is fixed; replication then auto-resumes without a restart. Related entries:[downstream (EDR)](#downstream-edr) ### histogram A visual representation of statistical information that uses rectangles to show the frequency of data items in successive, equal intervals or bins. ### historic fill (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the process that replicates data that existed on the source before EDR was deployed (or before a destination was added), backfilling from snapshot and compacted files below the historic/live boundary. Related entries:[gap fill (EDR)](#gap-fill-edr),[WAL replication (EDR)](#wal-replication-edr) I ---------- ### identifier Identifiers are tokens that refer to specific database objects such as database names, field keys, measurement names, tag keys, etc. Related entries:[database](#database),[field key](#field-key),[measurement](#measurement),[tag key](#tag-key) ### influx `influx` is a command line interface (CLI) that interacts with the InfluxDB v1.x and v2.x server. ### influxctl [`influxctl`](/influxdb3/cloud-dedicated/reference/cli/influxctl/) is a CLI that performs [administrative tasks](/influxdb3/cloud-dedicated/admin/) for an InfluxDB Cloud Dedicated cluster. ### influxd `influxd` is the InfluxDB OSS v1.x and v2.x daemon that runs the InfluxDB server and other required processes. ### InfluxDB An open source time series database (TSDB) developed by InfluxData, optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics. ### influxdb3 `influxdb3` is: * the InfluxDB 3 Core and Enterprise daemon that runs the InfluxDB 3 server * the InfluxDB 3 CLI that interacts with the server for InfluxDB 3 Core and Enterprise ### InfluxQL The SQL-like query language used to query data in InfluxDB. ### input plugin Telegraf input plugins actively gather metrics and deliver them to the core agent, where aggregator, processor, and output plugins can operate on the metrics. To activate an input plugin, enable and configure it in the Telegraf configuration file. Related entries:[aggregator plugin](#aggregator-plugin),[collection interval](#collection-interval),[output plugin](#output-plugin),[processor plugin](#processor-plugin) ### influxdb3 CLI The command line interface for InfluxDB 3. Use the `influxdb3` CLI to administer your instance–for example, to create and manage databases, tables, and tokens–and to write and query data. ### instance An entity comprising data on a server (or virtual server in cloud computing). ### integer A whole number that is positive, negative, or zero (`0`, `-5`, `143`). InfluxDB supports 64-bit integers (minimum: `-9223372036854775808`, maximum: `9223372036854775807`). Related entries:[unsigned integer](#unsigned-integer) ### IOx The IOx storage engine (InfluxDB 3 storage engine) is a real-time, columnar database optimized for time series data built in Rust on top of[Apache Arrow](https://arrow.apache.org/) and[DataFusion](https://arrow.apache.org/datafusion/user-guide/introduction.html). IOx replaces the [TSM (Time Structured Merge tree)](#tsm-time-structured-merge-tree) storage engine. Related entries:[WAL tail](#wal-tail) For InfluxDB 3 Enterprise-specific detail, see[Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). J ---------- ### JWT Typically, JSON web tokens (JWT) are used to authenticate users between an identity provider and a service provider. A server can generate a JWT to assert any business processes. For example, an “admin” token sent to a client can prove the client is logged in as admin. Tokens are signed by one party’s private key (typically, the server). Private keys are used by both parties to verify that a token is legitimate. JWT uses an open standard specified in [RFC 7519](https://tools.ietf.org/html/rfc7519). ### Jaeger Open source tracing used in distributed systems to monitor and troubleshoot transactions. ### JSON JavaScript Object Notation (JSON) is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types. K ---------- ### keyword A keyword is reserved by a program because it has special meaning. Every programming language has a set of keywords (reserved names) that cannot be used as identifiers–for example, you can’t use `SELECT` (an SQL keyword) as a variable name in an SQL query. See keyword lists: * [SQL keywords](/influxdb3/enterprise/reference/sql/#keywords) * [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords) L ---------- ### literal A literal is a value in an expression, a number, character, string, function, record, or array. Literal values are interpreted as defined. ### load balancing Improves workload distribution across multiple computing resources in a network. Load balancing optimizes resource use, maximizes throughput, minimizes response time, and avoids overloading a single resource. Using multiple components with load balancing instead of a single component may increase reliability and availability. If requests to any server in a network increase, requests are forwarded to another server with more capacity. Load balancing can also refer to the communications channels themselves. ### logs Logs record information. Event logs describe system events and activity that help to describe and diagnose problems. Transaction logs describe changes to stored data that help recover data if a database crashes or other errors occur. ### line protocol (LP) The text based format for writing points to InfluxDB. See [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/). M ---------- ### measurement The part of InfluxDB’s data structure that describes the data stored in associated fields. Measurements are strings. Related entries:[field](#field), [series](#series) ### metric Data tracked over time. ### metric buffer The metric buffer caches individual metrics when writes are failing for an Telegraf output plugin. Telegraf will attempt to flush the buffer upon a successful write to the output. The oldest metrics are dropped first when this buffer fills. Related entries:[output plugin](#output-plugin) ### missing values Denoted by a null value. Identifies missing information, which may be useful to include in an error message. N ---------- ### node An independent process or server in a cluster. Related entries:[cluster](#cluster),[server](#server) ### now The local server’s nanosecond timestamp. ### null A data type that represents a missing or unknown value. Denoted by the `null` value. Values of [tags](#tag) and [fields](#field) may be `null`, but timestamp values are never `null`. O ---------- ### operator A symbol that usually represents an action or process. For example: `+`, `-`, `>`. Related entries:[operand](#operand) ### operand The object or value on either side of an [operator](#operator). Related entries:[operator](#operator) ### organization An InfluxDB v2 concept that describes workspace for a group of users. All InfluxDB v2 dashboards, tasks, buckets, members, and so on, belong to an organization. Organizations are not part of InfluxDB 3 Enterprise. ### owner A type of role for a user. Owners have read/write permissions. Users can have owner roles for databases and other resources. Role permissions are separate from API token permissions. For additional information on API tokens, see [token](#token). ### output plugin Telegraf output plugins deliver metrics to their configured destination. To activate an output plugin, enable and configure the plugin in Telegraf’s configuration file. Related entries:[aggregator plugin](#aggregator-plugin),[flush interval](#flush-interval),[input plugin](#input-plugin),[processor plugin](#processor-plugin) P ---------- ### parameter A key-value pair used to pass information to a function that determines how the function operates. Related entries:[argument](#argument) ### pipe Method for passing information from one process to another. For example, an output parameter from one process is input to another process. Information passed through a pipe is retained until the receiving process reads the information. ### plugin A Python file with a specific function signature that corresponds to a[trigger](#trigger) type. Plugins run in the InfluxDB 3 [Processing engine](#processing-engine) to process data, respond to database events, and connect to external systems. Related entries:[Processing engine](#processing-engine),[trigger](#trigger) ### point Single data record identified by its *measurement*, *tag keys*, *tag values*,*field key*, and *timestamp*. In a [series](#series), each point has a unique timestamp. If you write a point to a series with a timestamp that matches an existing point, the field set becomes a union of the old and new field set, where any ties go to the new field set. Related entries:[measurement](#measurement),[tag set](#tag-set),[field set](#field-set),[timestamp](#timestamp) ### primary key With the InfluxDB 3 storage engine, the primary key is the list of columns used to uniquely identify each row in a table. Rows are uniquely identified by their timestamp and tag set. A row’s primary key tag set does not include tags with null values. ### precision The precision configuration setting determines the timestamp precision retained for input data points. All incoming timestamps are truncated to the specified precision. Valid precisions are `ns`, `us` or `µs`, `ms`, and `s`. In Telegraf, truncated timestamps are padded with zeros to create a nanosecond timestamp. Telegraf output plugins emit timestamps in nanoseconds. For example, if the precision is set to `ms`, the nanosecond epoch timestamp `1480000000123456789` is truncated to `1480000000123` in millisecond precision and padded with zeroes to make a new, less precise nanosecond timestamp of `1480000000123000000`. Telegraf output plugins do not alter the timestamp further. The precision setting is ignored for service input plugins. Related entries:[aggregator plugin](#aggregator-plugin),[input plugin](#input-plugin),[output plugin](#output-plugin),[processor plugin](#processor-plugin),[service input plugin](#service-input-plugin) ### predicate expression A predicate expression compares two values and returns `true` or `false` based on the relationship between the two values. A predicate expression is comprised of a left operand, a comparison operator, and a right operand. ### process A set of predetermined rules. A process can refer to instructions being executed by the computer processor or refer to the act of manipulating data. ### Processing engine The Processing engine is a Python virtual machine (VM) embedded within InfluxDB 3 Core and Enterprise for automatically processing data and responding to database events. It executes Python plugins in response to events defined by triggers. The Processing engine runs Python code directly in your database, allowing plugins to react to specific triggers without requiring external services. Related entries:[plugin](#plugin),[trigger](#trigger) ### processor plugin Telegraf processor plugins transform, decorate, and filter metrics collected by input plugins, passing the transformed metrics to the output plugins. Related entries:[aggregator plugin](#aggregator-plugin),[input plugin](#input-plugin),[output plugin](#output-plugin) ### Prometheus format A simple text-based format for exposing metrics and ingesting them into Prometheus. Q ---------- ### query A request for information. An InfluxDB query returns time series data. See [Query data in InfluxDB](/influxdb3/enterprise/query-data/). ### query group An operator-defined, ordered list of query nodes in an InfluxDB 3 Enterprise cluster. Each query group has a name and a replication factor. Query groups are intended to define how query nodes participate in distributed query processing, but aren’t yet operational as of v3.11–the CLI commands store query group definitions in the catalog, but the server doesn’t use them to affect query routing, data placement, or replication. The order of query nodes in a group is significant. When you create or update a group, provide node IDs in the intended order. ### query plan A sequence of steps (*nodes*) that the InfluxDB Querier devises and executes to calculate the result of the query in the least amount of time. A *logical plan* is a high level representation of a query and doesn’t consider cluster configuration or data organization. A *physical plan* represents the query execution plan and data flow through plan nodes that read (*scan*), deduplicate, merge, filter, and sort data. A physical plan is optimized for the cluster configuration and data organization. R ---------- ### REPL A Read-Eval-Print Loop (REPL) is an interactive programming environment where you type a command and immediately see the result. ### regular expressions Regular expressions (regex or regexp) are patterns used to match character combinations in strings. ### rejected points In a batch of data, points that InfluxDB couldn’t write to a database. Field type conflicts are a common cause of rejected points. ### retention period The [duration](#duration) of time that a database retains data. InfluxDB drops points with timestamps older than their database’s retention period relative to [now](#now). The minimum retention period is **one hour**. In InfluxDB Cloud Serverless, *bucket* is synonymous with database. Related entries:[bucket](#bucket)[database](#database) ### retention policy (RP) A retention policy is part of the InfluxDB 1.x data model that describes how long InfluxDB keeps data (duration), how many copies of the data to store when in a in the cluster (replication factor), and the time range covered by shard groups (shard group duration). RPs are unique per database and along with the measurement and tag set define a series. In InfluxDB 3 Enterprise, the equivalent is [retention period](#retention-period), however retention periods are not part of the data model. The retention period describes the data persistence behavior of a database. In InfluxDB Cloud Serverless, *bucket* is synonymous with database. Related entries:[retention period](#retention-period), ### RFC3339 timestamp A timestamp that uses the human-readable DateTime format proposed in[RFC 3339](https://tools.ietf.org/html/rfc3339) (for example: `2020-01-01T00:00:00.00Z`). Related entries:[RFC3339Nano timestamp](#rfc3339nano-timestamp),[timestamp](#timestamp),[unix timestamp](#unix-timestamp) ### RFC3339Nano timestamp A [Golang representation of the RFC 3339 DateTime format](https://go.dev/src/time/format.go)that uses nanosecond resolution–for example:`2006-01-02T15:04:05.999999999Z07:00`. InfluxDB clients can return RFC3339Nano timestamps in log events and CSV-formatted query results. Related entries:[RFC3339 timestamp](#rfc3339-timestamp),[timestamp](#timestamp),[unix timestamp](#unix-timestamp) ### row A row in a [table](#table) represents a specific record or instance of data.[Column](#column) values in a row represent specific attributes or properties of the instance. Each row has a [primary key](#primary-key) that makes the row unique from other rows in the table. Related entries:[column](#column),[primary key](#primary-key),[series](#series),[table](#table) S ---------- ### schema How data is organized in InfluxDB. The fundamentals of the InfluxDB schema are databases, measurements, tag keys, tag values, and field keys. Related entries:[bucket](#bucket),[database](#database),[field key](#field-key),[measurement](#measurement),[series](#series),[tag key](#tag-key),[tag value](#tag-value) ### secret Secrets are key-value pairs that contain information you want to control access to, such as API keys, passwords, or certificates. ### selector A function that returns a single point from the range of specified points. See [SQL selector functions](/influxdb3/enterprise/reference/sql/functions/selector/)for a complete list of available SQL selector functions. Related entries:[aggregate](#aggregate),[function](#function),[transformation](#transformation) ### series In the InfluxDB 3 data structure, a collection of data that share a common*measurement* and *tag set*. Related entries:[field set](#field-set),[measurement](#measurement),[tag set](#tag-set) ### series cardinality The number of unique measurement, tag set, and field key combinations in an InfluxDB 3 Enterprise database. For example, assume that an InfluxDB database has one measurement. The single measurement has two tag keys: `email` and `status`. If there are three different `email` tag values, and each email address is associated with two different `status` tag values, then the series cardinality for the measurement is 6 (3 × 2 = 6): | email |status| |--------------------|------| |lorr@influxdata.com |start | |lorr@influxdata.com |finish| |marv@influxdata.com |start | |marv@influxdata.com |finish| |cliff@influxdata.com|start | |cliff@influxdata.com|finish| In some cases, performing this multiplication may overestimate series cardinality because of the presence of dependent tags. Dependent tags are scoped by another tag and do not increase series cardinality. If we add the tag `firstname` to the preceding example, the series cardinality would not be 18 (3 × 2 × 3 = 18). The series cardinality would remain unchanged at 6, as `firstname` is already scoped by the `email` tag: | email |status|firstname| |--------------------|------|---------| |lorr@influxdata.com |start |lorraine | |lorr@influxdata.com |finish|lorraine | |marv@influxdata.com |start | marvin | |marv@influxdata.com |finish| marvin | |cliff@influxdata.com|start |clifford | |cliff@influxdata.com|finish|clifford | Related entries:[field key](#field-key),[measurement](#measurement),[tag key](#tag-key),[tag set](#tag-set) ### series key A series key identifies a particular series by measurement, tag set, and field key. For example: ```text # measurement, tag set, field key h2o_level, location=santa_monica, h2o_feet ``` Related entries:[series](#series) ### server A computer, virtual or physical, running InfluxDB. ### service input plugin Telegraf input plugins that run in a passive collection mode while the Telegraf agent is running. Service input plugins listen on a socket for known protocol inputs, or apply their own logic to ingested metrics before delivering metrics to the Telegraf agent. Related entries:[aggregator plugin](#aggregator-plugin),[input plugin](#input-plugin),[output plugin](#output-plugin),[processor plugin](#processor-plugin) ### sidecar agent A process that runs alongside a primary application to extend it without modifying it. [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/) runs as a sidecar agent (`influxdb3-edr`) beside an unmodified InfluxDB 3 Enterprise instance, reading its object store directly. Related entries:[downstream (EDR)](#downstream-edr),[upstream (EDR)](#upstream-edr) ### string A data type used to represent text. T ---------- ### TCP Transmission Control Protocol. ### table A collection of related data organized in a structured way with a predefined set of columns and data types. Each row in the table represents a specific record or instance of the data, and each column represents a specific attribute or property of the data. In InfluxDB 3 Enterprise, a table represents a measurement. Related entries:[column](#column),[measurement](#measurement),[primary key](#primary-key),[row](#row) ### tag The key-value pair in InfluxDB’s data structure that records metadata. Tags are an optional part of InfluxDB’s data structure but they are useful for storing commonly queried metadata. Related entries:[field](#field),[tag key](#tag-key),[tag set](#tag-set),[tag value](#tag-value) ### tag key The key of a tag key-value pair. Tag keys are strings and store metadata. Related entries:[field key](#field-key),[tag](#tag),[tag set](#tag-set),[tag value](#tag-value) ### tag set The collection of tag keys and tag values on a point. Related entries:[point](#point),[primary key](#primary-key),[series](#series),[tag](#tag),[tag key](#tag-key),[tag value](#tag-value) ### tag value The value of a tag key-value pair. Tag values are strings and they store metadata. Related entries:[tag](#tag),[tag key](#tag-key),[tag set](#tag-set) ### Telegraf A plugin-driven agent that collects, processes, aggregates, and writes metrics. Related entries:[Telegraf plugins](/telegraf/v1/plugins/),[Use Telegraf to collect data](/influxdb3/enterprise/write-data/use-telegraf/), ### time (data type) A data type that represents a single point in time with nanosecond precision. ### time series data Sequence of data points typically consisting of successive measurements made from the same source over a time interval. Time series data shows how data evolves over time. On a time series data graph, one of the axes is always time. Time series data may be regular or irregular. Regular time series data changes in constant intervals. Irregular time series data changes at non-constant intervals. ### timestamp The date and time associated with a point. Time in InfluxDB is in UTC. To specify time when writing data, see[Elements of line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/#elements-of-line-protocol). Related entries:[point](#point),[unix timestamp](#unix-timestamp),[RFC3339 timestamp](#rfc3339-timestamp) ### token Tokens provide authorization to perform specific actions in InfluxDB. InfluxDB 3 Enterprise supports different types of authorization tokens: * **Admin token:** A token that grants full access to InfluxDB 3 server actions. * **Resource token:** Tokens that grant read and write access to server resources, such as databases and system information. Database tokens allow for reading and writing data in your InfluxDB 3 Enterprise instance. System tokens allow for reading system information and metrics for your server. For more information, see [Manage tokens](/influxdb3/enterprise/admin/tokens/). ### transformation Data transformation refers to the process of converting or modifying input data from one format, value, or structure to another. InfluxQL [transformation functions](/influxdb3/enterprise/reference/influxql/functions/transformations/)modify and return values in each row of queried data, but do not return an aggregated value across those rows. Related entries: [aggregate](#aggregate), [function](#function), [selector](#selector) ### trigger With InfluxDB 3 Core and Enterprise, a Processing engine trigger connects a Processing engine plugin to a specific type of event. A trigger has: * a *trigger specification* that defines the event (data ingestion, a time interval or schedule, or an HTTP request) that runs the plugin. * a plugin file that contains the code to run when the trigger fires. * optional configuration parameters that are passed to the plugin. Related entries:[plugin](#plugin),[Processing engine](#processing-engine) ### TSM (Time Structured Merge tree) The InfluxDB v1 and v2 data storage format that allows greater compaction and higher write and read throughput than B+ or LSM tree implementations. The TSM storage engine has been replaced by the [InfluxDB 3 storage engine (IOx)](#iox). Related entries:[IOx](#iox) U ---------- ### UDP User Datagram Protocol is a packet of information. When a request is made, a UDP packet is sent to the recipient. The sender doesn’t verify the packet is received. The sender continues to send the next packets. This means computers can communicate more quickly. This protocol is used when speed is desirable and error correction is not necessary. ### unix epoch The date and time from which Unix system times are measured. The Unix epoch is `1970-01-01T00:00:00Z`. ### unix timestamp Counts time since **Unix Epoch (1970-01-01T00:00:00Z UTC)** in specified units ([precision](#precision)). Specify timestamp precision when [writing data to InfluxDB](/influxdb3/enterprise/write-data/). InfluxDB supports the following unix timestamp precisions: |Precision|Description | Example | |---------|------------|---------------------| | `ns` |Nanoseconds |`1577836800000000000`| | `us` |Microseconds| `1577836800000000` | | `ms` |Milliseconds| `1577836800000` | | `s` | Seconds | `1577836800` | *The examples above represent **2020-01-01T00:00:00Z UTC**.* Related entries:[timestamp](#timestamp),[RFC3339 timestamp](#rfc3339-timestamp) ### unsigned integer A whole number that is positive or zero (`0`, `143`). Also known as a “uinteger.” InfluxDB supports 64-bit unsigned integers (minimum: `0`, maximum: `18446744073709551615`). Related entries:[integer](#integer) ### upstream (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the source node that sends replicated data to a[downstream](#downstream-edr) node. The upstream initiates all data transfer. A node can be upstream-only (a pure source), or both upstream and downstream at once (a regional relay). Related entries:[downstream (EDR)](#downstream-edr),[sidecar agent](#sidecar-agent) ### user InfluxDB users are granted permission to access InfluxDB. V ---------- ### values per second The preferred measurement of the rate at which data is persisted to InfluxDB. Write speeds are generally quoted in values per second. To calculate the values per second rate, multiply the number of points written per second by the number of values stored per point. For example, if the points have four fields each, and a batch of 5000 points is written 10 times per second, the values per second rate is: **4 field values per point** × **5000 points per batch** × **10 batches per second** = **200,000 values per second** Related entries:[batch](#batch),[field](#field),[point](#point) ### variable A storage location (identified by a memory address) paired with an associated symbolic name (an identifier). A variable contains some known or unknown quantity of information referred to as a value. ### variable assignment A statement that sets or updates the value stored in a variable. W ---------- ### WAL (Write-Ahead Log) The temporary cache for recently written points. To reduce the frequency that permanent storage files are accessed, InfluxDB caches new points in the WAL until their total size or age triggers a flush to more permanent storage. This allows for efficient batching of the writes into the storage engine. Points in the WAL are queryable and persist through a system reboot. On process start, all points in the WAL must be flushed before the system accepts new writes. ### WAL replication (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the live replication path that discovers new [WAL](#wal-write-ahead-log)files as they appear on the source’s object store and ships them downstream. WAL replication is EDR’s primary, most precise data source;[historic fill](#historic-fill-edr) and [gap fill](#gap-fill-edr) recover data that outlives WAL retention from compacted files instead. Related entries:[historic fill (EDR)](#historic-fill-edr),[gap fill (EDR)](#gap-fill-edr) ### WAL tail The most recent points in the [WAL](#wal-write-ahead-log) that have not yet been durably persisted beyond the WAL. For details, see[Data durability](/influxdb3/enterprise/reference/internals/durability/#wal-tail). ### windowing Grouping data based on specified time intervals. This is also referred to as “time binning” or “date binning.” [glossary](/influxdb3/enterprise/tags/glossary/) | email | status | | --- | --- | | email | status | | lorr@influxdata.com | start | | lorr@influxdata.com | finish | | marv@influxdata.com | start | | marv@influxdata.com | finish | | cliff@influxdata.com | start | | cliff@influxdata.com | finish | | email | status | firstname | | --- | --- | --- | | email | status | firstname | | lorr@influxdata.com | start | lorraine | | lorr@influxdata.com | finish | lorraine | | marv@influxdata.com | start | marvin | | marv@influxdata.com | finish | marvin | | cliff@influxdata.com | start | clifford | | cliff@influxdata.com | finish | clifford | | Precision | Description | Example | | --- | --- | --- | | Precision | Description | Example | | ns | Nanoseconds | 1577836800000000000 | | us | Microseconds | 1577836800000000 | | ms | Milliseconds | 1577836800000 | | s | Seconds | 1577836800 | --- # EDR limitations Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/limitations/ * EDR 1.0.0 ## What EDR is not EDR is not a synchronous replication layer, a conflict resolution system, a backup tool, or a transformation layer. The destination is eventually consistent. Data arrives byte-for-byte as written at the source, scoped to the configured databases and tables. ## Requirements and availability EDR is specified exclusively for the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)and its associated data formats. The Parquet-based storage engine is not supported. EDR is a commercial feature of InfluxDB 3 Enterprise. It is distinct from the open source Synchronizer plugin and is not available with InfluxDB 3 Core as a source. In deployments where Core might otherwise be chosen for its simplicity, InfluxDB 3 Enterprise can be deployed as a single node to serve as an EDR source. ## Known implementation limitations Current limitations of the implementation: 1. **No retention hold enforcement.** The agent requests holds on[Gen0 and cv2](/influxdb3/enterprise/edr/reference/architecture/#storage-engine-terms)files during recovery but the InfluxDB compactor does not enforce them. Correctness does not depend on holds (recovery falls through WAL to cv2 with per-file tracking), but a hold would narrow boundary overlap, reduce over-replication, and close the residual window where a file deleted mid-recovery defers to a wider cv2 pass. This is an outstanding ask of the InfluxDB team. 2. **Receiver-side scope enforcement is not implemented.** The downstream does not validate incoming data against the upstream’s declared scope. In peer-to-peer topologies, sender-side scope filtering is currently the only guard against circular amplification. If you’re evaluating EDR for a multi-tenant or systems-integrator deployment, tenant isolation relies on sender-side config discipline, not a receiver-enforced guarantee. 3. **Silent windows are not propagated to the downstream.** A fully silent schedule window (`silent_reports: false`) causes the downstream to mark the channel unhealthy even though the silence is intentional. 4. **PT-wire partial application.** A multi-database PT batch that fails partway might have applied earlier databases; the failure is reported to the sender without classification (schema vs auth vs write). Idempotent re-delivery converges the state, but error reporting should improve. 5. **Topology cycle rendering.** The UI tree renderer does not detect cycles; peer-to-peer topologies need `share_topology: false` on both peers. 6. **Historic fill over-replication residue.** Sending whole cv2 files over-delivers for windows narrower than the file; block-level time filters help only time-partitioned workloads. Row-level filtering at encode time is future work. 7. **Series disjointness is unenforced.** When multiple upstreams write to the same downstream table, each series (database, table, and tag set) should be owned by exactly one upstream, so per-point write ordering stays a purely local concern within one upstream’s stream. This is a documented operator responsibility, not enforced. 8. **Open downstream registration, upstream and downstream scale-out, and Cloud-as-upstream** are design-stage items, not yet implemented. ## Sizing and capacity Size WAL retention to your expected outage window—see[Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/)—and test your specific workload’s cardinality and write volume before committing to constrained hardware. ## Fleet upgrade orchestration EDR’s protocol negotiation supports mixed-version fleets without configuration (see [Compatibility](/influxdb3/enterprise/edr/reference/compatibility/)), and the recommended order is to upgrade the reader (EDR) before the source server. Orchestrating an EDR upgrade across many unattended edge nodes—for example, with configuration management or fleet-rollout tooling—is outside the scope of this documentation. --- # EDR state and recovery Source: https://docs.influxdata.com/influxdb3/enterprise/edr/reference/state-and-recovery/ * EDR 1.0.0 EDR remembers its replication progress between restarts. This page covers where it keeps that state, what each piece means, and what happens if you delete it. 1. [Where the state lives](#where-the-state-lives) 2. [What each file is](#what-each-file-is) 3. [Read before you touch](#read-before-you-touch) 4. [The golden rules](#the-golden-rules) 5. [Corruption vs. deliberate deletion](#corruption-vs-deliberate-deletion) 6. [The deletion matrix](#the-deletion-matrix) 7. [Sanctioned recovery recipes](#sanctioned-recovery-recipes) 8. [Risk summary](#risk-summary) ## Where the state lives Everything the agent remembers is under `--state-location` (env`INFLUXDB3_EDR_STATE_LOCATION`, default `edr-state`). A bare path or`file://` URL selects the local filesystem; `s3://bucket/prefix`, `gs://...`,`az://...` select object storage (credentials from the same `AWS_*`,`GOOGLE_*`, or `AZURE_*` environment the agent uses for replicated data). Each item is one JSON object named `{key}.json`. The state location is**independent of the object store that holds your replicated data**—moving or resetting it never touches your data; it only changes what the agent believes it has already done. ### Per-destination namespaces Journals are namespaced **per downstream destination**:`{state-location}/{destination}/wal_cursor.json` (and siblings). Each destination’s cursor, gap ledger, and historic progress are fully independent—a fan-out node has one namespace per destination. Two items live at the root: `state_layout.json` (the layout format marker) and`wal_cleanup_watermark.json` (agent-level—the cleanup watermark is shared by all destinations). The layout is self-maintaining, and **config is the source of truth** (a destination’s namespace exists if the destination is configured): * **Migration**: on the first start of a fan-out-capable build, journals from the old un-namespaced layout are moved into the (single) configured destination’s namespace, one log line per file. With multiple destinations configured and un-namespaced journals present, the agent refuses to start—it cannot guess whose history they are. * **Orphan sweep**: at startup, a namespace with no matching configured destination (removed while the agent was down) is **deleted, loudly**. * **Rename guard**: if the same pass sees an orphan namespace AND a configured destination with no state—the signature of a typo’d rename—the agent refuses to start rather than delete a real cursor. The error names both and the fix (correct the name, or delete the orphan namespace deliberately). The same guard rejects a live reload that removes one destination and adds a stateless new one in one pass. * **Removing a destination through reload deletes its namespace**immediately; re-adding the same name later starts fresh. The token store (`--token-store`) is separate and is not covered here—it holds secrets, not progress. ## What each file is * **`wal_cursor.json`** (live replication)—per-ingest-node high-water mark (`last_replicated_wal_id`), out-of-order completions, and**skip receipts**: durable records that a WAL range was handed to gap fill. The dispatcher writes it on every file completion, and the replicator on seeding and receipts. `wal_cursor_prev.json` is its safety mirror. * **`gap_ledger.json`** (gap fill)—recovery obligations: each detected gap with its status (pending, in-progress, resolved, or unrecoverable) and the files that covered it. The gap-fill worker writes it; `gap_ledger_prev.json` is its safety mirror. * **`historic_manifest.json`** (historic fill)—the backfill plan: snapshot work list, cv2 work list, per-file statuses, and the completed flag. The historic planner writes it (full saves);`historic_manifest_prev.json` is its safety mirror. * **`historic_progress.json`** (historic fill)—compact status checkpoints overlaid on the plan, guarded by a sequence number so a stale overlay can never resurrect finished work. The historic fill task writes it frequently. * **`live_seed_done.json`** (live replication)—marker: first-start seeding already happened—a restart resumes instead of re-seeding. The replicator writes it once. * **`wal_cleanup_watermark.json`** (WAL cleanup)—per-node “deleted through id N” resume points (only with `--wal-cleanup-enabled`).**Lives at the state ROOT**—agent-level, shared by all destinations. The cleanup sweep writes it. All files except `wal_cleanup_watermark.json` and `state_layout.json` live inside a destination’s namespace; a recovery recipe that says “delete the cursor” means **that destination’s** cursor—the other destinations’ state is untouched. ## Read before you touch `edr-inspect` ships in the EDR image and renders all three—live replication, gap fill, and historic fill—read-only: ```bash edr-inspect state /var/lib/edr/state ``` In Docker, run `docker exec edr-inspect state` with no path. That form picks up the agent’s own `INFLUXDB3_EDR_STATE_LOCATION`. Start here when you diagnose a problem.`edr-inspect state` answers most questions—“is it stuck?”, “what does it think it owes?"—without touching anything. ## The golden rules 1. **Never modify state while the agent is running.** The in-memory copy is authoritative; the next persist silently overwrites your edit. (Deleting files while running is harmless but pointless—see the matrix.) 2. **Never hand-edit the JSON at all.** The schemas evolve between releases and the fields interlock (a cursor that disagrees with its own skip receipts is worse than either alone). Recovery is done by *deleting*specific files with the agent stopped—never by editing. 3. **Resets are only safe when the destination absorbs re-sends**—that is,`idempotent_writes: true`. With `idempotent_writes: false`, any reset that causes re-delivery corrupts the destination (duplicate points at new timestamps). Do not reset state in that mode without also clearing the affected destination range. 4. **Your `historic_fill` mode decides what a full reset means.** With`mode: full` the agent re-plans and re-sends all history (safe, expensive). With `mode: none` it seeds at the current leading edge—**any backlog the old cursor still owed is silently skipped**. That’s genuine data loss with no gap recorded, because the record of the debt was the thing you deleted. ## Corruption vs. deliberate deletion The cursor, gap ledger, and historic manifest each write a **previous-good mirror** alongside the primary (`{key}_prev.json`) on every successful save. On load, a corrupt or unreadable primary silently falls back to the mirror—logged as a WARN, no operator action needed, no restart required. This is different from the deletion matrix below, which is about what happens when *both* copies are gone (deliberately, or because corruption hit both). When **both** the primary and mirror are corrupt or unreadable—“state loss”—`on_state_loss` (a downstream config knob) governs the agent’s behavior: * **`recover`** (the default when `idempotent_writes: true`): rebuild a conservative per-node cursor position from the oldest surviving snapshot and continue—re-replicates the retained WAL window (bounded, absorbed by idempotent writes), rather than defaulting to position 0 and silently re-sending the *entire* retained backlog. The historic manifest, on its own state loss, rebuilds the plan from current store state the same way it always has (loud, double-send absorbed)—it never needed the mirror to have a safe fallback. * **`halt`** (the default when `idempotent_writes: false`, since re-sending isn’t safe there): refuse to start; an operator decides. Restore the state location from backup, or switch to `recover` (requires`idempotent_writes: true`). `edr-inspect state` reports which tier each journal was actually read from—healthy primary, recovered-from-mirror, or state loss—so this is visible without touching anything. ## The deletion matrix What happens when a state item is removed, by agent state: * **Entire state location** * *Agent running*: self-heals piecemeal (files reappear as each subsystem writes its state again), but you have destroyed receipts mid-flight—don’t do this; stop first. * *Agent stopped, then restarted*: the sanctioned **full reset**. Golden rules 3 and 4 govern the behavior entirely: a `full`fill with idempotent writes gives a safe re-fill; `none` means owed backlog is silently skipped. * **`wal_cursor.json`** only (`wal_cursor_prev.json` intact) * *Agent running*: harmless: in-memory state is authoritative, delivery continues, the journal re-persists on the next file completion. * *Agent stopped, then restarted*: recovers transparently **from the mirror** (WARN logged)—resumes at the mirror’s position, not a fresh start, no re-transmission storm. * **`wal_cursor.json`** and its mirror * *Agent running*: harmless (same as above—in-memory state is authoritative). * *Agent stopped, then restarted*: **state loss**—see`on_state_loss` above. `recover`: rebuilds a conservative per-node floor and re-sends the retained window (bounded).`halt`: refuses to start. * **`gap_ledger.json`** * *Agent running*: recreated on the next ledger save. * *Agent stopped, then restarted*: the most forgiving: outstanding obligations are **rebuilt from the cursor’s skip receipts** at startup (works even with the mirror also gone, as long as the cursor survives). You lose the resolved and unrecoverable*history* (audit trail), not the obligations. * **`historic_manifest.json`** only (`historic_manifest_prev.json`intact, fill incomplete) * *Agent running*: recreated only at the next *full plan save*— routine checkpoints write the overlay, not the plan. * *Agent stopped, then restarted*: recovers transparently **from the mirror** (WARN logged)—resumes the existing plan, no rebuild, no double-send. * **`historic_manifest.json`** and its mirror (historic\_fill configured, fill incomplete) * *Agent running*: same as above. * *Agent stopped, then restarted*: with a **non-zero cursor**: the agent **refuses to start** (`CursorExists`—historic\_fill added late vs. manifest vanished are indistinguishable, and a same-session cursor rebuild doesn’t resolve that ambiguity either). Restore a file, do a full reset, or remove`historic_fill`. With a genuinely **fresh cursor**: normal fresh start, plan rebuilt. * **`historic_progress.json`** * *Agent running*: recreated at the next checkpoint. * *Agent stopped, then restarted*: statuses revert to the last full plan save—some already-sent files are re-sent (absorbed by idempotence). The sequence guard prevents any stale overlay from marking unfinished work done. * **`live_seed_done.json`** * *Agent running*: no effect until restart. * *Agent stopped, then restarted*: with a cursor present: harmless (a non-zero cursor is itself proof of a previous start). With the cursor *also* gone: a genuine fresh start—see full reset. * **`wal_cleanup_watermark.json`** * *Agent running*: next sweep reseeds from the oldest surviving snapshot. * *Agent stopped, then restarted*: same—the sweep re-derives its position; at worst it re-issues deletes for already-deleted files (harmless NotFounds). ## Sanctioned recovery recipes **Full reset (start replication over)** ```bash # 1. Stop the agent. # 2. Confirm your posture: idempotent_writes true? historic_fill mode? # (Golden rules 3 and 4 -- this decides whether the reset is safe.) # 3. Delete the state location contents. rm -rf /var/lib/edr/state/* # 4. Start the agent. With historic_fill: full it re-plans all history. ``` **Force a re-backfill without touching live progress** ```bash # Stop the agent, then remove the historic plan, ITS MIRROR, and the overlay # -- leaving historic_manifest_prev.json in place would just recover the old # plan from the mirror instead of triggering a rebuild: rm /var/lib/edr/state/historic_manifest.json \ /var/lib/edr/state/historic_manifest_prev.json \ /var/lib/edr/state/historic_progress.json # ALSO delete the cursor (and its mirror) if you want history re-planned # from scratch -- without it the agent refuses to start (see the matrix # row). In most cases you actually want the full reset above. ``` **Clear a poisoned gap ledger** (for example, an Unrecoverable entry you have resolved out of band) ```bash # Stop the agent, delete the ledger and its mirror: rm /var/lib/edr/state/gap_ledger.json /var/lib/edr/state/gap_ledger_prev.json # On restart, genuinely outstanding gaps are re-created from the # cursor's skip receipts and re-attempted; settled history is gone. ``` **Move the state location** ```bash # Stop the agent, copy the whole directory (it is just files), point # --state-location / INFLUXDB3_EDR_STATE_LOCATION at the new place, # start. Copying while stopped is also the correct backup procedure. ``` ## Risk summary | Action | Risk | |------------------------------------------------------------------------------|---------------------------------------------------------------------------| | Full reset, `historic_fill: full`, idempotent | Re-transmission cost only | | Full reset, `historic_fill: none` | **Silent loss of owed backlog** | | Any reset with `idempotent_writes: false` | **Destination corruption** | | Delete cursor primary only (mirror intact) | None—recovers from mirror | | Delete cursor primary AND mirror (stopped) |`recover`: bounded re-send from the snapshot floor. `halt`: startup refusal| | Delete manifest primary only (mirror intact, fill incomplete) | None—recovers from mirror | |Delete manifest primary AND mirror (stopped, fill incomplete, non-zero cursor)| Startup refusal until resolved (`CursorExists`) | | Delete gap ledger alone | Lost audit history; obligations survive | | Hand-editing any file | Undefined; never sanctioned | | Anything while the agent runs | Ineffective (overwritten)—stop first | | Action | Risk | | --- | --- | | Action | Risk | | Full reset, historic_fill: full , idempotent | Re-transmission cost only | | Full reset, historic_fill: none | Silent loss of owed backlog | | Any reset with idempotent_writes: false | Destination corruption | | Delete cursor primary only (mirror intact) | None—recovers from mirror | | Delete cursor primary AND mirror (stopped) | recover : bounded re-send from the snapshot floor. halt : startup refusal | | Delete manifest primary only (mirror intact, fill incomplete) | None—recovers from mirror | | Delete manifest primary AND mirror (stopped, fill incomplete, non-zero cursor) | Startup refusal until resolved ( CursorExists ) | | Delete gap ledger alone | Lost audit history; obligations survive | | Hand-editing any file | Undefined; never sanctioned | | Anything while the agent runs | Ineffective (overwritten)—stop first | --- # EDR release notes Source: https://docs.influxdata.com/influxdb3/enterprise/edr/release-notes/ * EDR 1.0.0 EDR follows [SemVer](https://semver.org). During the beta, the version stays below `1.0.0`; within that range, EDR bumps the minor number for each beta drop and the patch number for a re-spin of the same drop. `1.0.0` marks general availability (release candidates for it are numbered`1.0.0-0.rc.N`). Every release is immutable once published—a given version always refers to exactly the same bytes—and the agent identifies its build precisely: `influxdb3-edr --version` prints the version plus the source revision it was built from (for example, `0.2.0 (682ce264f)`), the same identity appears in the agent’s startup log, and each agent reports it to its downstream, so the destination’s API and UI show which build every connected edge is running. Each release is built against a pinned InfluxDB 3 Enterprise revision, encoded in the image tag (`edr:-pro-`). ## v1.0.0 General availability of EDR for InfluxDB 3 Enterprise. Built against Enterprise 3.12.0, revision `7dd361197e`. The image tags are `edr:1.0.0-pro7dd361197e-amd64` and `edr:1.0.0-pro7dd361197e-arm64`. ### Compatibility and upgrade EDR 1.0.0 uses EDR replication protocol version 3 and accepts version 1 peers. Supported source server lines are **3.11.x and 3.12.x**. InfluxDB 3 Enterprise 3.10.x is unvalidated and unsupported for GA, even though the binary’s displayed list includes it. For rc.1, rc.2, or rc.3 on a 3.11 source, stop the agent and start GA with the same configuration, state location, and source data location. No configuration changes are required. For 0.2.0 on a 3.10 source, upgrade the agent to rc.3, the server to 3.11, the agent to GA, and then the server to 3.12 if needed. Rc.3 cannot read a 3.12 source. See the [compatibility matrix and upgrade procedure](/influxdb3/enterprise/edr/reference/compatibility/#upgrade-to-edr-100). ### Capabilities * [Kubernetes and Helm installation](/influxdb3/enterprise/edr/install/kubernetes/) supports sender, receiver, and combined roles alongside an unmodified Enterprise Helm release. Agent state persists in object storage; tokens mount from Kubernetes Secrets. * S3, Google Cloud Storage, and Azure Blob Storage are supported for source reads and agent state. State locations accept `s3://`, `gs://`, and `az://` URLs. * The token store accepts server token JSON files with a `token` field. Set its location with `--token-store` or `INFLUXDB3_EDR_TOKEN_STORE`. The agent warns once about group or world token-file permissions. See [Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). * High server WAL retention plus opt-in agent WAL cleanup is the supported production configuration. Server-negotiated retention holds are not included in GA. See [Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/). ### Fixes * Failed-WAL skip receipts hold the cleanup floor until the gap ledger records the failure. Cleanup cannot delete the file during that transition. * `edr_owed_work` no longer briefly reports zero while gaps move from pending to in progress. * The pending-gap counter no longer underflows. * Wire batches containing more than 65,535 schemas fail explicitly instead of truncating a schema index. For release artifacts and deployment coordination, [contact the InfluxData Sales team](https://www.influxdata.com/contact-sales/). ## v1.0.0-0.rc.3 Third release candidate, built against Enterprise 3.11.0, revision `e5242f505d`. Supports 3.10.x and 3.11.x sources, EDR replication protocol version 3, and version 1 peers. It does not support 3.12.x sources. Upgrades from rc.2, rc.1, or 0.2.0 preserve configuration, state, and source data locations. > [!Warning] > #### Verify wide-table field values after upgrading > > All earlier builds, including 0.2.0, rc.1, and rc.2, can pair field values with the wrong field names in tables wider than 100 fields. > Different field types can cause a schema conflict and halt replication. > Matching types can silently store mislabeled values. > After upgrading, verify field-level values downstream or re-replicate affected tables. > Row counts alone cannot detect this defect. ### Fixes and changes * Recovery records an entire evicted WAL span as one gap and resumes live replication above the retention floor. Earlier builds could remain behind retention indefinitely without sending data. * Direct-mode liveness treats HTTP 404 or 405 as proof of life for InfluxDB 3 Cloud, which has no `/health` route. Authentication failures and server errors still fail the probe. * Checkpoint versions 12 and 13 are correctly labeled as 3.11-line state stamps. Version 13 includes bulk-import watermarks; these stamps do not establish 3.12 source support. * `gapfill_blocked_on_halt` and a log message identify recovery blocked by a halted channel. * The `historic_fill: { mode: none }` startup notice clarifies that all in-scope tables replicate from the startup boundary; only older history is skipped. * Historic-fill width follows `concurrent_sends`. Gen0 processing no longer drains the pipeline at each snapshot boundary. Strict-order `concurrent_sends: 1` processes one file at a time. * Signals demo host ports move to 18181–18184 and remain overridable in `.env`. ## v1.0.0-0.rc.2 Second release candidate, built against Enterprise 3.11.0, revision `e5242f505d`. Supports 3.10.x and 3.11.x sources, EDR replication protocol version 3, and version 1 peers. Upgrades from rc.1 or 0.2.0 require no configuration changes. Keep the same state and source data locations; 0.2.0 state migrates automatically into per-destination namespaces. ### Fixes and changes * Gap recovery replays exact raw WAL files when available. Transient read failures wait briefly and retry; evicted files fall back to compacted-file recovery.`gapfill_files_from_wal_total` reports raw-WAL recovery. * Active recovery reports `gaps_in_progress`, discovery progress, and replication progress.`gapfill_files_vanished_total` counts files removed by compaction during recovery. Alert on stalled recovery rather than pending gaps alone. * `rows_sent` now counts delivered physical rows. Wide tables split logical rows across field families, so this can exceed SQL row counts. * Dashboard health panels use stable numeric series and reflect channel changes within a scrape interval. The Recovery row shows backlog, progress, and recovery medium. * Direct-mode liveness authenticates like the write path, allowing auth-gated `/health` endpoints. * `edr_owed_work` reports outstanding replication work per destination for drain checks. * High server WAL retention plus opt-in agent cleanup becomes the supported production pattern. Server-negotiated retention holds are deferred beyond GA. ## v1.0.0-0.rc.1 Release candidate for the 1.0.0 general-availability release. Built against InfluxDB 3 Enterprise **3.11.0** (revision `e5242f505d`). Images:`edr:1.0.0-0.rc.1-proe5242f505d-amd64` and `-arm64`. **Compatibility** (see[Compatibility](/influxdb3/enterprise/edr/reference/compatibility/)): EDR replication protocol **version 3** (stateless data path; version-1 peers accepted—a mixed fleet with 0.2.0 agents interoperates in both directions); InfluxDB storage matrix **3.10.x and 3.11.x**—the agent runs unchanged across a source server’s 3.10 to 3.11 upgrade (upgrade EDR first). **Upgrading from 0.2.0**: drop-in. On first start the agent migrates its on-disk state (v1 to v2, adding journal mirrors) automatically; no configuration change is required. One behavioral change to review:**bandwidth schedules are now interpreted in UTC** by default (previously the agent process’s local timezone). If your schedule windows were written for a local clock, set the new `bandwidth_timezone` field (IANA name, for example `Asia/Kolkata`) to keep them meaning what they meant. **Highlights over 0.2.0:** * **Multi-destination fan-out.** A sender can replicate to several downstreams at once—for example, a live feed plus a full-history archive—with per-destination scope, schedule, journal, and the health and UI channel. See[Replicate to multiple destinations](/influxdb3/enterprise/edr/replicate/to-multiple-destinations/). * **Scope exclusions.** `exclude: { databases: [...], tables: [...] }`composes with instance- and database-level scope, so “everything except`internal_edr`” is now a two-line config. * **State-loss resilience.** Journal files (cursor, gap ledger, historic manifest) are mirrored; a corrupt or vanished primary recovers from the mirror automatically. Behavior when both copies are lost is explicit config: `on_state_loss: recover` (default, requires idempotent writes) or`halt`. * **WAL loss is accounted, not buried.** WAL evicted by the server before replication is classified distinctly (`WalFileMissing`), counted in a new`wal_files_lost_total` metric, and recovered from gen0/compacted files through gap fill where possible. * **Cleanup correctness.** Agent-side WAL cleanup now holds a fourth floor under queued and in-flight work and unresolved gap-ledger ranges, so a slow or recovering destination can never have its input deleted out from under it. * **Bandwidth schedule hardening.** Stricter validation (unknown fields, malformed times and dates, empty day lists are config errors), UTC-by-default semantics with the `bandwidth_timezone` opt-in above. * **Refreshed embedded UI**, matching InfluxDB 3 Explorer’s branding. * **`edr-inspect`**—a read-only triage CLI (`state`, `metrics`, and`topology`), shipped in the image and run through `docker exec`. * **Optional observability overlay** in the signals demo: Telegraf and Grafana dashboards layered on with a second compose file, like licensing. **Known limitation**: the source server’s WAL retention (`wal-snapshots-to-keep`, un-prefixed spelling on 3.11) is the only hold on WAL files EDR has not yet replicated; size it generously on constrained or intermittently connected edges. Server-negotiated retention holds are not included in GA; use[high server retention and opt-in agent cleanup](/influxdb3/enterprise/edr/size-wal-retention/). ## v0.2.0 First customer beta release. Built against InfluxDB 3 Enterprise 3.10.0 (revision `0119b6f42f`). Images: `edr:0.2.0-pro0119b6f42f-amd64` and`-arm64`. **Compatibility** (see[Compatibility](/influxdb3/enterprise/edr/reference/compatibility/)): EDR replication protocol **version 1** (implicit—predates protocol versioning); InfluxDB storage matrix **3.10.x only**. **Highlights over the preview builds (0.1.0):** * **Delivery-ordering guarantees made explicit.** Strict-order delivery is now the default (`concurrent_sends: 1`), and every feature that reorders or re-sends data—concurrent sending, priority routing, historic fill, multi-ingest—requires the `idempotent_writes: true` assertion and is rejected by config validation otherwise. See[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/#performance-vs-correctness). * **Agent-side WAL cleanup** (`--wal-cleanup-enabled`, opt-in): bounded deletion of already-replicated WAL from the source object store, so WAL no longer accumulates when server-side retention is set high. Deletion is bounded below the replication cursor, the snapshot boundary (minus a configurable margin), and the historic-fill floor—validated under chaos testing with zero loss and zero replication gaps. * **Robustness hardening** from a full-codebase review: line-protocol encoding of names with special characters, schema-change detection during replication, wire-format hardening on the receiver, and crash-consistency fixes in historic fill and the replication cursor. * **Build identity**: `--version` on both `influxdb3-edr` and`edr-inspect`, version in the startup log, and per-edge version visibility at the destination. ## v0.1.0 Internal preview builds circulated ahead of the first customer release (preview). Superseded by v0.2.0; the v0.1.0 artifact line is retired. #### Related * [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/) --- # Replicate with EDR Source: https://docs.influxdata.com/influxdb3/enterprise/edr/replicate/ * EDR 1.0.0 Use the following guides to configure EDR replication for your destination. | Your destination | Guide | |--------------------------------------|---------------------------------------------------------------------------------------------------| |Another InfluxDB 3 Enterprise instance| [Replicate to InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/replicate/to-enterprise/) | | InfluxDB 3 Cloud | [Replicate to InfluxDB 3 Cloud](/influxdb3/enterprise/edr/replicate/to-cloud/) | | Multiple destinations at once |[Replicate to multiple destinations](/influxdb3/enterprise/edr/replicate/to-multiple-destinations/)| For the full configuration schema, see[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/). ### [Replicate to InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/replicate/to-enterprise/) Set up EDR to replicate data from one InfluxDB 3 Enterprise instance to another, in edge-to-regional-to-central topologies. ### [Replicate to InfluxDB 3 Cloud](/influxdb3/enterprise/edr/replicate/to-cloud/) Set up EDR to replicate data from InfluxDB 3 Enterprise to InfluxDB 3 Cloud. ### [Replicate to multiple destinations](/influxdb3/enterprise/edr/replicate/to-multiple-destinations/) Configure one EDR source to replicate to several destinations at once, each with independent scope, schedule, and health. | Your destination | Guide | | --- | --- | | Your destination | Guide | | Another InfluxDB 3 Enterprise instance | Replicate to InfluxDB 3 Enterprise | | InfluxDB 3 Cloud | Replicate to InfluxDB 3 Cloud | | Multiple destinations at once | Replicate to multiple destinations | --- # Replicate to InfluxDB 3 Cloud Source: https://docs.influxdata.com/influxdb3/enterprise/edr/replicate/to-cloud/ * EDR 1.0.0 Use EDR to replicate data from InfluxDB 3 Enterprise to[InfluxDB 3 Cloud](/influxdb3/cloud/). This guide covers only what’s different when Cloud is the destination—auth and endpoint. For shared configuration (token store, `historic_fill`, scope, bandwidth scheduling), see [Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/). ## Direct mode: write straight to InfluxDB 3 Cloud When the destination does not run an EDR agent, EDR replicates directly to a plain InfluxDB v3 write endpoint: ```yaml downstream: name: "cloud-influx" address: "https://cloud.influxdata.com" auth_token: "cloud-api-token" mode: direct ``` In direct mode: * There is no `/connect` handshake. * Health checks use the destination’s `GET /health`. * No topology is propagated to the destination. * Auth failures and schema conflicts are handled identically to agent-to-agent mode. * The `pt` wire encoding is not available—direct mode always uses `lp`(Line Protocol). Replace `cloud-api-token` with a token-store entry containing an InfluxDB 3 Cloud API token with write access to the target database. See[Manage tokens](/influxdb3/enterprise/edr/manage-tokens/). ## Compatibility EDR’s agent-to-agent compatibility matrix (protocol version negotiation, storage-format preflight) applies to hops between two `influxdb3-edr`agents. A direct-mode connection to InfluxDB 3 Cloud doesn’t negotiate an agent protocol version—the InfluxDB 3 write API itself governs compatibility. See[Compatibility](/influxdb3/enterprise/edr/reference/compatibility/) for the full agent-to-agent and agent-to-InfluxDB compatibility matrices. ## Next * [Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/) on the source Enterprise instance to cover your expected outage windows. * [Monitor EDR](/influxdb3/enterprise/edr/monitor/) to watch replication health and lag to Cloud. --- # Replicate to InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/edr/replicate/to-enterprise/ * EDR 1.0.0 Use EDR to replicate data between InfluxDB 3 Enterprise instances—edge to regional relay to central hub, or a direct edge-to-central hop. This guide covers the Enterprise-to-Enterprise topology end to end. For general config and CLI flag reference, see[Configuration file reference](/influxdb3/enterprise/edr/reference/config-file/). ## Prerequisites * InfluxDB 3 Enterprise on the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)(the default for new clusters on 3.11+; on 3.10.x pass`--upgrade-pacha-tree`) on every node. * The `influxdb3-edr` binary installed on every node that sources or receives data. See [Install EDR](/influxdb3/enterprise/edr/install/). * A token store directory with auth and write tokens. * Sufficient WAL retention on each source node. See[Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/). ## Topology shapes Every EDR deployment, however large, is composed of the same three node shapes: | Role | What its config has | |------------------------------------------|-----------------------------------------| | **Upstream-only (edge)**—a pure source |a `downstream:` (or `downstreams:`) block| |**Regional relay**—receives *and* forwards| both `upstreams:` and `downstream:` | | **Downstream-only (hub)**—a pure sink | an `upstreams:` list | A node with both `downstream` (or `downstreams`) and `upstreams` is a regional hub: it receives from edges and forwards on. There’s no architectural depth limit; practical deployments are 2-3 levels. ## 1. Create a token store ```bash mkdir -p /etc/edr/secrets # Auth token—identifies this source to the destination. openssl rand -hex 32 > /etc/edr/secrets/my-auth-token # Write token—used by the destination's EDR agent to write into its # local InfluxDB. Typically the destination's InfluxDB admin/operator token. cp /path/to/destination-write-token /etc/edr/secrets/dest-write-token ``` ## 2. Configure the source node ```yaml # /etc/edr/config.yaml -- source node name: "factory-floor-01" downstream: name: "regional-hub" address: "http://hub.internal:9090" auth_token: "my-auth-token" # name of file in token store historic_fill: mode: none # required -- none | full | since comms: interval_secs: 10 degraded_after: 3 unhealthy_after: 6 retry: initial_backoff_secs: 1 max_backoff_secs: 30 multiplier: 2 ``` ## 3. Configure the destination node ```yaml # /etc/edr/config.yaml -- destination node name: "regional-hub" upstreams: - name: "factory-floor-01" auth_token: "factory-01-auth" # verifies the source's identity write_token: "dest-write-token" # used to write into local InfluxDB - name: "factory-floor-02" auth_token: "factory-02-auth" write_token: "dest-write-token" ``` To build a regional relay (receives from edges and forwards to a central hub), give one node both an `upstreams:` list and a `downstream:` block. ## 4. Start the agents ```bash # Source node: influxdb3-edr \ --config /etc/edr/config.yaml \ --data-dir /var/lib/influxdb3 \ --token-store /etc/edr/secrets \ --listen 0.0.0.0:9090 \ --state-location /var/lib/edr/state \ --poll-interval-ms 1000 # Destination node: EDR_WRITE_ENDPOINT="http://localhost:8181" \ influxdb3-edr \ --config /etc/edr/config.yaml \ --data-dir /var/lib/influxdb3 \ --token-store /etc/edr/secrets \ --listen 0.0.0.0:9090 \ --state-location /var/lib/edr/state ``` ## 5. Verify Open `http://127.0.0.1:9091/ui`—the embedded UI shows topology, health, replication lag, and data-flow metrics. The UI and `/metrics` bind to loopback by default (`--observability-listen`); expose them deliberately if you need off-host access. See [Monitor EDR](/influxdb3/enterprise/edr/monitor/). ## Next * [Replicate to multiple destinations](/influxdb3/enterprise/edr/replicate/to-multiple-destinations/)if this source also needs to feed a central archive. * [Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/) for your expected outage windows. * [Troubleshoot EDR](/influxdb3/enterprise/edr/troubleshoot/) if replication doesn’t start flowing. | Role | What its config has | | --- | --- | | Role | What its config has | | Upstream-only (edge) —a pure source | a downstream: (or downstreams: ) block | | Regional relay —receives and forwards | both upstreams: and downstream: | | Downstream-only (hub) —a pure sink | an upstreams: list | --- # Replicate to multiple destinations Source: https://docs.influxdata.com/influxdb3/enterprise/edr/replicate/to-multiple-destinations/ * EDR 1.0.0 One agent can replicate to several destinations at once—an edge feeding its regional hub and a central archive, or dual centers for redundancy.`downstreams:` is a list of the exact same destination shape as `downstream:`(singular); `downstream:` remains valid indefinitely and is exactly a one-entry list. The following generic example fans one source out to a regional relay and a separate long-term archive: ```yaml name: edge-01 downstreams: - name: regional-relay # identity -- also the journal namespace key address: http://relay:9090 auth_token: relay-token historic_fill: { mode: none } - name: archive address: http://archive:9090 auth_token: archive-token idempotent_writes: true historic_fill: { mode: full } # per-destination: the archive takes # full history; the relay is live-only ``` Scope (selective fan-out), encoding, ordering guarantees, retry and comms settings, schedules, priorities, historic intent, and protocol negotiation are all independent per destination. There are no shared destination defaults—each entry is explicit. One shared WAL discovery feeds every destination, so discovery cost does not grow with destination count. ## Validation rules * Destination `name`s must be unique—they key the journals, metrics, and UI (and their sanitized forms must not collide). * Two destinations must not resolve to the same endpoint (normalized host:port), irrespective of `auth_token` or `name`—one endpoint fed by two pipelines is a double-send. * `historic_fill` remains required per destination. ## State Each destination’s journals live in their own namespace,`{state-location}/{destination}/...`—fully independent cursors and recovery. See [State and recovery](/influxdb3/enterprise/edr/reference/state-and-recovery/). ## Live add and remove (config reload, no restart) * **Adding** a destination spawns its pipeline immediately: it seeds at the leading edge or runs its declared historic fill, while existing destinations keep flowing undisturbed. * **Removing** a destination stops its pipeline and deletes its journal namespace—config is the source of truth for state. Re-adding the same name later starts fresh (with idempotent writes the cost is over-replication, never corruption). * **Renaming is a remove and an add** (the name is the journal identity). A single reload that removes one name and adds a brand-new one is rejected as a probable typo—apply the remove and the add as two separate reloads if intentional. * To **pause** a destination without losing its cursor, don’t remove it— give it a `silent` bandwidth schedule. A paused destination keeps holding the WAL-cleanup floor. See[Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/). --- # Size WAL retention Source: https://docs.influxdata.com/influxdb3/enterprise/edr/size-wal-retention/ * EDR 1.0.0 [Write-ahead log (WAL)](/influxdb3/enterprise/edr/reference/glossary/#wal-write-ahead-log)retention on the source InfluxDB 3 Enterprise server determines whether EDR recovers a missed window from WAL files (fast, precise) or falls back to slower compacted-file recovery: an outage or disconnection longer than the retention window forces the fallback. WAL files are compact (about 3x smaller than[gen0](/influxdb3/enterprise/reference/storage-engine-config-options/#gen0)), so generous retention is cheap—a source ingesting 1 MB/s needs about 86 GB for 24 hours of WAL retention. This page covers sizing that retention and, if the resulting WAL buildup becomes a storage concern, using EDR’s optional WAL cleanup to bound it. > [!Important] > #### Size retention to your longest expected outage > > Size WAL retention on the source to exceed your maximum expected outage > or disconnection, plus a safety margin. See[Why retention matters](#why-retention-matters) for a sizing table. ## Why retention matters The InfluxDB flag[`--wal-snapshots-to-keep`](/influxdb3/enterprise/reference/storage-engine-config-options/#wal)(spelled `--pt-wal-snapshots-to-keep` on InfluxDB 3.10.x) controls how many WAL snapshots are retained before deletion: ``` retention_window ~= snapshots_to_keep x snapshot_interval (~10-20s) ``` |`--wal-snapshots-to-keep`|Approximate retention window| |-------------------------|----------------------------| | 5 *(InfluxDB default)* | \~1-2 minutes | | 100 | \~17-33 minutes | | 1,000 | \~3-6 hours | | 10,000 | \~28-56 hours | | 100,000 | \~12-23 days | The InfluxDB default (5) retains only about one to two minutes of data. Any agent restart or network interruption longer than that sends EDR to compacted-file recovery. EDR is **correct** either way—delivery is at-least-once regardless, and data that outlives WAL retention is recovered from the compacted files (see[Historic fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill)and[Gap fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#gap-fill)). But EDR replicates faster with sized retention: WAL replication is precise and cheap, while recovery from[cv2](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor)is slower and replicates more bytes than strictly necessary. If you see gaps regularly (`gaps_pending` in metrics), increase`--wal-snapshots-to-keep` on the source InfluxDB—recovery from compacted files is a safety net, not the intended steady state. ## WAL cleanup for production Generous WAL retention also means WAL files linger on the source object store long after EDR has replicated them. The **source server** owns their lifecycle and deletes them once they’re rolled into gen0 (governed by`--wal-snapshots-to-keep`). If you raise that retention so EDR has a wide replication window, WAL files can accumulate well past the point EDR has already replicated them. For production, set server WAL retention high enough for your outage window and enable the opt-in `--wal-cleanup-enabled` sweep to bound the replicated tail’s storage cost. The agent deletes already-replicated WAL below four safety floors. Cleanup is off by default because it deletes source WAL. Server-negotiated retention holds are not part of EDR 1.0.0. ``` influxdb3-edr ... --wal-cleanup-enabled \ --wal-cleanup-interval-secs 300 \ --wal-cleanup-snapshot-margin 100 ``` | Flag | Default | Description | |------------------------------------|---------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--wal-cleanup-enabled` | `false` | Enable agent-side deletion of already-replicated WAL files. | | `--wal-cleanup-interval-secs` | `300` | How often the cleanup sweep runs. | | `--wal-cleanup-snapshot-margin` | `100` | WAL files to keep below the last snapshotted WAL id—the safety margin behind the snapshot boundary. | |`--wal-cleanup-max-destination-hold`|unbounded|Fan-out only: a destination whose cursor hasn’t advanced for this long (`"7d"`, `"12h"`, `"30m"`) stops holding the cleanup floor; it recovers the evicted range through gap fill when it returns (over-replication, never loss). Unset = a down destination pins WAL indefinitely.| When enabled, every `--wal-cleanup-interval-secs` (default 5 minutes) the agent deletes WAL files for each source node **only below the lowest of four floors**, so it can never strand anything that still needs them: 1. **The replication cursor**—only files EDR has already replicated. (This also means cleanup can never make a not-yet-replicated file look “missing” and trigger a false gap.) 2. **The snapshot boundary minus `--wal-cleanup-snapshot-margin`**—WAL above the source’s last snapshot is not yet in gen0, so the source node itself still needs it to recover. The margin (default 100 files) keeps a safety buffer below that boundary. 3. **The historic-fill consumed floor**—historic fill reads older WAL during backfill; cleanup stays below what it still needs (and does nothing until historic fill’s plan exists). 4. **The agent’s own pipeline floor**—the oldest WAL file still queued or in flight in the agent, plus any range still unresolved in the gap ledger. A failed WAL file’s skip receipt holds this floor until the gap ledger records the failure. The replication cursor alone is not a safe bound for the agent’s own input: skip receipts can advance it past still-queued files, and without this floor a sufficiently-lagged agent could delete its own unprocessed backlog—a silent, unrecoverable shortfall. With **multiple destinations** the first, third, and fourth floors are the**min across every destination**—WAL is retained for the most-behind consumer. A destination that is down therefore pins WAL indefinitely by default; `--wal-cleanup-max-destination-hold` bounds that: a destination whose cursor hasn’t advanced within the hold stops holding the floor (one loud WARN naming it), and when it returns it recovers the evicted range from gen0 and cv2 files through gap fill—the standard over-replication trade, never loss. Deletion is by WAL-id range (no object-store listing) and resumes from a persisted per-node watermark, so steady-state sweeps only touch the small new increment. Normal sweeps log at debug; a sweep that deletes files logs one info line per node (`wal cleanup: deleted replicated WAL files`, with the watermark and the four floors); delete errors warn and the watermark is held (retried next sweep). > [!Note] > `edr_upstream_wal_files_deleted_total` is an **agent-level** count > (cleanup is one task for the whole source), mirrored onto every > destination’s series—don’t sum it across the `destination` label, that > multiplies the real count by the number of destinations. After enabling WAL cleanup, confirm `gaps_pending` and`wal_files_lost_total` stay flat—cleanup is designed never to create gaps, so a rise is a signal to investigate. | --wal-snapshots-to-keep | Approximate retention window | | --- | --- | | --wal-snapshots-to-keep | Approximate retention window | | 5 (InfluxDB default) | ~1-2 minutes | | 100 | ~17-33 minutes | | 1,000 | ~3-6 hours | | 10,000 | ~28-56 hours | | 100,000 | ~12-23 days | | Flag | Default | Description | | --- | --- | --- | | Flag | Default | Description | | --wal-cleanup-enabled | false | Enable agent-side deletion of already-replicated WAL files. | | --wal-cleanup-interval-secs | 300 | How often the cleanup sweep runs. | | --wal-cleanup-snapshot-margin | 100 | WAL files to keep below the last snapshotted WAL id—the safety margin behind the snapshot boundary. | | --wal-cleanup-max-destination-hold | unbounded | Fan-out only: a destination whose cursor hasn’t advanced for this long ( "7d" , "12h" , "30m" ) stops holding the cleanup floor; it recovers the evicted range through gap fill when it returns (over-replication, never loss). Unset = a down destination pins WAL indefinitely. | --- # Troubleshoot EDR Source: https://docs.influxdata.com/influxdb3/enterprise/edr/troubleshoot/ * EDR 1.0.0 When something looks wrong, start with the[symptom quick index](/influxdb3/enterprise/edr/troubleshoot/common-issues/#quick-index). It maps a symptom to the first tool to run and what that tool tells you. | Your situation | Guide | |-------------------------------------------------------------|-----------------------------------------------------------------------------------------------| | Something looks wrong and you need a starting point | [Symptom quick index](/influxdb3/enterprise/edr/troubleshoot/common-issues/#quick-index) | |The agent won’t start, data isn’t flowing, or lag is climbing| [Troubleshoot common EDR issues](/influxdb3/enterprise/edr/troubleshoot/common-issues/) | | The agent crashed or restarted and you have its log |[EDR log post-mortem playbook](/influxdb3/enterprise/edr/troubleshoot/log-postmortem-playbook/)| Before you act on a live `metrics` or UI reading, cross-check it against`edr-inspect state`. A `metrics` snapshot is point-in-time and resets each run, so it can catch a transient at its worst instant. See[Data not flowing](/influxdb3/enterprise/edr/troubleshoot/common-issues/#data-not-flowing). ### [Troubleshoot common EDR issues](/influxdb3/enterprise/edr/troubleshoot/common-issues/) Symptom-to-tool quick index, agent startup errors, halted-state recovery, replication lag, and Docker-specific troubleshooting for EDR. ### [EDR log post-mortem playbook](/influxdb3/enterprise/edr/troubleshoot/log-postmortem-playbook/) Diagnose what happened to a crashed or restarted EDR agent from a captured log, using structured JSON log fields and a deterministic procedure. | Your situation | Guide | | --- | --- | | Your situation | Guide | | Something looks wrong and you need a starting point | Symptom quick index | | The agent won’t start, data isn’t flowing, or lag is climbing | Troubleshoot common EDR issues | | The agent crashed or restarted and you have its log | EDR log post-mortem playbook | --- # Troubleshoot common EDR issues Source: https://docs.influxdata.com/influxdb3/enterprise/edr/troubleshoot/common-issues/ * EDR 1.0.0 When something looks wrong, start here. The quick index maps a symptom to the first tool to run; the sections below go deeper. ## Quick index Symptom, what to run first, and what it tells you: * **Replication stopped (“halted”)**—run `edr-inspect metrics` (live), or check the log’s last `REPLICATION HALTED ...` (after a crash). Tells you the halt **class** (auth, schema, or network). Fix the cause—the halted batch retries as a probe and **auto-resumes**; nothing is skipped. * **Lag climbing (“behind”)**—run `edr-inspect state`, then`edr-inspect metrics`. Tells you whether anything is actually**lost** (rare) or just **owed backlog** that drains once the link recovers. Lag != loss. * **Data missing**—run `edr-inspect state` (lost and gap counts).**The only authority on loss.** A non-zero `lost(unrecoverable)`here is real; a scary `metrics` reading is not. * **Agent crashed or restarted**—capture the log, then follow the log post-mortem procedure below. In-memory metrics reset on restart—the log is the only witness. > [!Important] > Trust order when the evidence disagrees: durable `state` \> the log’s > sequence \> a single live `metrics` snapshot. A `metrics` or UI > reading is point-in-time and resets each run, so it can catch a > transient at its worst instant. Always confirm a conclusion against`edr-inspect state` before acting—see[Data not flowing](#data-not-flowing). ## Agent won’t start * **“config must include at least one of ‘downstream’ or ‘upstreams’”**—config has neither section. Add one. * **“downstream.historic\_fill must be set explicitly …”**—no historic intent declared. Add `historic_fill: { mode: none }` (live-only),`full`, or `since`. See[Historic fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill). * **“historic\_fill mode ‘since’ requires a ‘since’ value”, or “… is not parseable”**—`since` missing or malformed. Set `since` to a whole-day duration (`7d`) or an ISO date (`2026-06-01`); sub-day units (`30m` or `6h`) are rejected. * **“Multiple ingest nodes detected… idempotent\_writes is false”**—multi-node store without the guarantee. Add`idempotent_writes: true` or point at a single-node store. * **“failed to resolve token”**—token file missing from store. Check`--token-store` path and file names. * **"… changed — requires full restart" (on reload)**—a non-reloadable field changed. Restart the agent. ## Data not flowing 1. First-line triage: `edr-inspect state ` (what’s owed and what’s in progress, even if the agent is wedged) and`edr-inspect topology` and `edr-inspect metrics` (live health and flow). See[Triage CLI](/influxdb3/enterprise/edr/monitor/#triage-cli-edr-inspect). For what the state files mean—and before deleting ANY of them—see[State and recovery](/influxdb3/enterprise/edr/reference/state-and-recovery/). 2. Check node info: `curl http://127.0.0.1:9091/edr/v1/node_info`(observability listener, loopback) 3. Check connectivity: `curl http://destination:9090/health` (network listener) 4. Check [write-ahead log (WAL)](/influxdb3/enterprise/edr/reference/glossary/#wal-write-ahead-log)files exist: `ls //pt_wal/` 5. Check scope includes the database and table. 6. Verbose logs: restart with `RUST_LOG=debug`. > [!Important] > Before acting on a `metrics` or UI reading, cross-check `state`. A live`metrics` snapshot is point-in-time and resets each run, so it can catch a > transient at its worst instant—for example, `health: Error`, a high`live-lag`, or `REPLICATION HALTED: N batch(es) since 1s ago` shown > alongside `consecutive-fails=0` (a halt only \~1s old, already recovering).`edr-inspect state` is the durable authority on what is genuinely lost or > gapped; if it shows no loss and no pending gaps, the system is caught up > regardless of how the live reading looks. ## Common log messages * **`historic fill: MANIFEST GAP ...`**—expected: the compactor deleted old manifests; recovery falls back to WAL files, then to[cv2](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor). See[Historic fill](/influxdb3/enterprise/edr/monitor/historic-and-gap-fill/#historic-fill). * **cv2 over-replication warnings**—expected when recovering compacted data; idempotent writes absorb it. * **`... POTENTIAL DATA LOSS`**—not expected: data was deleted from every tier (WAL,[gen0](/influxdb3/enterprise/reference/storage-engine-config-options/#gen0), cv2) before EDR could read it. Investigate retention settings. Historic fill emits this as `historic fill: MANIFEST GAP — ... no cv2 files found covering the gap — POTENTIAL DATA LOSS`. * **Schema conflict, or channel blocked, retrying every 60s**—destination rejected a write due to a type mismatch. Drop the conflicting table on the destination; the retry recreates it with the source schema. * **Unauthorised**—destination rejected the auth token. Update the token file; the next retry picks it up. * **Cursor reset detected**—InfluxDB WAL IDs reset (for example, after a data wipe with state retained). The agent auto-detects and resets its cursor. * **`failed to connect to downstream` followed by a burst of`batch send failed`, right after a restart**—restart-ordering gap: the source came up before the destination’s listener was accepting. Clears at the next `connected to downstream` or `edge connected`. Benign and self-healing; not data loss. * **`rejecting data: upstream has not called /connect` (destination side)**—momentary: data arrived a beat before the `/connect` handshake re-registered (typical just after a restart). Self-clears once the source re-issues `/connect`. * **Source `REPLICATION HALTED: auth failure` and destination `batch write failed: auth failure` on the same `batch_id`**—both ends agree, so this is a genuine token mismatch on that hop, not a source-side phantom. Update the token (see *Unauthorised* above); the next probe resumes. ## Protocol incompatible (agents of different versions) Symptoms: nothing flows to one downstream; health shows **Halted**; the log carries `protocol halt: replication to this downstream is STOPPED` once at ERROR and a `PROTOCOL-HALTED` WARN every minute; the downstream’s log (or the upstream’s error) names both protocol versions and both minimums. * This is deliberate: the two agents’ protocol versions cannot interoperate (or the downstream lacks a capability the config requires, for example`encoding: pt`). The message says which side must be upgraded. * Upgrade the named agent. Negotiation re-runs on **every report** (default 10s), so the halted upstream resumes by itself within one report interval (`protocol compatibility restored` in the log); a config reload also re-negotiates immediately. * Agents released before protocol versioning (0.1.x and 0.2.x) are always accepted as protocol v1—they can never be refused by current builds. See [Compatibility](/influxdb3/enterprise/edr/reference/compatibility/) for the full protocol version matrix. ## Stale binaries After replacing a binary (for example, an EDR upgrade), make sure the old process is actually gone before concluding the new one misbehaves: stop the agent, start it again, and confirm the running binary’s modification time predates the process start time. A lingering old process is a common cause of “the new version didn’t take effect”. ## Port conflicts and orphan processes EDR agents, proxies, and InfluxDB instances hold their listen ports until fully stopped. If a start fails with “address in use”, find the holder with`lsof -ti :`. Process managers that track PID files can lose track of processes across crashes—always verify with`ps aux | grep influxdb3 | grep -v grep` that no orphans remain before restarting. Orphaned old agents reading the same state file as a new agent will fight over the cursor. ## Replication lag * **Lag is not loss.** A high `live-lag` or `pending_bytes` is almost always **owed backlog** that drains once the destination catches up, not missing data. `edr-inspect state` (`lost(unrecoverable)`, pending gaps) is the authority on what—if anything—is actually unrecoverable. Confirm there before treating lag as an incident. * High `pending_bytes` indicates that the destination is slow or unreachable. * If WAL files are deleted before replication, increase`--wal-snapshots-to-keep` on the source server. See[Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/). Gap fill will recover, but at a cost. * If WAL discovery is slow, lower `--poll-interval-ms` to list the object store more frequently. * If the link is constrained, check whether a `bandwidth_schedule` window is active (`limited` or `silent` shows in the UI and logs). ## Incorrect field values in wide tables If a table has more than 100 fields and was replicated by 0.2.0, rc.1, or rc.2, upgrade to rc.3 or GA on a supported server line. Earlier builds can silently assign values to the wrong field names when types match, or halt with a schema conflict when types differ. Verify field-level values at the destination or re-replicate affected tables after upgrading. Matching row counts do not prove that field values are correct. See the [rc.3 release warning](/influxdb3/enterprise/edr/release-notes/#v100-0rc3). ## Source WAL / disk growing * The **source** server owns WAL lifecycle; a large or growing `pt_wal`footprint usually means `--wal-snapshots-to-keep` is set high (often to cover long outages). That is working as intended—WAL is retained until the source rolls it to gen0. * If the buildup of already-replicated WAL is a problem, enable EDR’s opt-in `--wal-cleanup-enabled` so the agent prunes the replicated tail itself. See [Size WAL retention](/influxdb3/enterprise/edr/size-wal-retention/). High server WAL retention plus opt-in agent cleanup is the supported production pattern for EDR 1.0.0. The sweep stays below all four floors: cursor, snapshot boundary minus margin, historic-fill floor, and queued or in-flight work plus unresolved gap ranges. * After enabling, confirm `gaps_pending` and `wal_files_lost_total`stay flat—a rise is not expected (cleanup is designed never to create gaps) and warrants a look. A sweep that prunes anything logs`wal cleanup: deleted replicated WAL files`; delete failures warn and hold the watermark. ## Log post-mortem (after a crash or restart) When an agent has **crashed, been killed, or restarted**, the live views can’t help—its in-memory metrics reset on restart, so the logs are the only witness. EDR logs structured JSON to stdout; capture them first (a`--rm` container discards them on removal): ```bash docker logs > edr.log 2>&1 # or: journalctl -u -o cat > edr.log ``` Then reconstruct the final run: 1. The last startup banner marks the final run (count banners to get the number of restarts). 2. The last successful send is last-known-good. 3. A `shutdown signal received` followed by a `... shutting down`cascade means a clean stop; **no** shutdown markers means an unclean exit (crash, out-of-memory (OOM), or kill). 4. Identify the dominant ERROR or WARN class and its fields (`table`,`upstream`, `error`). 5. Write a one-line verdict and remediation. For the complete rule set—delivery failures, data loss, cursor and state, connectivity, config, and infrastructure signatures—see the[EDR log post-mortem playbook](/influxdb3/enterprise/edr/troubleshoot/log-postmortem-playbook/). The manual procedure above always works without any additional tooling. ## Docker-specific troubleshooting See [Run EDR in Docker](/influxdb3/enterprise/edr/install/docker/#docker-specific-troubleshooting). --- # EDR log post-mortem playbook Source: https://docs.influxdata.com/influxdb3/enterprise/edr/troubleshoot/log-postmortem-playbook/ * EDR 1.0.0 Diagnose what happened to an EDR agent from a captured log, after the fact. This helps most when the agent crashed or restarted. The live views (`edr-inspect metrics` and `topology`) can’t help in that case, because the agent’s in-memory counters reset on every restart. EDR logs are structured JSON, so this playbook relies on deterministic field extraction, not guesswork. 1. [Capture the log first](#capture-the-log-first) 2. [Log shape](#log-shape) 3. [Reconstruct the final run](#reconstruct-the-final-run) 4. [Detector rules](#detector-rules) 5. [Worked example](#worked-example) 6. [Limits](#limits) 7. [Corroborate with `edr-inspect`](#corroborate-with-edr-inspect) ## Capture the log first EDR logs JSON to **stdout**. There’s no log file by default, so capture the evidence before it’s gone. ```bash # Docker -- do this BEFORE removing a --rm container (removal discards logs): docker logs > edr.log 2>&1 # systemd: journalctl -u -o cat > edr.log ``` If you ship logs to an aggregator such as Loki, ELK, or CloudWatch, export the agent’s stream for the incident window instead. This playbook covers the dead or restarted case. Pair it with `edr-inspect`: * **`state`** reads the durable journals **offline**, so it works even on a dead agent. * **`metrics`** and **`topology`** give the live picture, but only **if the agent is running again**. See [Corroborate with `edr-inspect`](#corroborate-with-edr-inspect). ## Log shape Each line is a JSON object with these fields: | Field | Value | |-----------|-------------------------------------------------------------------------------------------------| |`timestamp`| Time of the log line | | `level` | `ERROR`, `WARN`, `INFO`, or `DEBUG` | | `target` | The module | | `fields` |Structured context, for example `node_id`, `upstream`, `wal_id`, `database`, `table`, and `error`| | `message` | The log message | What each level means: * **`ERROR`** and **`WARN`** always indicate a real problem. EDR doesn’t log warnings for expected conditions. * **`INFO`** is routine. It covers startup, shutdown, and other lifecycle transitions, not faults. Useful filters: ```bash jq -c 'select(.level=="ERROR" or .level=="WARN")' edr.log # just the abnormal jq -c 'select(.fields.upstream=="Dublin")' edr.log # one channel jq -rc '[.timestamp,.level,.message]|@tsv' edr.log | tail -50 # the tail, compact ``` ## Reconstruct the final run [Capture the log first](#capture-the-log-first), then run this procedure. It reconstructs what happened in the agent’s final run. The live tools can’t give you that once the agent is gone, because their counters reset on restart. Reconstruct the final run in five steps: 1. **Find the final run.** Locate the last startup banner:`upstream replication pipeline started` or `starting HTTP servers`. Everything after it belongs to the run that ended.**Count the banners** across the whole log to get the number of (re)starts. 2. **Find last-known-good.** Find the last successful send or replication before the tail—that is, “last good at T.” 3. **Classify the ending.** * **Clean stop**—the tail contains `shutdown signal received` followed by the orderly cascade (`WAL replicator shutting down`,`tiered dispatcher shutting down`, `... shutting down`). An operator or orchestrator stopped the agent. * **Unclean**—the tail has **no** shutdown markers: the log just stops, or ends on an `ERROR` or a panic. This indicates a crash, an out-of-memory (OOM) kill, or `SIGKILL`. 4. **Find the dominant failure.** Identify the most frequent `ERROR` or`WARN` class in the final run, with its fields (`table=`, `upstream=`, `error=`). 5. **State the verdict in one line.** Combine the findings above, for example: > > > *“Final run started 11:31; last good send 11:33; then 18 > schema-conflict warnings on db=sensors table=air\_quality;**no shutdown marker, so the run ended uncleanly at 11:41**. > Likely cause: schema conflict—drop the table downstream.”* > > ## Detector rules Each rule keys on a real log signal. Lead with evidence: when you cite a rule, record its **count, its first and last timestamp, and a sample line**. Never report a bare verdict. Each entry below gives the signal, its severity, and what it means. ### Exit and lifecycle * **`upstream replication pipeline started`** or**`starting HTTP servers`** (info)—marks a run boundary.**Count occurrences to find restarts.**Many, tightly spaced, indicate a **crash loop**. The live tools can’t show a crash loop, because counters reset each run. * **`shutdown signal received`** plus the `... shutting down` cascade at the tail (info)—indicates a **clean stop** initiated by an operator or orchestrator. Not a fault. * **The tail has none of the above signals**; it ends on `ERROR` or just stops (high)—indicates an **unclean exit**: a crash, an OOM kill, or `SIGKILL`. The headline post-mortem fact. ### Delivery failures (why replication stalled) * **Halt**, or `halted batch ... is the probe` (high)—indicates a**halted** channel: it’s wedged on a batch it can’t deliver. The cause is the **preceding** auth, schema, or write error. Resolve that cause; the halted batch retries as the probe and resumes. * **`Unauthorised`** (high)—the downstream rejected the auth token. Update the token file; the next retry picks it up. * **Schema conflict**, or `channel blocked, retrying` (high)—the downstream rejected a write on a type mismatch. Drop the conflicting table downstream; the retry recreates it with the source schema. * **`schema conflict resolved — replication resumed`** (info)—the agent recovered, so the conflict was **transient**. Closes the loop on an earlier halt. ### Data loss (durable—always report) * **`POTENTIAL DATA LOSS`** (critical)—data is missing from **all** tiers. Record the time window. Check retention and eviction settings; this doesn’t self-heal. * **`historic fill: ... no Gen0 AND no cv2 coverage — data may be missing`** (critical)—a historic range is unrecoverable from any medium. * **WAL files lost**, or eviction outrunning replication (high)—the source evicted WAL before EDR replicated it. Raise `--wal-snapshots-to-keep`; gap fill recovers what it can, at a cost. ### State and cursor * **`failed to save cursor — will not survive restart`** (high)—the cursor didn’t persist, which explains **post-restart rework** (re-replication). Check the state location’s durability and permissions. * **`cursor state unparseable — falling back to previous-good copy`**(medium)—the primary was corrupt. The agent recovered from the mirror and self-healed, so no operator action is needed. This explains nothing else, and it isn’t a large re-replication. * **`previous-good cursor also unparseable — starting from default`** or**`cursor state loss: primary AND mirror both corrupt/unreadable`**(high)—indicates **state loss**: both copies are gone. What happens next depends on `on_state_loss`; see the next two entries. * **`cursor rebuilt from snapshot floor`** (per node) (high)—indicates`on_state_loss: recover` fired. It explains a re-replication bounded by the node’s retained WAL, not the whole backlog from WAL 1. It isn’t silent either: the position and the reason are both in this line. * **`cursor state loss ... Refusing to start per on_state_loss: halt`**(critical)—the agent **did not start**. Restore the state location from backup, or switch to `on_state_loss: recover` (requires `idempotent_writes: true`). * **`historic fill manifest state loss ... Refusing to start per on_state_loss: halt`** (critical)—same as the previous entry, but for the historic manifest specifically. It’s distinct from `historic_fill is configured but live replication has already started`, which is a config-ambiguity refusal, not a corruption event. * **`Cursor reset detected`** (medium)—the upstream InfluxDB WAL IDs reset, for example after a data wipe with state retained. The agent auto-resets; this explains a discontinuity. * **`failed to save gap ledger ...`** (medium)—gap obligations might not have survived a restart. ### Connectivity * **`upstream timed out — marking disconnected`** alternating with**`reconnect requested`** (medium)—indicates **flapping** if the two cycle. Report the rate over the window. This identifies network instability, not an EDR fault. ### Config and startup refusal (the agent never started) * **`config must include at least one of 'downstream' or 'upstreams'`**(high)—the agent **refused to start**. Fix the config. * **`historic fill: cannot parse 'since' value — aborting`** or**`... requires a 'since' value`** (high)—the agent refused to start. Set `since` to a whole-day duration (`7d`) or an ISO date; sub-day units are rejected. * **`config reload skipped — requires full restart`** (medium)—a non-reloadable field changed, so the agent kept the old config and the change was **silently not applied**. Restart to apply the change. ### Infrastructure * **`catalog could not be read from the object store (transient?)`** or**`verify the object store is reachable, then restart`** (high)—the object store or InfluxDB is unreachable. The root cause is **infrastructure**, not an EDR fault. * **`catalog not found ... cannot resolve database/table names`** or**`ensure the InfluxDB instance has been started at least once`**(high)—the source InfluxDB never wrote a catalog. Start the InfluxDB 3 Enterprise instance before the agent. If it runs on 3.10.x, start it with `--upgrade-pacha-tree` so it writes the[upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)’s catalog format. ### WAL cleanup (only when `--wal-cleanup-enabled`) * **`wal cleanup: deleted replicated WAL files`** (info)—**routine**. The agent pruned already-replicated WAL and logs the watermark and floors. This isn’t data loss: only WAL below`min(cursor, snapshot-margin, historic-fill floor)` is pruned. * **`wal cleanup: delete failed`** or**`sweep had delete errors; watermark held`** (low)—transient object-store delete errors. The watermark is held and the range retries on the next sweep (idempotent). Persistent errors warrant checking store reachability and permissions. * **`wal cleanup: historic fill not ready — skipping sweep`** (info)— cleanup is configured, but historic fill’s plan isn’t built yet, so the sweep no-ops (conservative). Expected early in a backfill. ### Everything else Roll up the remaining `ERROR` and `WARN` lines **by `target`** with counts, so the long tail is visible and you don’t silently drop anything. ## Worked example ```json {"timestamp":"...11:31:02","level":"INFO","message":"upstream replication pipeline started"} {"timestamp":"...11:33:10","level":"INFO","message":"...replicated...","fields":{"wal_id":1042}} {"timestamp":"...11:33:41","level":"WARN","message":"schema conflict","fields":{"database":"sensors","table":"air_quality"}} ... (17 more schema-conflict WARNs, ~1/min) ... {"timestamp":"...11:41:55","level":"ERROR","message":"...halt..."} ``` **Verdict:** one run (one banner, no restarts). Last good send at 11:33:10. Starting at 11:33:41, 18 schema-conflict `WARN` lines on`sensors/air_quality`, escalating to a halt at 11:41:55.**No shutdown marker, so this is an unclean end**—the process crashed or was killed while halted.**Likely cause:** schema conflict. Drop `air_quality` downstream; the retry recreates it with the source schema. ## Limits Keep these limits in mind so you don’t overstate the verdict: * **The procedure needs a captured log.**The evidence is gone if a `--rm` container was removed, or if the log rotated past the incident.[Capture the log first](#capture-the-log-first). * **A clean exit is an inference, not a fact.**You infer it from shutdown markers, so a truncated or rotated log can look unclean. Phrase the finding as “no shutdown marker *in the captured window*.” * **`SIGKILL` and OOM kills emit nothing.**You can infer only an “abrupt end,” never prove the cause. To find the real cause, correlate with the host’s OOM killer or orchestrator events. ## Corroborate with `edr-inspect` The log says how the agent got there.`edr-inspect` says where it ended up and whether it’s still broken. ### Check the evidence in this order Confirm and quantify the log’s findings with the evidence below. Each kind of evidence answers a different question: 1. `edr-inspect state `—the durable truth: where did it actually end up? Always check it first. It needs no live agent, because it reads the journals offline, and the journals survive on the volume. It shows the WAL cursor, the gap ledger (pending and unrecoverable), and historic progress and lost-counts. This is the primary corroborator. 2. The log—the sequence: did it recover, and what was the cause? The log shows the incident. 3. `edr-inspect metrics [addr]` and `edr-inspect topology [addr]`—current health: is it still failing? Check these only if the agent is back up, because both need a live agent. * `metrics` shows the live health of the restarted agent. Its counters cover the current run, so they show recovery state, not the incident. Treat the numbers as data from “the new run.” A single reading might be a blip. * `topology` shows per-channel health after recovery: which upstream or downstream is degraded now. ### Map a log finding to its corroborator * **`POTENTIAL DATA LOSS` or WAL lost**—check `state`: HISTORIC lost-counts and GAP FILL `unrecoverable` entries. * **`Cursor reset` or a corrupt-cursor re-seed**—check `state`: LIVE REPLICATION (current cursor compared to expected). * **Gap advisories**—check `state`: the gap ledger (what’s `pending` or `in-progress` now). * **Halt, `Unauthorised`, or schema conflict**, if the agent is **now running**—check `metrics` for `REPLICATION HALTED` and `last-write`. Check `topology` for channel health. These views show whether the agent is still halted or has recovered. For the corresponding JSON fields and numeric values, see the [halt field values](/influxdb3/enterprise/edr/reference/api/#halt-fields). ### When the evidence disagrees No single kind of evidence is trustworthy alone, so cross-check before you conclude. A metrics reading is a point-in-time snapshot of the current run. It can capture a transient failure at its peak. The snapshot can overstate the incident’s severity. For example, a captured snapshot might show`REPLICATION HALTED: 1 batch(es)` with high lag. That output can look like the agent stopped while authentication was failing. Yet the log shows `REPLICATION RESUMED` the same second.`edr-inspect state` shows the cursor fully caught up with zero loss. When the evidence disagrees, the log and `state` win over a lone`metrics` snapshot. A clean conclusion needs at least the durable `state` to confirm it. If you captured metrics or state snapshots at the time of the incident, and not just live readings, feed all of them into the post-mortem and reconcile them against the log. Don’t act on any one in isolation. | Field | Value | | --- | --- | | Field | Value | | timestamp | Time of the log line | | level | ERROR , WARN , INFO , or DEBUG | | target | The module | | fields | Structured context, for example node_id , upstream , wal_id , database , table , and error | | message | The log message | --- # Get started with InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/ This guide walks through the basic steps of getting started with InfluxDB 3 Enterprise, including the following: 1. [Set up InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/setup/) 2. [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/) 3. [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/) 4. [Query data in InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/query/) 5. [Migrate from InfluxDB v1 or v2](/influxdb3/enterprise/get-started/migrate-from-influxdb-v1-v2/) 6. [Process data in InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/process/) > [!Tip] > #### Find support for InfluxDB 3 Enterprise > > The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. > For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. ## Data model The InfluxDB 3 Enterprise server contains logical databases; databases contain tables; and tables are comprised of columns. Compared to previous versions of InfluxDB, you can think of a database as an InfluxDB v2 `bucket` in v2 or an InfluxDB v1 `db/retention_policy`. A `table` is equivalent to an InfluxDB v1 and v2 `measurement`. Columns in a table represent time, tags, and fields. Columns can be one of the following types: * String dictionary (tag) * `int64` (field) * `float64` (field) * `uint64` (field) * `bool` (field) * `string` (field) * `time` (time with nanosecond precision) In InfluxDB 3 Enterprise, every table has a primary key–the ordered set of tags and the time–for its data. The primary key uniquely identifies each and determines the sort order for all Parquet files related to the table. When you create a table, either through an explicit call or by writing data into a table for the first time, it sets the primary key to the tags in the order they arrived. Although InfluxDB is still a *schema-on-write* database, the tag column definitions for a table are immutable. Tags should hold unique identifying information like `sensor_id`, `building_id`, or `trace_id`. All other data should be stored as fields. ## Tools to use The following table compares tools that you can use to interact with InfluxDB 3 Enterprise. This tutorial covers many of the recommended tools. | Tool |Administration|Write|Query| |------------------------------------------------------------------------------------|--------------|-----|-----| | **[`influxdb3` CLI](/influxdb3/enterprise/reference/cli/influxdb3/)** | **✓** |**✓**|**✓**| | **[InfluxDB HTTP API](/influxdb3/enterprise/reference/api/)** | **✓** |**✓**|**✓**| | **[InfluxDB 3 Explorer](/influxdb3/explorer/)** | **✓** |**✓**|**✓**| |[InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) | \- |**✓**|**✓**| |[InfluxDB v2 client libraries](/influxdb3/enterprise/reference/client-libraries/v2/)| \- |**✓**| \- | |[InfluxDB v1 client libraries](/influxdb3/enterprise/reference/client-libraries/v1/)| \- |**✓**|**✓**| | [InfluxDB 3 processing engine](/influxdb3/enterprise/plugins/) | |**✓**|**✓**| | [Telegraf](/telegraf/v1/) | \- |**✓**| \- | | [Chronograf](/chronograf/v1/) | \- | \- | \- | | `influx` CLI | \- | \- | \- | | `influxctl` CLI | \- | \- | \- | | InfluxDB v2.x user interface | \- | \- | \- | | **Third-party tools** | | | | | Flight SQL clients | \- | \- |**✓**| | [Grafana](/influxdb3/enterprise/visualize-data/grafana/) | \- | \- |**✓**| [Set up InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/setup/) #### Related * [Query system data](/influxdb3/enterprise/admin/query-system-data/) * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/) * [Query data in InfluxDB 3 Enterprise](/influxdb3/enterprise/query-data/) | Tool | Administration | Write | Query | | --- | --- | --- | --- | | Tool | Administration | Write | Query | | influxdb3 CLI | ✓ | ✓ | ✓ | | InfluxDB HTTP API | ✓ | ✓ | ✓ | | InfluxDB 3 Explorer | ✓ | ✓ | ✓ | | InfluxDB 3 client libraries | - | ✓ | ✓ | | InfluxDB v2 client libraries | - | ✓ | - | | InfluxDB v1 client libraries | - | ✓ | ✓ | | InfluxDB 3 processing engine | | ✓ | ✓ | | Telegraf | - | ✓ | - | | Chronograf | - | - | - | | influx CLI | - | - | - | | influxctl CLI | - | - | - | | InfluxDB v2.x user interface | - | - | - | | Third-party tools | | | | | Flight SQL clients | - | - | ✓ | | Grafana | - | - | ✓ | --- # Migrate from InfluxDB v1 or v2 Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/migrate-from-influxdb-v1-v2/ InfluxDB 3 provides compatibility APIs and tools for migrating existing InfluxDB v1 and v2 workloads. Use existing client libraries and tools with minimal changes to your code. ## Write data InfluxDB 3 supports v1 and v2 compatible write endpoints: * **`/api/v2/write`**: Compatible with InfluxDB v2 clients and tools * **`/write`**: Compatible with InfluxDB v1 clients and tools Both endpoints accept line protocol and write data the same way. For more information, see [Use compatibility APIs to write data](/influxdb3/enterprise/write-data/http-api/compatibility-apis/). ## Query data InfluxDB 3 supports the v1 HTTP query API for InfluxQL queries: * **`/query`**: Compatible with InfluxDB v1 query clients For more information, see [Use the v1 HTTP query API](/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/). ## Client libraries Use InfluxDB v1 and v2 client libraries with InfluxDB 3 Enterprise: * [v2 client libraries](/influxdb3/enterprise/reference/client-libraries/v2/) * [v1 client libraries](/influxdb3/enterprise/reference/client-libraries/v1/) --- # Create a multi-node cluster Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/multi-server/ Create a multi-node InfluxDB 3 Enterprise cluster for high availability, performance, and workload isolation. Configure nodes with specific *modes* (ingest, query, process, compact) to optimize for your use case. ## Prerequisites * Shared object store ## Basic multi-node setup > [!Note] > The examples in this guide assume each node runs on a distinct host and uses the default port (8181). > If you run multiple nodes on the same host for testing or development, specify a different port for each node using the `--http-bind` option–for example, `--http-bind localhost:8282`. ```bash ## NODE 1 compacts stored data # Example variables # node-id: 'host01' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --mode ingest,query,compact \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --http-bind localhost:8181 \ --aws-access-key-id \ --aws-secret-access-key ``` ``` ## NODE 2 handles writes and queries # Example variables # node-id: 'host02' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host02 \ --cluster-id cluster01 \ --mode ingest,query \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY ``` Learn how to set up a multi-node cluster for different use cases, including high availability, read replicas, processing data, and workload isolation. * [Create an object store](#create-an-object-store) * [Connect to your object store](#connect-to-your-object-store) * [Server modes](#server-modes) * [Cluster configuration examples](#cluster-configuration-examples) * [Writing and querying in multi-node clusters](#writing-and-querying-in-multi-node-clusters) ## Create an object store With the InfluxDB 3 Enterprise diskless architecture, all data is stored in a common object store. In a multi-node cluster, you connect all nodes to the same object store. A multi-node cluster requires one of the following object stores: * AWS S3 (or S3-compatible, such as [MinIO](/influxdb3/enterprise/admin/object-storage/minio/)) * Azure Blob Storage * Google Cloud Storage The `file` object store supports a single node only. Don’t share a [`--data-dir`](/influxdb3/enterprise/reference/config-options/#data-dir) between nodes, including over a network file system. > [!Note] > Refer to your object storage provider’s documentation for > setting up an object store. ## Connect to your object store When starting your InfluxDB 3 Enterprise node, include provider-specific options for connecting to your object store–for example: [S3 or S3-compatible](#s3-or-s3-compatible)[Azure Blob Storage](#azure-blob-storage)[Google Cloud Storage](#google-cloud-storage) To use an AWS S3 or S3-compatible object store, provide the following options with your `influxdb3 serve` command: * `--object-store`: `s3` * `--bucket`: Your AWS S3 bucket name * `--aws-access-key-id`: Your AWS access key ID *(can also be defined using the `AWS_ACCESS_KEY_ID` environment variable)* * `--aws-secret-access-key`: Your AWS secret access key *(can also be defined using the `AWS_SECRET_ACCESS_KEY` environment variable)* ```bash influxdb3 serve \ # ... --object-store s3 \ --bucket AWS_BUCKET_NAME \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY ``` *For information about other S3-specific settings, see[Configuration options - AWS](/influxdb3/enterprise/reference/config-options/#aws).* To use Azure Blob Storage as your object store, provide the following options with your `influxdb3 serve` command: * `--object-store`: `azure` * `--bucket`: Your Azure Blob Storage container name * `--azure-storage-account`: Your Azure Blob Storage account name *(can also be defined using the `AZURE_STORAGE_ACCOUNT` environment variable)* * `--azure-storage-access-key`: Your Azure Blob Storage access key *(can also be defined using the `AZURE_STORAGE_ACCESS_KEY` environment variable)* ```bash influxdb3 serve \ # ... --object-store azure \ --bucket AZURE_CONTAINER_NAME \ --azure-storage-account AZURE_STORAGE_ACCOUNT \ --azure-storage-access-key AZURE_STORAGE_ACCESS_KEY ``` To use Google Cloud Storage as your object store, provide the following options with your `influxdb3 serve` command: * `--object-store`: `google` * `--bucket`: Your Google Cloud Storage bucket name * `--google-service-account`: The path to your Google credentials JSON file*(can also be defined using the `GOOGLE_SERVICE_ACCOUNT` environment variable)* ```bash influxdb3 serve \ # ... --object-store google \ --bucket GOOGLE_BUCKET_NAME \ --google-service-account GOOGLE_SERVICE_ACCOUNT ``` ## Server modes InfluxDB 3 Enterprise *modes* determine what subprocesses the Enterprise node runs. These subprocesses fulfill required tasks including data ingestion, query processing, compaction, and running the processing engine. The `influxdb3 serve --mode` option defines what subprocesses a node runs. Each node can run in one *or more* of the following modes: * **all** *(default)*: Runs all necessary subprocesses. * **ingest**: Runs the data ingestion subprocess to handle writes. * **query**: Runs the query processing subprocess to handle queries. * **process**: Runs the processing engine subprocess to trigger and execute plugins. * **compact**: Runs the compactor subprocess to optimize data in object storage. > [!Important] > Only *one* node in your cluster can run in `compact` mode. ### Server mode examples #### Configure a node to only handle write requests ```bash influxdb3 serve \ # ... --mode ingest ``` #### Configure a node to only run the Compactor ```bash influxdb3 serve \ # ... --mode compact ``` #### Configure a node to only run the Processing Engine ```bash influxdb3 serve \ # ... --mode process \ --plugin-dir /path/to/plugins ``` > [!Important] > When using `process` mode, you must also specify `--plugin-dir` to configure > the Python plugin environment. Without this flag, starting a node with `process`mode will fail with an error. See the [plugin-dir configuration option](/influxdb3/enterprise/reference/config-options/#plugin-dir)for more information. > > We recommend running `process` alone on a dedicated node. > For the complete setup, see [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). ## Cluster configuration examples * [High availability cluster](#high-availability-cluster) * [High availability with a dedicated Compactor](#high-availability-with-a-dedicated-compactor) * [High availability with read replicas and a dedicated Compactor](#high-availability-with-read-replicas-and-a-dedicated-compactor) ### High availability cluster A minimum of two nodes are required for basic high availability (HA), with both nodes reading and writing data. Basic high availability setup In a basic HA setup: * Two nodes both write data to the same object store and both handle queries * Node 1 and Node 2 are *read replicas* that read from each other’s object store directories * One of the nodes is designated as the Compactor node > [!Note] > Only one node can be designated as the Compactor. > Compacted data is meant for a single writer, and many readers. The following examples show how to configure and start two nodes for a basic HA setup. * *Node 1* is for compaction * *Node 2* is for ingest and query ```bash ## NODE 1 # Example variables # node-id: 'host01' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --mode ingest,query,compact \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --http-bind localhost:8181 \ --aws-access-key-id \ --aws-secret-access-key ``` ```bash ## NODE 2 # Example variables # node-id: 'host02' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host02 \ --cluster-id cluster01 \ --mode ingest,query \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY ``` After the nodes have started, querying either node returns data for both nodes, and *NODE 1* runs compaction. To add nodes to this setup, start more read replicas with the same cluster ID. ### High availability with a dedicated Compactor Data compaction in InfluxDB 3 Enterprise is one of the more computationally demanding operations. To ensure stable performance in ingest and query nodes, set up a compactor-only node to isolate the compaction workload. Dedicated Compactor setup The following examples sets up high availability with a dedicated Compactor node: 1. Start two read-write nodes as read replicas, similar to the previous example. ``` ## NODE 1 — Writer/Reader Node #1 # Example variables # node-id: 'host01' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --mode ingest,query \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --http-bind localhost:8181 \ --aws-access-key-id \ --aws-secret-access-key ``` ``` ## NODE 2 — Writer/Reader Node #2 # Example variables # node-id: 'host02' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host02 \ --cluster-id cluster01 \ --mode ingest,query \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id \ --aws-secret-access-key ``` 2. Start the dedicated compactor node with the `--mode=compact` option to ensure the node **only** runs compaction. ``` ## NODE 3 — Compactor Node # Example variables # node-id: 'host03' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host03 \ --cluster-id cluster01 \ --mode compact \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id \ --aws-secret-access-key ``` ### High availability with read replicas and a dedicated Compactor For a robust and effective setup for managing time-series data, you can run ingest nodes alongside query nodes and a dedicated Compactor node. Workload Isolation Setup 1. Start ingest nodes with the **`ingest`** mode. > [!Note] > Send all write requests to only your ingest nodes. ``` ## NODE 1 — Writer Node #1 # Example variables # node-id: 'host01' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --mode ingest \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --http-bind localhost:8181 \ --aws-access-key-id \ --aws-secret-access-key ``` ``` ## NODE 2 — Writer Node #2 # Example variables # node-id: 'host02' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host02 \ --cluster-id cluster01 \ --mode ingest \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id \ --aws-secret-access-key ``` 2. Start the dedicated Compactor node with the `compact` mode. ``` ## NODE 3 — Compactor Node # Example variables # node-id: 'host03' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host03 \ --cluster-id cluster01 \ --mode compact \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id \ ``` 3. Finally, start the query nodes using the `query` mode. > [!Note] > Send all query requests to only your query nodes. ``` ## NODE 4 — Read Node #1 # Example variables # node-id: 'host04' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host04 \ --cluster-id cluster01 \ --mode query \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id \ --aws-secret-access-key ``` ``` ## NODE 5 — Read Node #2 # Example variables # node-id: 'host05' # cluster-id: 'cluster01' # bucket: 'influxdb-3-enterprise-storage' influxdb3 serve \ --node-id host05 \ --cluster-id cluster01 \ --mode query \ --object-store s3 \ --bucket influxdb-3-enterprise-storage \ --aws-access-key-id \ ``` ## Writing and querying in multi-node clusters You can use the default port `8181` for any write or query request without changing any of the commands. > [!Note] > #### Specify hosts for write and query requests > > To benefit from this multi-node, isolated architecture: > > * Send write requests to a node that you have designated as an ingester. > * Send query requests to a node that you have designated as a querier. > > When running multiple local instances for testing or separate nodes in > production, specifying the host ensures writes and queries are routed to the > correct instance. ```bash # Example querying a specific host # HTTP-bound Port: 8585 influxdb3 query \ --host http://localhost:8585 --token AUTH_TOKEN \ --database DATABASE_NAME \ "QUERY" ``` Replace the following placeholders with your values: * `http://localhost:8585`: the host and port of the node to query * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to query the specified database * `DATABASE_NAME`: the name of the database to query * `QUERY`: the SQL or InfluxQL query to run against the database [Set up InfluxDB](/influxdb3/enterprise/get-started/setup/)[Write data](/influxdb3/enterprise/get-started/write/) #### Related * [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/) [cluster](/influxdb3/enterprise/tags/cluster/)[multi-node](/influxdb3/enterprise/tags/multi-node/)[multi-server](/influxdb3/enterprise/tags/multi-server/) --- # Process data in InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/process/ The InfluxDB 3 Enterprise processing engine is an embedded Python virtual machine (VM) that runs code inside the database to process and transform data. Create processing engine [plugins](#plugin) that run when [triggered](#trigger)by specific events. * [Before you begin](#before-you-begin) * [Processing engine terminology](#processing-engine-terminology) * [Plugin](#plugin) * [Trigger](#trigger) * [Trigger types](#trigger-types) * [Activate the processing engine](#activate-the-processing-engine) * [Create a plugin](#create-a-plugin) * [Test a plugin on the server](#test-a-plugin-on-the-server) * [Create a trigger](#create-a-trigger) * [Enable the trigger](#enable-the-trigger) ## Processing engine terminology ### Plugin A plugin is a Python function that has a signature compatible with a processing engine [trigger](#trigger). ### Trigger When you create a trigger, you specify a [plugin](#plugin), a database, optional arguments, and a trigger specification, which defines when the plugin is executed and what data it receives. #### Trigger types InfluxDB 3 provides the following types of triggers, each with specific specifications: * **WAL rows** (`table:` or `all_tables`): Sends a batch of written data (for a specific table or all tables) to a plugin when the database flushes data to the [Write-Ahead Log (WAL)](/influxdb3/enterprise/reference/internals/durability/#write-ahead-log-wal-persistence (by default, every second). * **Scheduled** (`every:` or `cron:`): Executes a plugin on a user-configured schedule (using a crontab or a duration). This trigger type is useful for data collection and deadman monitoring. * **HTTP request** (`request:`): Binds a plugin to a custom HTTP API endpoint at`/api/v3/engine/`. The plugin receives the HTTP request headers and content, and can parse, process, and send the data into the database or to third-party services. ## Before you begin Before you can use the processing engine, you need: * A running InfluxDB 3 Enterprise server * An admin token for authentication * A database to attach triggers to If you haven’t completed these steps, see [Set up InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/setup/). ## Activate the processing engine To activate the processing engine, include the `--plugin-dir ` option when starting the InfluxDB 3 Enterprise server.`PLUGIN_DIR` is your file system location for storing [plugin](#plugin) files for the processing engine to run. In a cluster, `--plugin-dir` automatically adds `process` mode to the node. In a cluster, we recommend running the processing engine on a standalone process node. See [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). > [!Note] > **Docker and DEB/RPM installations**: The Processing Engine is already enabled—no additional configuration needed. > To disable it, see [Enable and disable the Processing Engine](/influxdb3/enterprise/reference/processing-engine/#enable-and-disable-the-processing-engine). > [!Note] > If you manually installed InfluxDB 3 Enterprise from a tar archive, ensure the `influxdb3` binary and `python/` directory remain in the same parent directory. The install script handles this automatically. ```bash influxdb3 serve \ # ... --plugin-dir PLUGIN_DIR ``` Replace `PLUGIN_DIR`with the path to your plugin directory. This path can be absolute or relative to the current working directory of the `influxdb3` server. ## Create a plugin To create a plugin, write and store a Python file in your configured `PLUGIN_DIR`. The following example shows a WAL rows plugin, which receives data as it’s flushed to the Write-Ahead Log. The example shows how to process, query, and write data from within a plugin. ##### Example Python plugin for data writes ``` # Example: A data write plugin for the InfluxDB 3 Processing Engine # When creating a trigger, you can provide runtime arguments to your plugin, # allowing you to write generic code that uses variables such as monitoring # thresholds, environment variables, and host names. # # This example implements the process_writes signature compatible with a wal_rows # trigger specification. # Use the exact signature to define the function. # When you create a wal_rows trigger, you specify the database # and tables that the plugin receives written data from on every WAL flush # (default is once per second). def process_writes(influxdb3_local, table_batches, args=None): # Use logging to track plugin execution if args and "arg1" in args: influxdb3_local.info("arg1: " + args["arg1"]) # here we're using arguments provided at the time the trigger was set up # to feed into parameters that we'll put into a query query_params = {"room": "Kitchen"} # The following example shows how to execute a parameterized query. Only SQL is supported. # It queries the database that the trigger is configured for. query_result = influxdb3_local.query("SELECT * FROM home where room = '$room'", query_params) # The result is a list of Dict that have the column name as key and value as # value. influxdb3_local.info("query result: " + str(query_result)) # this is the data that is sent when data is written to the database and flushed to the WAL. # One batch for each table (will only be one if triggered on a single table) for table_batch in table_batches: # here you can see that the table_name is available. influxdb3_local.info("table: " + table_batch["table_name"]) # example to skip the table we're later writing data into if table_batch["table_name"] == "some_table": continue # and then the individual rows, which are Dict with keys of the column names and values for row in table_batch["rows"]: influxdb3_local.info("row: " + str(row)) # this shows building a line of LP to write back to the database. tags must go first and # their order is important and must always be the same for each individual table. Then # fields and lastly an optional time, which you can see in the next example below line = LineBuilder("some_table")\ .tag("tag1", "tag1_value")\ .tag("tag2", "tag2_value")\ .int64_field("field1", 1)\ .float64_field("field2", 2.0)\ .string_field("field3", "number three") # this writes it back (it actually just buffers it until the completion of this function # at which point it will write everything back that you put in) influxdb3_local.write(line) # here's another example, but with us setting a nanosecond timestamp at the end other_line = LineBuilder("other_table") other_line.int64_field("other_field", 1) other_line.float64_field("other_field2", 3.14) other_line.time_ns(1302) # and you can see that we can write to any DB in the server influxdb3_local.write_to_db("mytestdb", other_line) # just some log output as an example influxdb3_local.info("done") ``` ## Test a plugin on the server Use one of the [`influxdb3 test`](/influxdb3/enterprise/reference/cli/influxdb3/test/)CLI commands to test your processing engine plugin safely without affecting actual data. During a plugin test: * A query executed by the plugin queries against the server you send the request to. * Writes aren’t sent to the server but are returned to you. To test a `process_writes` (WAL) plugin: 1. Save the [example plugin code](#example-python-plugin-for-data-writes) to a plugin file inside of the plugin directory. If you haven’t yet written data to the table in the example, comment out the lines where it queries. 2. To run the test, enter the following command with the following options: * `--lp` or `--file`: The line protocol to test * Optional: `--input-arguments`: A comma-delimited list of `=` arguments for your plugin code ```bash influxdb3 test wal_plugin \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --lp INPUT_LINE_PROTOCOL \ --input-arguments INPUT_ARGS \ PLUGIN_FILENAME ``` Replace the following: * `INPUT_LINE_PROTOCOL`: the line protocol to test * Optional: `INPUT_ARGS`: a comma-delimited list of `=` arguments for your plugin code–for example, `arg1=hello,arg2=world` * `DATABASE_NAME`: the name of the database to test against * `AUTH_TOKEN`: your [token](/influxdb3/enterprise/admin/tokens/)for your InfluxDB 3 Enterprise server * `PLUGIN_FILENAME`: the name of the plugin file to test. Provide only the filename (for example, `test.py`), not a relative or absolute path. ### Example: Test a plugin ```bash # Test a plugin # Requires: # - A database named `mydb` with a table named `foo` # - A Python plugin file named `test.py` # Test a plugin influxdb3 test wal_plugin \ --lp "my_measure,tag1=asdf f1=1.0 123" \ --token apiv3_0xxx0o0XxXxx00Xxxx000xXXxoo0== \ --database sensors \ --input-arguments "arg1=hello,arg2=world" \ test.py ``` The command runs the plugin code, yields the test data to the plugin, and then responds with the plugin result. You can quickly see how the plugin behaves, what data it would have written to the database, and any errors. You can then edit your Python code in the plugins directory, and rerun the test. The server reloads the file for every request to the `test` API. For more information about testing plugins, see the following: * [`influxdb3 test wal_plugin`](/influxdb3/enterprise/reference/cli/influxdb3/test/wal_plugin/)or run `influxdb3 test wal_plugin -h` * [`influxdb3 test schedule_plugin`](/influxdb3/enterprise/reference/cli/influxdb3/test/schedule_plugin/)or run `influxdb3 test schedule_plugin -h` ## Create a trigger With the plugin code inside the server plugin directory, and a successful test, you’re ready to create a trigger to run the plugin. Use the[`influxdb3 create trigger` command](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/)to create a trigger. ```bash # Create a trigger that runs the single-file plugin influxdb3 create trigger \ --token apiv3_0xxx0o0XxXxx00Xxxx000xXXxoo0== \ --database sensors \ --path test_plugin.py \ --trigger-spec "table:foo" \ --trigger-arguments "arg1=hello,arg2=world" \ trigger1 ``` > [!Note] > #### Plugin paths > > * For **single-file plugins**, provide just the `.py` filename to `--path` (for example, `test_plugin.py`). > * For **multi-file plugins**, provide the directory name containing `__init__.py`. > > When not using `--upload`, the server resolves paths relative to the configured `--plugin-dir`. > For details about multi-file plugin structure, see [Create your plugin file](/influxdb3/enterprise/plugins/#create-your-plugin-file). ## Enable the trigger After you have created a plugin and trigger, enter the following command to enable the trigger and have it run the plugin as you write data: ```bash influxdb3 enable trigger \ --token AUTH_TOKEN \ --database DATABASE_NAME \ TRIGGER_NAME ``` Replace the following placeholders with your values: * `DATABASE_NAME`: the name of the database to enable the trigger in * `AUTH_TOKEN`: your [token](/influxdb3/enterprise/admin/tokens/) * `TRIGGER_NAME`: the name of the trigger to enable For example, to enable the trigger named `trigger1` in the `sensors` database: ```bash influxdb3 enable trigger \ --token apiv3_0xxx0o0XxXxx00Xxxx000xXXxoo0== \ --database sensors \ trigger1 ``` ## Next steps If you’ve completed this Get Started guide for InfluxDB 3 Enterprise, learn more about tools and options for: * [Writing data](/influxdb3/enterprise/write-data/) * [Querying data](/influxdb3/enterprise/query-data/) * [Processing data with plugins](/influxdb3/enterprise/plugins/) * [Visualizing data](/influxdb3/enterprise/visualize-data/) #### Related * [Processing engine and Python plugins](/influxdb3/enterprise/plugins/) * [influxdb3 create trigger](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/) --- # Query data in InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/query/ InfluxDB 3 Enterprise supports both native SQL and InfluxQL for querying data. InfluxQL is an SQL-like query language designed for InfluxDB v1 and customized for time series queries. > [!Note] > Flux, the language introduced in InfluxDB v2, is **not** supported in InfluxDB 3. * [Query data with the influxdb3 CLI](#query-data-with-the-influxdb3-cli) * [Example queries](#example-queries) * [Other tools for executing queries](#other-tools-for-executing-queries) * [SQL vs InfluxQL](#sql-vs-influxql) * [SQL](#sql) * [InfluxQL](#influxql) * [Optimize queries](#optimize-queries) * [Last values cache](#last-values-cache) * [Distinct values cache](#distinct-values-cache) * [File indexes](#file-indexes) ## Query data with the influxdb3 CLI To get started querying data in InfluxDB 3 Enterprise, use the[`influxdb3 query` command](/influxdb3/enterprise/reference/cli/influxdb3/query/)and provide the following: * `-H`, `--host`: The host URL of the server *(default is `http://127.0.0.1:8181`)* * `-d`, `--database`: *(Required)* The name of the database to query * `-l`, `--language`: The query language of the provided query string * `sql` *(default)* * `influxql` * SQL or InfluxQL query as a string > [!Important] > If the `INFLUXDB3_AUTH_TOKEN` environment variable defined in[Set up InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/setup/#set-your-token-for-authorization)isn’t set in your environment, set it or provide your token using > the `-t, --token` option in your command. To query the home sensor sample data you wrote in[Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/#write-data-using-the-cli), run the following command: [SQL](#sql-2)[InfluxQL](#influxql-2) ```bash influxdb3 query \ --database DATABASE_NAME \ "SELECT * FROM home ORDER BY time" ``` ```bash influxdb3 query \ --database DATABASE_NAME \ --language influxql \ "SELECT * FROM home" ``` *Replace `DATABASE_NAME`with the name of the database to query.* To query from a specific time range, use the `WHERE` clause to designate the boundaries of your time range. [SQL](#sql-3)[InfluxQL](#influxql-3) ```bash influxdb3 query \ --database DATABASE_NAME \ "SELECT * FROM home WHERE time >= now() - INTERVAL '7 days' ORDER BY time" ``` ```bash influxdb3 query \ --database DATABASE_NAME \ --language influxql \ "SELECT * FROM home WHERE time >= now() - 7d" ``` ### Example queries [](#list-tables-in-a-database) List tables in a database [SQL](#sql-4)[InfluxQL](#influxql-4) ```sql SHOW TABLES ``` ```sql SHOW MEASUREMENTS ``` [](#return-the-average-temperature-of-all-rooms) Return the average temperature of all rooms [SQL](#sql-5)[InfluxQL](#influxql-5) ```sql SELECT avg(temp) AS avg_temp FROM home ``` ```sql SELECT MEAN(temp) AS avg_temp FROM home ``` [](#return-the-average-temperature-of-the-kitchen) Return the average temperature of the kitchen [SQL](#sql-6)[InfluxQL](#influxql-6) ```sql SELECT avg(temp) AS avg_temp FROM home WHERE room = 'Kitchen' ``` ```sql SELECT MEAN(temp) AS avg_temp FROM home WHERE room = 'Kitchen' ``` [](#query-data-from-an-absolute-time-range) Query data from an absolute time range [SQL](#sql-7)[InfluxQL](#influxql-7) ```sql SELECT * FROM home WHERE time >= '2022-01-01T12:00:00Z' AND time <= '2022-01-01T18:00:00Z' ``` ```sql SELECT * FROM home WHERE time >= '2022-01-01T12:00:00Z' AND time <= '2022-01-01T18:00:00Z' ``` [](#query-data-from-a-relative-time-range) Query data from a relative time range [SQL](#sql-8)[InfluxQL](#influxql-8) ```sql SELECT * FROM home WHERE time >= now() - INTERVAL '7 days' ``` ```sql SELECT * FROM home WHERE time >= now() - 7d ``` [](#calculate-average-humidity-in-3-hour-windows-per-room) Calculate average humidity in 3-hour windows per room [SQL](#sql-9)[InfluxQL](#influxql-9) ```sql SELECT date_bin(INTERVAL '3 hours', time) AS time, room, avg(hum) AS avg_hum FROM home GROUP BY 1, room ORDER BY room, 1 ``` ```sql SELECT MEAN(hum) AS avg_hum FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(3h), room ``` ## Other tools for executing queries Other tools are available for querying data in InfluxDB 3 Enterprise, including the following: [](#query-using-the-api) Query using the API #### Query using the API InfluxDB 3 supports Flight (gRPC) APIs and an HTTP API. To query your database using the HTTP API, send a request to the `/api/v3/query_sql` or `/api/v3/query_influxql` endpoints. In the request, specify the database name in the `db` parameter and a query in the `q` parameter. You can pass parameters in the query string or inside a JSON object. Use the `format` parameter to specify the response format: `pretty`, `jsonl`, `parquet`, `csv`, and `json`. Default is `json`. ##### Example: Query passing URL-encoded parameters The following example sends an HTTP `GET` request with a URL-encoded SQL query: ```bash curl -G "http://localhost:8181/api/v3/query_sql" \ --header 'Authorization: Bearer AUTH_TOKEN' \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=select * from cpu limit 5" ``` Replace the following placeholders with your values: * `DATABASE_NAME`: the name of the database to query * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to query the specified database ##### Example: Query passing JSON parameters The following example sends an HTTP `POST` request with parameters in a JSON payload: ```bash curl http://localhost:8181/api/v3/query_sql \ --data '{"db": "DATABASE_NAME", "q": "select * from cpu limit 5"}' ``` Replace the following placeholders with your values: * `DATABASE_NAME`: the name of the database to query * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to query the specified database [](#query-using-the-python-client) Query using the Python client #### Query using the Python client Use the InfluxDB 3 Python library to interact with the database and integrate with your application. We recommend installing the required packages in a Python virtual environment for your specific project. To get started, install the `influxdb3-python` package. ```bash pip install influxdb3-python ``` From here, you can connect to your database with the client library using just the **host** and \*\*database name: ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3( token='AUTH_TOKEN', host='http://localhost:8181', database='DATABASE_NAME' ) ``` Replace the following placeholders with your values: * `DATABASE_NAME`: the name of the database to query * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to query the specified database The following example shows how to query using SQL, and then use PyArrow to explore the schema and process results. To authorize the query, the example retrieves the [database token](/influxdb3/enterprise/admin/tokens/) from the `INFLUXDB3_AUTH_TOKEN` environment variable. ```python from influxdb_client_3 import InfluxDBClient3 import os client = InfluxDBClient3( token=os.environ.get('INFLUXDB3_AUTH_TOKEN'), host='http://localhost:8181', database='servers' ) # Execute the query and return an Arrow table table = client.query( query="SELECT * FROM cpu LIMIT 10", language="sql" ) print("\n#### View Schema information\n") print(table.schema) print("\n#### Use PyArrow to read the specified columns\n") print(table.column('usage_active')) print(table.select(['host', 'usage_active'])) print(table.select(['time', 'host', 'usage_active'])) print("\n#### Use PyArrow compute functions to aggregate data\n") print(table.group_by('host').aggregate([])) print(table.group_by('cpu').aggregate([('time_system', 'mean')])) ``` For more information about the Python client library, see the[`influxdb3-python` repository](https://github.com/InfluxCommunity/influxdb3-python)in GitHub. [](#query-using-influxdb-3-explorer) Query using InfluxDB 3 Explorer #### Query using InfluxDB 3 Explorer You can use the InfluxDB 3 Explorer web-based interface to query and visualize data, and administer your InfluxDB 3 Enterprise instance. For more information, see how to [install InfluxDB 3 Explorer](/influxdb3/explorer/install/)using Docker and get started querying your data. ## SQL vs InfluxQL InfluxDB 3 Enterprise supports two query languages–SQL and InfluxQL. While these two query languages are similar, there are important differences to consider. ### SQL The InfluxDB 3 SQL implementation provides a full-featured SQL query engine powered by [Apache DataFusion](https://datafusion.apache.org/). InfluxDB extends DataFusion with additional time series-specific functionality and supports the complex SQL queries, including queries that use joins, unions, window functions, and more. * [SQL query guides](/influxdb3/enterprise/query-data/sql/) * [SQL reference](/influxdb3/enterprise/reference/sql/) * [Apache DataFusion SQL reference](https://datafusion.apache.org/user-guide/sql/index.html) ### InfluxQL InfluxQL is a SQL-like query language built for InfluxDB v1 and supported in InfluxDB 3 Enterprise. Its syntax and functionality is similar SQL, but specifically designed for querying time series data. InfluxQL does not offer the full range of query functionality that SQL does. If you are migrating from previous versions of InfluxDB, you can continue to use InfluxQL and the established InfluxQL-related APIs you have been using. * [InfluxQL query guides](/influxdb3/enterprise/query-data/influxql/) * [InfluxQL reference](/influxdb3/enterprise/reference/influxql/) * [InfluxQL feature support](/influxdb3/enterprise/reference/influxql/feature-support/) ## Optimize queries InfluxDB 3 Enterprise provides the following optimization options to improve specific kinds of queries: * [Last values cache](#last-values-cache) * [Distinct values cache](#distinct-values-cache) * [File indexes](#file-indexes) ### Last values cache The InfluxDB 3 Enterprise last values cache (LVC) stores the last N values in a series or column hierarchy in memory. This gives the database the ability to answer these kinds of queries in under 10 milliseconds. For information about configuring and using the LVC, see: * [Manage a last values cache](/influxdb3/enterprise/admin/last-value-cache/) * [Query the last values cache](/influxdb3/enterprise/admin/last-value-cache/query/) ### Distinct values cache The InfluxDB 3 Enterprise distinct values cache (DVC) stores distinct values for specified columns in a series or column hierarchy in memory. This is useful for fast metadata lookups, which can return in under 30 milliseconds. For information about configuring and using the DVC, see: * [Manage a distinct values cache](/influxdb3/enterprise/admin/distinct-value-cache/) * [Query the distinct values cache](/influxdb3/enterprise/admin/distinct-value-cache/query/) ### File indexes InfluxDB 3 Enterprise lets you customize how your data is indexed to help optimize query performance for your specific workload, especially workloads that include single-series queries. Define custom indexing strategies for databases or specific tables. For more information, see[Manage file indexes](/influxdb3/enterprise/admin/file-index/). [Write data](/influxdb3/enterprise/get-started/write/)[Processing engine](/influxdb3/enterprise/get-started/process/) #### Related * [Query data in InfluxDB 3 Enterprise](/influxdb3/enterprise/query-data/) * [SQL reference documentation](/influxdb3/enterprise/reference/sql/) * [Apache DataFusion SQL reference](https://datafusion.apache.org/user-guide/sql/index.html) * [InfluxQL reference documentation](/influxdb3/enterprise/reference/influxql/) --- # Set up InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/setup/ * [Prerequisites](#prerequisites) * [Quick-Start Mode (Development)](#quick-start-mode-development) * [Start InfluxDB](#start-influxdb) * [Object store examples](#object-store-examples) * [Set up licensing](#set-up-licensing) * [Available license types](#available-license-types) * [Set up authorization](#set-up-authorization) * [Create an operator token](#create-an-operator-token) * [Set your token for authorization](#set-your-token-for-authorization) ## Prerequisites To get started, you’ll need: * **InfluxDB 3 Enterprise**: [Install and verify the latest version](/influxdb3/enterprise/install/) on your system. * If you want to persist data, have access to one of the following: * A directory on your local disk where you can persist data (used by examples in this guide) * S3-compatible object store and credentials ## Quick-Start Mode (Development) For development, testing, and home use, you can start InfluxDB 3 Enterprise without any arguments. The system automatically generates required configuration values based on your system’s hostname: ```bash influxdb3 ``` When you run `influxdb3` without arguments, the following values are auto-generated: * **`node-id`**: `{hostname}-node` (or `primary-node` if hostname is unavailable) * **`cluster-id`**: `{hostname}-cluster` (or `primary-cluster` if hostname is unavailable) * **`object-store`**: `file` * **`data-dir`**: `~/.influxdb` The system displays warning messages showing the auto-generated identifiers: ``` Using auto-generated node id: mylaptop-node. For production deployments, explicitly set --node-id Using auto-generated cluster id: mylaptop-cluster. For production deployments, explicitly set --cluster-id ``` > [!Important] > #### When to use quick-start mode > > Quick-start mode is designed for development, testing, and home lab environments > where simplicity is prioritized over explicit configuration. > > **For production deployments**, use explicit configuration values with the[`influxdb3 serve` command](/influxdb3/enterprise/reference/cli/influxdb3/serve/)as shown in the [Start InfluxDB](#start-influxdb) section below. > > Quick-start mode listens on all network interfaces. > Before you start the server on a network that others can reach, see[Create your admin token before you expose the server](#start-influxdb). **Configuration precedence**: Environment variables override auto-generated defaults. For example, if you set `INFLUXDB3_NODE_ID=my-node`, the system uses `my-node` instead of generating `{hostname}-node`. ## Start InfluxDB Use the [`influxdb3 serve` command](/influxdb3/enterprise/reference/cli/influxdb3/serve/)to start InfluxDB 3 Enterprise. Provide the following: * `--node-id`: A string identifier that distinguishes individual server instances within the cluster. This forms the final part of the storage path:`//`. In a multi-node setup, this ID is used to reference specific nodes. * `--cluster-id`: A string identifier that determines part of the storage path hierarchy. All nodes within the same cluster share this identifier. The storage path follows the pattern `//`. In a multi-node setup, this ID is used to reference the entire cluster. * `--object-store`: Specifies the type of object store to use. InfluxDB supports the following: * `file`: local file system * `memory`: in memory *(no object persistence)* * `memory-throttled`: like `memory` but with latency and throughput that somewhat resembles a cloud-based object store * `s3`: AWS S3 and S3-compatible services like Ceph or Minio * `google`: Google Cloud Storage * `azure`: Azure Blob Storage * Other object store parameters depending on the selected `object-store` type. For example, if you use `s3`, you must provide the bucket name and credentials. * *(Optional)* `--http-bind`: The address and port for the HTTP API*(default is `0.0.0.0:8181`, which listens on all network interfaces)*. To accept only local connections until you create your first admin token, specify `127.0.0.1:8181`. * *(Optional, v3.11+)* `--mode all,webui`: Serves the InfluxDB 3 Explorer web UI from the server process.`all` doesn’t include `webui`, so name `webui` explicitly. This mode also requires `--webui-session-secret`. Explorer is served at the root path of the server’s HTTP address–for example, [http://localhost:8181/](http://localhost:8181/). For the requirements and the full startup command, see[Use the integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/). > [!Caution] > #### Create your admin token before you expose the server > > Until the first admin token exists, the `/api/v3/configure/token/admin`endpoint accepts unauthenticated requests, and the server listens on all > network interfaces by default. > Anyone who can reach the port during that window can claim the operator token > and then use the processing engine to run code on the host. > > Close the window using one of the following methods: > > * Start the server with `--http-bind 127.0.0.1:8181` (with Docker, publish > the port as `127.0.0.1:8181:8181`),[create your admin token](#create-an-operator-token), and then restart the > server on the address you intend to use. > * Block the server port with a firewall until you create the token. > * Start the server with a[preconfigured admin token](/influxdb3/enterprise/admin/tokens/admin/preconfigured/)so the endpoint never accepts unauthenticated requests. > We recommend this method for automated and production deployments. > [!Note] > #### Diskless architecture > > InfluxDB 3 supports a diskless architecture that can operate with object > storage alone, eliminating the need for locally attached disks. > InfluxDB 3 Enterprise can also work with only local disk storage when needed. > > The combined path structure `//` ensures > proper organization of data in your object store, allowing for clean > separation between clusters and individual nodes. For this getting started guide, use the `file` object store to persist data to your local disk. ```bash # File system object store # Provide the filesystem directory influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --object-store file \ --data-dir ~/.influxdb3 ``` ### Object store examples [](#file-system-object-store) File system object store Store data in a specified directory on the local filesystem. This is the default object store type. Replace the following with your values: ```bash # Filesystem object store # Provide the filesystem directory influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --object-store file \ --data-dir ~/.influxdb3 ``` [](#docker-with-a-mounted-file-system-object-store) Docker with a mounted file system object store To run the [Docker image](/influxdb3/enterprise/install/#pull-the-docker-image) and persist data to the local file system, mount a volume for the object store–for example, provide the following options with your `docker run` command: * `--volume /path/on/host:/path/in/container`: Mounts a directory from your file system to the container * `--object-store file --data-dir /path/in/container`: Uses the volume for object storage ``` # File system object store with Docker # Create a mount # Provide the mount path docker run -it \ --volume /path/on/host:/path/in/container \ influxdb:3-enterprise influxdb3 serve \ --node-id my_host \ --cluster-id my_cluster \ --object-store file \ --data-dir /path/in/container ``` > [!Note] > The InfluxDB 3 Enterprise Docker image exposes port `8181`, the `influxdb3`server default for HTTP connections. > To map the exposed port to a different port when running a container, see the > Docker guide for [Publishing and exposing ports](https://docs.docker.com/get-started/docker-concepts/running-containers/publishing-ports/). [](#docker-compose-with-a-mounted-file-system-object-store) Docker compose with a mounted file system object store Open `compose.yaml` for editing and add a `services` entry for InfluxDB 3 Enterprise–for example: ``` # compose.yaml services: influxdb3-enterprise: image: influxdb:3-enterprise ports: - 8181:8181 command: - influxdb3 - serve - --node-id=node0 - --cluster-id=cluster0 - --object-store=file - --data-dir=/var/lib/influxdb3/data - --plugin-dir=/var/lib/influxdb3/plugins # Optional: only needed for processing engine plugins environment: - INFLUXDB3_LICENSE_EMAIL=EMAIL_ADDRESS volumes: - type: bind # Path to store data on your host system source: ~/.influxdb3/data # Path to store data in the container target: /var/lib/influxdb3/data - type: bind # Path to store plugins on your host system source: ~/.influxdb3/plugins # Path to store plugins in the container target: /var/lib/influxdb3/plugins ``` Replace `EMAIL_ADDRESS` with your email address to bypass the email prompt when generating a trial or at-home license. For more information, see [Manage your InfluxDB 3 Enterprise license](/influxdb3/enterprise/admin/license/). Use the Docker Compose CLI to start the server–for example: ```bash docker compose pull && docker compose up influxdb3-enterprise ``` The command pulls the latest InfluxDB 3 Enterprise Docker image and starts`influxdb3` in a container with host port `8181` mapped to container port`8181`, the server default for HTTP connections. > [!Tip] > #### Custom port mapping > > To customize your `influxdb3` server hostname and port, specify the[`--http-bind` option or the `INFLUXDB3_HTTP_BIND_ADDR` environment variable](/influxdb3/enterprise/reference/config-options/#http-bind). > > For more information about mapping your container port to a specific host port, see the > Docker guide for [Publishing and exposing ports](https://docs.docker.com/get-started/docker-concepts/running-containers/publishing-ports/). [](#s3-object-storage) S3 object storage Store data in an S3-compatible object store. This is useful for production deployments that require high availability and durability. Provide your bucket name and credentials to access the S3 object store. ```bash # S3 object store # Specify the object store type and associated options influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --object-store s3 \ --bucket OBJECT_STORE_BUCKET \ --aws-default-region AWS_REGION \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY ``` > [!Note] > If not specified, `--aws-default-region` defaults to `us-east-1`. > Specify a different region if your bucket is in another AWS region to avoid redirect errors. ```bash # Minio or other open source object store # (using the AWS S3 API with additional parameters) # Specify the object store type and associated options influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --object-store s3 \ --bucket OBJECT_STORE_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --aws-endpoint ENDPOINT \ --aws-allow-http ``` [](#memory-based-object-store) Memory-based object store Store data in RAM without persisting it on shutdown. It’s useful for rapid testing and development. ```bash # Memory object store # Stores data in RAM; doesn't persist data influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --object-store memory ``` For more information about server options, use the CLI help or view the[InfluxDB 3 CLI reference](/influxdb3/enterprise/reference/cli/influxdb3/serve/): ```bash influxdb3 serve --help ``` ## Set up licensing When you first start a new instance, InfluxDB 3 Enterprise prompts you to select a license type. InfluxDB 3 Enterprise licenses: * **Authorize** usage of InfluxDB 3 Enterprise software for a single cluster. * **Apply per cluster**, with limits based primarily on CPU cores. * **Vary by license type**, each offering different capabilities and restrictions. ### Available license types * **Trial**: 30-day trial license with full access to InfluxDB 3 Enterprise capabilities. * **At-Home**: For at-home hobbyist use with limited access to InfluxDB 3 Enterprise capabilities. * **Commercial**: Commercial license with full access to InfluxDB 3 Enterprise capabilities. > [!Important] > #### Trial and at-home licenses with Docker > > To generate the trial or home license in Docker, bypass the email prompt. > The first time you start a new instance, provide your email address with the`--license-email` option or the `INFLUXDB3_LICENSE_EMAIL` environment variable. > > *Currently, if you use Docker and enter your email address in the prompt, a bug may > prevent the container from generating the license .* > > For more information, see [the Docker Compose example](/influxdb3/enterprise/admin/license/?t=Docker+compose#start-the-server-with-your-license-email). > [!Tip] > #### Use the InfluxDB 3 Explorer query interface > > You can complete the remaining steps in this guide using InfluxDB 3 Explorer, > the web-based query and administrative interface for InfluxDB 3. > Explorer provides visual management of databases and tokens and an > easy way to write and query your time series data. > > For more information, see the [InfluxDB 3 Explorer documentation](/influxdb3/explorer/). ## Set up authorization InfluxDB 3 Enterprise uses token-based authorization to authorize actions in the database. Authorization is enabled by default when you start the server. With authorization enabled, you must provide a token with `influxdb3` CLI commands and HTTP API requests. InfluxDB 3 Enterprise supports the following types of tokens: * **admin token**: Grants access to all CLI actions and API endpoints. * **resource tokens**: Tokens that grant read and write access to specific resources (databases and system information endpoints) on the server. * A database token grants access to write and query data in a database * A system token grants read access to system information endpoints and metrics for the server > [!Tip] > ### Preconfigured admin tokens for automated deployments > > For CI/CD pipelines or automated deployments, you can start InfluxDB 3 Enterprise with a preconfigured admin token file instead of creating tokens manually after startup. > For more information, see [Use a preconfigured admin token](/influxdb3/enterprise/admin/tokens/admin/preconfigured/). ### Create an operator token After you start the server, create your first admin token. The first admin token you create is the *operator* token for the server. Use the [`influxdb3 create token` command](/influxdb3/enterprise/reference/cli/influxdb3/create/token/)with the `--admin` option to create your operator token: [CLI](#cli)[Docker](#docker) ```bash influxdb3 create token --admin ``` ```bash # With Docker — in a new terminal: docker exec -it CONTAINER_NAME influxdb3 create token --admin ``` Replace `CONTAINER_NAME` with the name of your running Docker container. The command returns a token string for authenticating CLI commands and API requests. > [!Important] > #### Store your token securely > > InfluxDB displays the token string only when you create it. > Store your token securely—you cannot retrieve it from the database later. Now that an admin token exists, the server requires a token for all requests. If you started the server on `127.0.0.1` to protect the[bootstrap window](#start-influxdb), you can now restart it on the address you want to use–for example, `--http-bind IP_ADDRESS:8181` to listen on a specific interface, or omit `--http-bind` to listen on all interfaces (default is`--http-bind 0.0.0.0:8181`). ### Set your token for authorization Use your operator token to authenticate server actions in InfluxDB 3 Enterprise, such as creating additional tokens, performing administrative tasks, and writing and querying data. #### Authorize CLI commands Use one of the following methods to provide your token and authenticate `influxdb3` CLI commands. In your command, replace `YOUR_AUTH_TOKEN` with your token string (for example, the [operator token](#create-an-operator-token) from the previous step). ##### Set an environment variable (recommended) [macOS and Linux](#macos-and-linux)[PowerShell](#powershell)[CMD](#cmd) ```bash export INFLUXDB3_AUTH_TOKEN=YOUR_AUTH_TOKEN ``` ```powershell $env:INFLUXDB3_AUTH_TOKEN = "YOUR_AUTH_TOKEN" ``` ``` set INFLUXDB3_AUTH_TOKEN=YOUR_AUTH_TOKEN # Make sure to include a space character at the end of this command. ``` ##### Use the `--token` option ```bash influxdb3 show databases --token YOUR_AUTH_TOKEN ``` #### Authorize HTTP API requests For HTTP API requests, include your token in the `Authorization` header–for example: ```bash curl "http://localhost:8181/api/v3/configure/database" \ --header "Authorization: Bearer YOUR_AUTH_TOKEN" ``` #### Learn more about tokens and permissions * [Manage admin tokens](/influxdb3/enterprise/admin/tokens/admin/) - Understand and manage operator and named admin tokens * [Manage resource tokens](/influxdb3/enterprise/admin/tokens/resource/) - Create, list, and delete resource tokens * [Authentication](/influxdb3/enterprise/reference/internals/authentication/) - Understand authentication, authorizations, and permissions in InfluxDB 3 Enterprise [Get started](/influxdb3/enterprise/get-started/)[Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/) #### Related * [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/) * [Manage tokens](/influxdb3/enterprise/admin/tokens/) * [Use the InfluxDB 3 MCP server](/influxdb3/enterprise/admin/mcp-server/) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) --- # Write data to InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/get-started/write/ InfluxDB 3 Enterprise is designed for high write-throughput and uses an efficient, human-readable write syntax called *[line protocol](#line-protocol)*. InfluxDB is a schema-on-write database, meaning you can start writing data and InfluxDB creates the logical database, tables, and their schemas automatically, without any required intervention. Once InfluxDB creates the schema, it validates future write requests against the schema before accepting new data. Both new tags and fields can be added later as your schema changes. * [Line protocol](#line-protocol) * [Construct line protocol](#construct-line-protocol) * [Write data using the CLI](#write-data-using-the-cli) * [Other tools for writing data](#other-tools-for-writing-data) ## Line protocol InfluxDB 3 Enterprise accepts data in[line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) syntax. Line protocol consists of the following elements: \* Required * \* **table**: A string that identifies the[table](/influxdb3/enterprise/reference/glossary/#table) to store the data in. * **tag set**: Comma-delimited list of key value pairs, each representing a tag. Tag keys and values are unquoted strings. *Spaces, commas, and equal characters must be escaped.* * \* **field set**: Comma-delimited list of key value pairs, each representing a field. Field keys are unquoted strings. *Spaces and commas must be escaped.*Field values can be one of the following types: * [strings](/influxdb3/clustered/reference/syntax/line-protocol/#string) (quoted) * [floats](/influxdb3/clustered/reference/syntax/line-protocol/#float) * [integers](/influxdb3/clustered/reference/syntax/line-protocol/#integer) * [unsigned integers](/influxdb3/clustered/reference/syntax/line-protocol/#uinteger) * [booleans](/influxdb3/clustered/reference/syntax/line-protocol/#boolean) * **timestamp**: [Unix timestamp](/influxdb3/clustered/reference/syntax/line-protocol/#unix-timestamp)associated with the data. InfluxDB supports up to nanosecond precision. [](#how-are-influxdb-line-protocol-elements-parsed) How are InfluxDB line protocol elements parsed? * **table**: Everything before the *first unescaped comma before the first whitespace*. * **tag set**: Key-value pairs between the *first unescaped comma* and the *first unescaped whitespace*. * **field set**: Key-value pairs between the *first and second unescaped whitespaces*. * **timestamp**: Integer value after the *second unescaped whitespace*. * Lines are separated by the newline character (`\n`). Line protocol is whitespace sensitive. *For schema design recommendations, see[InfluxDB schema design recomendations](/influxdb3/enterprise/write-data/best-practices/schema-design/).* myTable,tag1=val1,tag2=val2 field1="v1",field2=1i 0000000000000000000 ## Construct line protocol With a basic understanding of line protocol, you can now construct line protocol and write data to InfluxDB 3 Enterprise. Consider a use case where you collect data from sensors in your home. Each sensor collects temperature, humidity, and carbon monoxide readings. To collect this data, use the following schema: * **table**: `home` * **tags** * `room`: Living Room or Kitchen * **fields** * `temp`: temperature in °C (float) * `hum`: percent humidity (float) * `co`: carbon monoxide in parts per million (integer) * **timestamp**: Unix timestamp in *second* precision The following line protocol sample represents data collected hourly beginning at**2022-01-01T08:00:00Z (UTC)** until **2022-01-01T20:00:00Z (UTC)**.*These timestamps are dynamic and can be updated by clicking the icon in the bottom right corner.* ##### Home sensor data line protocol ```text home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200 ``` ## Write data using the CLI To quickly get started writing data, use the[`influxdb3 write` command](/influxdb3/enterprise/reference/cli/influxdb3/write/). Include the following: * `--database` option that identifies the target database * `--token` option that specifies the token to use *(unless the `INFLUXDB3_AUTH_TOKEN`environment variable is already set)* * Quoted line protocol data via standard input (stdin) ```bash influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --precision s \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200' ``` In the code samples, replace the following placeholders with your values: * `DATABASE_NAME`: the name of the [database](/influxdb3/enterprise/admin/databases/) to write to * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to write to the specified database ### Write data from a file To write line protocol you have saved to a file, pass the `--file` option–for example, save the[sample line protocol](#home-sensor-data-line-protocol) to a file named `sensor_data`and then enter the following command: ``` influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --precision s \ --accept-partial \ --file path/to/sensor_data ``` Replace the following placeholders with your values: * `DATABASE_NAME`: the name of the [database](/influxdb3/enterprise/admin/databases/) to write to. * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to write to the specified database ## Other tools for writing data There are many ways to write data to your InfluxDB 3 Enterprise database, including: * [InfluxDB HTTP API](/influxdb3/enterprise/write-data/http-api/): Recommended for batching and higher-volume write workloads. * [InfluxDB client libraries](/influxdb3/enterprise/write-data/client-libraries/): Client libraries that integrate with your code to construct data as time series points and write the data as line protocol to your InfluxDB 3 Enterprise database. * [Telegraf](/telegraf/v1/): A data collection agent with over 300 plugins for collecting, processing, and writing data. For more information, see [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/). [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/)[Query data](/influxdb3/enterprise/get-started/query/) #### Related * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/) * [Line protocol reference](/influxdb3/enterprise/reference/line-protocol/) --- # Install InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/install/ * [System Requirements](#system-requirements) * [Install](#install) * [Quick install for Linux and macOS](#quick-install-for-linux-and-macos) * [Download and install the latest build artifacts](#download-and-install-the-latest-build-artifacts) * [Pull the Docker image](#pull-the-docker-image) * [Linux DEB or RPM](#linux-deb-or-rpm) * [TOML configuration (Linux)](#toml-configuration-linux) * [Run as a system service (Linux)](#run-as-a-system-service-linux) * [Verify the installation](#verify-the-installation) > [!Note] > #### Upgrading from InfluxDB 3 Core? > > If you’re currently running InfluxDB 3 Core and want to upgrade to InfluxDB 3 Enterprise, > see [Upgrade from Core](/influxdb3/enterprise/admin/upgrade-from-core/)for step-by-step instructions. > [!Note] > #### Multi-node cluster setup > > For information about setting up a multi-node InfluxDB 3 Enterprise cluster, > see [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/) in the Get started guide. > > For steps to upgrade an existing InfluxDB 3 Enterprise cluster, > see [Upgrade InfluxDB](/influxdb3/enterprise/admin/upgrade/). ## System Requirements #### Operating system InfluxDB 3 Enterprise runs on **Linux**, **macOS**, and **Windows**. #### Object storage A key feature of InfluxDB 3 is its use of object storage to store time series data in Apache Parquet format. You can choose to store these files on your local file system. Performance on your local filesystem will likely be better, but object storage has the advantage of not running out of space and being accessible by other systems over the network. InfluxDB 3 Enterprise natively supports Amazon S3, Azure Blob Storage, and Google Cloud Storage. You can also use many local object storage implementations that provide an S3-compatible API, such as [Minio](https://min.io/). ## Install InfluxDB 3 Enterprise runs on **Linux**, **macOS**, and **Windows**. Choose one of the following methods to install InfluxDB 3 Enterprise: * [Quick install for Linux and macOS](#quick-install-for-linux-and-macos) * [Download and install the latest build artifacts](#download-and-install-the-latest-build-artifacts) * [Pull the Docker image](#pull-the-docker-image) * [Linux DEB or RPM](#linux-deb-or-rpm) ### Quick install for Linux and macOS To install InfluxDB 3 Enterprise on **Linux** or **macOS**, download and run the quick installer script for InfluxDB 3 Enterprise–for example, using [`curl`](https://curl.se/)to download the script: ```bash curl -O https://www.influxdata.com/d/install_influxdb3.sh \ && sh install_influxdb3.sh enterprise ``` > [!Note] > The quick installer script is updated with each InfluxDB 3 Enterprise release, > so it always installs the latest version. > [!Important] > #### Production deployment > > For production deployments, use [Linux DEB or RPM](#linux-deb-or-rpm)for built-in systemd sandboxing, or [Docker](#pull-the-docker-image) with your own > container security configuration. > > For detailed security options, see [Manage security](/influxdb3/enterprise/admin/security/). ### Download and install the latest build artifacts You can also download and install InfluxDB 3 Enterprise build artifacts directly: [](#linux-binaries) Linux binaries * [Linux | AMD64 (x86\_64) | GNU](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_linux_amd64.tar.gz)•[sha256](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_linux_amd64.tar.gz.sha256) * [Linux | ARM64 (AArch64) | GNU](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_linux_arm64.tar.gz)•[sha256](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_linux_arm64.tar.gz.sha256) [](#macos-binaries) macOS binaries * [macOS | Silicon (ARM64)](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_darwin_arm64.tar.gz)•[sha256](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2_darwin_arm64.tar.gz.sha256) > [!Note] > macOS Intel builds are coming soon. [](#windows-binaries) Windows binaries * [Windows (AMD64, x86\_64) binary](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2-windows_amd64.zip)•[sha256](https://dl.influxdata.com/influxdb/releases/influxdb3-enterprise-3.12.2-windows_amd64.zip.sha256) ### Pull the Docker image Run the following command to pull the [`influxdb:3-enterprise` image](https://hub.docker.com/_/influxdb/tags?tag=3-enterprise&name=3-enterprise), available for x86\_64 (AMD64) and ARM64 architectures: ```bash docker pull influxdb:3-enterprise ``` Docker automatically pulls the appropriate image for your system architecture. [](#pull-for-a-specific-system-architecture) Pull for a specific system architecture To specify the system architecture, use platform-specific tags–for example: ```bash # For x86_64/AMD64 docker pull \ --platform linux/amd64 \ influxdb:3-enterprise ``` ```bash # For ARM64 docker pull \ --platform linux/arm64 \ influxdb:3-enterprise ``` [](#pin-to-a-specific-version) Pin to a specific version The `3-enterprise` tag always points to the latest 3.x release. To keep a deployment on a fixed version, pull a version-specific tag instead: ```bash # Pin to a specific patch release docker pull influxdb:3.12.2-enterprise # Pin to the latest patch in a minor series--for example, 3.10 docker pull influxdb:3.10-enterprise ``` To browse recent InfluxDB 3 image tags (Core and Enterprise), newest first, see[`influxdb` tags on Docker Hub](https://hub.docker.com/_/influxdb/tags?name=3.&ordering=last_updated). [](#pull-from-an-alternative-registry-quay) Pull from an alternative registry (Quay) InfluxDB 3 Enterprise images are also published to[Quay](https://quay.io/repository/influxdb/influxdb3-enterprise?tab=tags), tagged by release commit. For example, to pull the 3.12.0 image: ```bash docker pull quay.io/influxdb/influxdb3-enterprise:7dd361197ee2834eaa56089b67ffe59ad02e349a ``` ### Linux DEB or RPM When installed via DEB or RPM on a `systemd`-enabled system, InfluxDB 3 Enterprise runs in a sandboxed environment. The included `systemd` unit file configures the environment to provide security isolation for typical deployments. For more information, see [Manage security](/influxdb3/enterprise/admin/security/). > [!Note] > DEB and RPM installation is **recommended for non-Docker production deployments** due to built-in `systemd` sandboxing. [](#deb-based-systems) DEB-based systems Use `apt-get` to install InfluxDB 3 Enterprise from the InfluxData repository: ```bash curl --silent --location -O https://repos.influxdata.com/influxdata-archive.key gpg --show-keys --with-fingerprint --with-colons ./influxdata-archive.key 2>&1 \ | grep -q '^fpr::*24C975CBA61A024EE1B631787C3D57159FC2F927:$' \ && cat influxdata-archive.key \ | gpg --dearmor \ | sudo tee /usr/share/keyrings/influxdata-archive.gpg > /dev/null \ && echo 'deb [signed-by=/usr/share/keyrings/influxdata-archive.gpg] https://repos.influxdata.com/debian stable main' \ | sudo tee /etc/apt/sources.list.d/influxdata.list sudo apt-get update && sudo apt-get install influxdb3-enterprise ``` [](#rpm-based-systems) RPM-based systems Use `yum` to install InfluxDB 3 Enterprise from the InfluxData repository: ```bash curl --silent --location -O https://repos.influxdata.com/influxdata-archive.key test -d /usr/share/influxdata-archive-keyring/keyrings || sudo mkdir -p /usr/share/influxdata-archive-keyring/keyrings gpg --show-keys --with-fingerprint --with-colons ./influxdata-archive.key 2>&1 \ | grep -q '^fpr::*24C975CBA61A024EE1B631787C3D57159FC2F927:$' \ && sudo cp ./influxdata-archive.key /usr/share/influxdata-archive-keyring/keyrings/influxdata-archive.asc \ && cat < [!Important] > #### License required > > InfluxDB 3 Enterprise requires an active license to start. > See how to [Activate a license](/influxdb3/enterprise/admin/license/#activate-a-license). #### Run as a system service (Linux) InfluxDB 3 Enterprise DEB and RPM installs include service files for running as a managed system service on Linux: * **systemd**: For modern Linux distributions * **SysV init**: For legacy system compatibility ##### Run using systemd On `systemd` systems, the `influxdb3-enterprise` unit file is`enabled` on install, but the unit is not started in order to allow configuration. ###### Start, stop, and restart > [!Note] > The following examples use `sudo` for systems that require elevated privileges. > On some systems (such as Amazon Linux or other RHEL-based distributions where you may already be running as root), you can omit `sudo` from the commands. ```bash # Start the service sudo systemctl start influxdb3-enterprise # Stop the service sudo systemctl stop influxdb3-enterprise # Restart the service (use after configuration changes) sudo systemctl restart influxdb3-enterprise ``` ###### Check status and logs ```bash # Check status (sudo to ensure full journal output) sudo systemctl status influxdb3-enterprise # Quick state checks (no sudo needed) systemctl is-enabled influxdb3-enterprise systemctl is-active influxdb3-enterprise # Recent logs sudo journalctl --unit influxdb3-enterprise -n 200 --no-pager # Follow logs sudo journalctl --unit influxdb3-enterprise -f ``` ###### Inspect the packaged unit The packaged unit configures security sandboxing for typical deployments (see [Manage security](/influxdb3/enterprise/admin/security/)). To inspect the packaged unit and its resolved properties: ```bash # Show the unit file systemctl cat influxdb3-enterprise # Show all resolved properties (paths, environment, sandboxing options) systemctl show influxdb3-enterprise ``` ###### Apply configuration changes Edit the TOML configuration file and restart the service to apply changes: ```bash sudoedit /etc/influxdb3/influxdb3-enterprise.conf sudo systemctl restart influxdb3-enterprise sudo systemctl status influxdb3-enterprise sudo journalctl --unit influxdb3-enterprise -n 100 --no-pager ``` `influxdb3 serve` does not support configuration reload; a restart is required after editing the TOML file or changing environment variables. The TOML file is read by the systemd launcher and converted to`INFLUXDB3_*` environment variables before `influxdb3 serve` runs; CLI flags still override values from the TOML file. For details, see[TOML configuration files](/influxdb3/enterprise/reference/config-options/#toml-configuration-files). ##### Run using SysV On SysV init systems, `influxdb3-enterprise` is disabled on install and can be enabled by adjusting `/etc/default/influxdb3-enterprise` to contain `ENABLED=yes`. To start the database, enter the following commands: ```bash # Start the database /etc/init.d/influxdb3-enterprise start # View status /etc/init.d/influxdb3-enterprise status # View logs tail -f /var/lib/influxdb3/influxdb3-enterprise.log ``` ### Verify the installation After installing InfluxDB 3 Enterprise, enter the following command to verify that it installed successfully: ```bash influxdb3 --version ``` If your system can’t locate `influxdb3` following a [quick install](#quick-install-for-linux-and-macos), `source` the configuration file (for example, `.bashrc`, `.zshrc`) for your shell–for example: ```zsh source ~/.zshrc ``` > [!Note] > For information about setting up a multi-node InfluxDB 3 Enterprise cluster, > see [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/) in the Get started guide. [Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/) #### Related * [Upgrade InfluxDB 3 Enterprise](/influxdb3/enterprise/admin/upgrade/) [install](/influxdb3/enterprise/tags/install/) --- # Deploy InfluxDB 3 Enterprise on Kubernetes Source: https://docs.influxdata.com/influxdb3/enterprise/install/kubernetes/ Use the official InfluxDB 3 Enterprise Helm chart to deploy InfluxDB 3 Enterprise on Kubernetes. The Helm chart packages recommended deployment patterns into a single chart, eliminating the need to maintain custom manifests or configure deployments from scratch. > [!Note] > #### Beta status > > The InfluxDB 3 Enterprise Helm chart is currently in **beta**. > During this phase, InfluxData is refining defaults and ensuring upgrades > behave as operators expect.[Find support for InfluxDB 3 Enterprise](#find-support) or[report a Helm chart issue](#report-an-issue-with-helm-charts). * [Key features](#key-features) * [Prerequisites](#prerequisites) * [Install the Helm chart](#install-the-helm-chart) * [Configuration options](#configuration-options) * [Object storage](#object-storage) * [Cluster configuration](#cluster-configuration) * [Ingress](#ingress) * [Upgrade the deployment](#upgrade-the-deployment) * [Uninstall](#uninstall) * [Troubleshooting](#troubleshooting) * [Report an issue with Helm charts](#report-an-issue-with-helm-charts) * [Find support](#find-support) ## Key features * **Simplified deployment**: Deploy InfluxDB 3 Enterprise with a single Helm chart instead of maintaining custom Kubernetes manifests * **Production patterns**: Includes recommended configurations for object storage, cluster settings, and environment-specific overrides * **Standard Helm workflows**: Use familiar Helm mechanisms for installs, upgrades, and rollouts * **Environment overrides**: Customize deployments for different environments (development, staging, production) ## Prerequisites Before deploying InfluxDB 3 Enterprise on Kubernetes, ensure you have: * A Kubernetes cluster (version 1.25 or later) * [Helm](https://helm.sh/docs/intro/install/) installed (version 3.0 or later) * [`kubectl`](https://kubernetes.io/docs/tasks/tools/) configured to access your cluster * Object storage configured (Amazon S3, Azure Blob Storage, Google Cloud Storage, or S3-compatible storage such as MinIO) ### Licensing InfluxDB 3 Enterprise requires a license to run. For more information about license types, activation, and management, see[Manage your InfluxDB 3 Enterprise license](/influxdb3/enterprise/admin/license/). ## Install the Helm chart 1. Add the InfluxData Helm repository: ``` helm repo add influxdata https://helm.influxdata.com/ ``` 2. Update your Helm repositories: ``` helm repo update ``` 3. Create a namespace for InfluxDB: ``` kubectl create namespace influxdb ``` 4. Create a `values.yaml` file with your configuration. See [Configuration options](#configuration-options) for available settings. 5. Install the chart: ``` helm install influxdb3-enterprise influxdata/influxdb3-enterprise \ --namespace influxdb \ --values values.yaml ``` 6. Verify the deployment: ``` kubectl get pods --namespace influxdb ``` ## Configuration options The Helm chart supports common production configurations through the `values.yaml` file. Download the default values file as a starting point: ```bash curl -O https://raw.githubusercontent.com/influxdata/helm-charts/master/charts/influxdb3-enterprise/values.yaml ``` ### Object storage Configure object storage for Parquet file storage. InfluxDB 3 Enterprise supports Amazon S3, Azure Blob Storage, Google Cloud Storage, and S3-compatible storage. [Amazon S3](#amazon-s3)[Azure Blob Storage](#azure-blob-storage)[Google Cloud Storage](#google-cloud-storage)[S3-compatible (MinIO)](#s3-compatible-minio) Configure Amazon S3 as your object store: ```yaml objectStore: type: s3 bucket: S3_BUCKET region: S3_REGION accessKey: S3_ACCESS_KEY secretKey: S3_SECRET_KEY ``` Replace the following: * `S3_BUCKET`: your S3 bucket name * `S3_REGION`: AWS region (for example, `us-east-1`) * `S3_ACCESS_KEY`: AWS access key ID * `S3_SECRET_KEY`: AWS secret access key Configure Azure Blob Storage as your object store: ```yaml objectStore: type: azure bucket: AZURE_CONTAINER account: AZURE_ACCOUNT accessKey: AZURE_ACCESS_KEY ``` Replace the following: * `AZURE_CONTAINER`: your Azure container name * `AZURE_ACCOUNT`: Azure storage account name * `AZURE_ACCESS_KEY`: Azure storage access key Configure Google Cloud Storage as your object store: ```yaml objectStore: type: gcs bucket: GCS_BUCKET serviceAccountSecret: name: gcs-credentials key: credentials.json ``` Replace `GCS_BUCKET`with your Google Cloud Storage bucket name. Create a Kubernetes secret with your service account credentials: ```bash kubectl create secret generic gcs-credentials \ --from-file=credentials.json=/path/to/service-account.json \ --namespace influxdb ``` Configure S3-compatible storage (such as MinIO) as your object store: ```yaml objectStore: type: s3 bucket: MINIO_BUCKET endpoint: MINIO_ENDPOINT accessKey: MINIO_ACCESS_KEY secretKey: MINIO_SECRET_KEY forcePathStyle: true ``` Replace the following: * `MINIO_BUCKET`: your MinIO bucket name * `MINIO_ENDPOINT`: MinIO endpoint URL (for example, `http://minio.minio.svc:9000`) * `MINIO_ACCESS_KEY`: MinIO access key * `MINIO_SECRET_KEY`: MinIO secret key ### Cluster configuration Configure cluster-level settings: ```yaml # Cluster name clusterName: my-influxdb-cluster # Resource limits and requests resources: limits: cpu: "4" memory: 8Gi requests: cpu: "2" memory: 4Gi # Persistent storage for Write-Ahead Log (WAL) persistence: enabled: true size: 10Gi storageClass: standard ``` ### Ingress Configure ingress to expose the InfluxDB API: ```yaml ingress: enabled: true hosts: - INFLUXDB_HOST tls: - secretName: influxdb-tls hosts: - INFLUXDB_HOST ``` Replace `INFLUXDB_HOST`with your InfluxDB hostname (for example, `influxdb.example.com`). ## Upgrade the deployment To upgrade your deployment after modifying `values.yaml`: ```bash helm upgrade influxdb3-enterprise influxdata/influxdb3-enterprise \ --namespace influxdb \ --values values.yaml ``` To upgrade to a new chart version: ```bash helm repo update helm upgrade influxdb3-enterprise influxdata/influxdb3-enterprise \ --namespace influxdb \ --values values.yaml ``` ## Uninstall To remove the InfluxDB deployment: ```bash helm uninstall influxdb3-enterprise --namespace influxdb ``` > [!Warning] > Uninstalling the Helm release does not delete persistent volumes or > data in object storage. > To remove *all* data, manually delete the persistent volume claims > and object storage bucket. ## Troubleshooting ### View pod logs To view logs from the InfluxDB pod: ```bash kubectl logs -l app.kubernetes.io/name=influxdb3-enterprise \ --namespace influxdb ``` ### Check pod status To check the status of InfluxDB pods: ```bash kubectl get pods --namespace influxdb -o wide ``` ### Describe pod for events To see detailed information and events for a pod: ```bash kubectl describe pod POD_NAME --namespace influxdb ``` Replace `POD_NAME`with the name of the pod (from `kubectl get pods`). ## Report an issue with Helm charts For Helm chart issues or feedback, see the Helm charts repository [issues page](https://github.com/influxdata/helm-charts/issues). ## Find support The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find community support for InfluxDB 3 Enterprise. Customers with a [commercial license](/influxdb3/enterprise/admin/license/) can contact [InfluxData Support](https://support.influxdata.com). [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/)[Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/) #### Related * [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) * [Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/) * [Configure object storage](/influxdb3/enterprise/admin/object-storage/) * [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/) * [InfluxDB 3 Enterprise Helm Chart (GitHub)](https://github.com/influxdata/helm-charts/tree/master/charts/influxdb3-enterprise) [install](/influxdb3/enterprise/tags/install/)[kubernetes](/influxdb3/enterprise/tags/kubernetes/)[helm](/influxdb3/enterprise/tags/helm/) --- # Processing engine and Python plugins Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/ Use the Processing Engine in InfluxDB 3 Enterprise to extend your database with custom Python code. Trigger your code on write, on a schedule, or on demand to automate workflows, transform data, and create API endpoints. ## What is the Processing Engine? The Processing Engine is an embedded Python virtual machine that runs inside your InfluxDB 3 Enterprise database. You configure *triggers* to run your Python *plugin* code in response to: * **Data writes** - Process and transform data as it enters the database * **Scheduled events** - Run code at defined intervals or specific times * **HTTP requests** - Expose custom API endpoints that execute your code You can use the Processing Engine’s in-memory cache to manage state between executions and build stateful applications directly in your database. This guide walks you through setting up the Processing Engine, creating your first plugin, and configuring triggers that execute your code on specific events. ## Before you begin Ensure you have: * A working InfluxDB 3 Enterprise instance * Access to command line * Python installed if you’re writing your own plugin * Basic knowledge of the InfluxDB CLI Once you have all the prerequisites in place, follow these steps to implement the Processing Engine for your data automation needs. * [Set up the Processing Engine](#set-up-the-processing-engine) * [Add a Processing Engine plugin](#add-a-processing-engine-plugin) * [Upload plugins from local machine](#upload-plugins-from-local-machine) * [Update existing plugins](#update-existing-plugins) * [View loaded plugins](#view-loaded-plugins) * [Create a trigger](#create-a-trigger) * [Manage plugin dependencies](#manage-plugin-dependencies) * [Plugin security](#plugin-security) * [Run plugins in a cluster](#run-plugins-in-a-cluster) ## Set up the Processing Engine The Processing Engine activates when `--plugin-dir` or `INFLUXDB3_PLUGIN_DIR` is configured. In a cluster, this automatically adds `process` mode to the node. We recommend running the Processing Engine on a standalone process node, see [Run plugins in a cluster](#run-plugins-in-a-cluster). ### Default behavior by deployment type | Deployment |Default state| Configuration | |----------------|-------------|-----------------------------------------| | Docker images | **Enabled** | `INFLUXDB3_PLUGIN_DIR=/plugins` | |DEB/RPM packages| **Enabled** |`plugin-dir="/var/lib/influxdb3/plugins"`| | Binary/source | Disabled | No `plugin-dir` configured | If you installed InfluxDB 3 Enterprise using Docker or a DEB/RPM package, the Processing Engine is already enabled—skip to [Add a Processing Engine plugin](#add-a-processing-engine-plugin). To disable the Processing Engine, see [Enable and disable the Processing Engine](/influxdb3/enterprise/reference/processing-engine/#enable-and-disable-the-processing-engine). ### Enable the Processing Engine manually To activate the Processing Engine when running from a binary or source build, start your InfluxDB 3 Enterprise server with the `--plugin-dir` flag. This flag tells InfluxDB where to load your plugin files. > [!Important] > #### Keep the influxdb3 binary with its python directory > > The influxdb3 binary requires the adjacent `python/` directory to function. > If you manually extract from tar.gz, keep them in the same parent directory: > > ``` > your-install-location/ > ├── influxdb3 > └── python/ > ``` > > Add the parent directory to your PATH; do not move the binary out of this directory. ```bash influxdb3 serve \ --NODE_ID \ --object-store OBJECT_STORE_TYPE \ --plugin-dir PLUGIN_DIR ``` In the example above, replace the following: * `NODE_ID`: Unique identifier for your instance * `OBJECT_STORE_TYPE`: Type of object store (for example, file or s3) * `PLUGIN_DIR`: Absolute path to the directory where plugin files are stored. Store all plugin files in this directory or its subdirectories. > [!Note] > #### Use custom plugin repositories > > By default, plugins referenced with the `gh:` prefix are fetched from the official[influxdata/influxdb3\_plugins](https://github.com/influxdata/influxdb3_plugins) repository. > To use a custom repository, add the `--plugin-repo` flag when starting the server. > See [Use a custom plugin repository](#option-3-use-a-custom-plugin-repository) for details. ### Configure a cluster In a cluster, run the Processing Engine on a standalone process node. See [Run plugins in a cluster](#run-plugins-in-a-cluster). ## Add a Processing Engine plugin A plugin is a Python script that defines a function with a trigger-compatible (*trigger spec*) signature. When the specified event occurs, InfluxDB runs the plugin. ### Choose a plugin strategy You have two main options for adding plugins to your InfluxDB instance: * [Use example plugins](#use-example-plugins) - Get started with prebuilt plugins * [Create a custom plugin](#create-a-custom-plugin) - Build your own for specialized use cases ### Use example plugins InfluxData maintains a repository of official and community plugins that you can use immediately in your Processing Engine setup. Browse the [plugin library](/influxdb3/enterprise/plugins/library/) to find examples and InfluxData official plugins for: * **Data transformation**: Process and transform incoming data * **Alerting**: Send notifications based on data thresholds * **Aggregation**: Calculate statistics on time series data * **Integration**: Connect to external services and APIs * **System monitoring**: Track resource usage and health metrics For community contributions, see the [influxdb3\_plugins repository](https://github.com/influxdata/influxdb3_plugins) on GitHub. #### Add example plugins You have two options for using plugins from the repository: ##### Option 1: Copy plugins locally Clone the `influxdata/influxdb3_plugins` repository and copy plugins to your configured plugin directory: ``` # Clone the repository git clone https://github.com/influxdata/influxdb3_plugins.git # Copy a plugin to your configured plugin directory cp influxdb3_plugins/influxdata/system_metrics/system_metrics.py /path/to/plugins/ ``` ##### Option 2: Reference plugins directly from GitHub Skip downloading plugins by referencing them directly from GitHub using the `gh:` prefix: ```bash # Create a trigger using a plugin from GitHub influxdb3 create trigger \ --trigger-spec "every:1m" \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --database my_database \ system_metrics ``` This approach: * Ensures you’re using the latest version * Simplifies updates and maintenance * Reduces local storage requirements **Syntax:** ``` gh: ``` The path after `gh:` is appended to the configured `--plugin-repo`. > [!Note] > #### The gh: prefix isn’t GitHub-specific > > Despite the name, `gh:` doesn’t require GitHub or a Git repository. `--plugin-repo` accepts any HTTP/HTTPS URL that serves raw plugin files–for example, an internal static file host or object storage endpoint. `gh:` just tells InfluxDB 3 Enterprise to fetch the plugin remotely from that URL instead of reading it from the local `--plugin-dir`. By default, `gh:`-prefixed plugins resolve against the official [`influxdata/influxdb3_plugins`](https://github.com/influxdata/influxdb3_plugins) repository at `https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/`. For example, `gh:examples/wal_plugin/wal_plugin.py` resolves to: ``` https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/examples/wal_plugin/wal_plugin.py ``` **How `gh:` plugin resolution works:** 1. InfluxDB 3 Enterprise detects the `gh:` prefix in the plugin path. 2. It strips the prefix and appends the remaining path to the configured plugin repository URL. 3. An HTTP `GET` request fetches the plugin source code. 4. If the fetch succeeds (HTTP 2xx), InfluxDB 3 Enterprise validates the plugin and creates the trigger. 5. If the fetch fails, the command returns an error with the HTTP status code and URL–for example: ``` error fetching plugin from repository: 404 Not Found https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/not_found.py ``` InfluxDB 3 Enterprise fetches the plugin at trigger creation time (to validate it), and again each time the trigger starts–for example, on server startup or when you re-enable a disabled trigger. Unlike local plugins, GitHub plugins aren’t automatically reloaded when the source changes–disable and re-enable the trigger to fetch updates. Only single-file plugins are supported through the `gh:` prefix; multi-file plugin directories must be uploaded locally (see [Upload plugins from local machine](#upload-plugins-from-local-machine)). ##### Option 3: Use a custom plugin repository For organizations that maintain their own plugin repositories or need to use private/internal plugins, configure a custom plugin repository URL: ```bash # Start the server with a custom plugin repository influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins \ --plugin-repo "https://internal.company.com/influxdb-plugins/" ``` Then reference plugins from your custom repository using the `gh:` prefix: ```bash # Fetches from: https://internal.company.com/influxdb-plugins/myorg/custom_plugin.py influxdb3 create trigger \ --trigger-spec "every:5m" \ --path "gh:myorg/custom_plugin.py" \ --database my_database \ custom_trigger ``` **Use cases for custom repositories:** * **Private plugins**: Host proprietary plugins not suitable for public repositories * **Air-gapped environments**: Use internal mirrors when external internet access is restricted * **Development and staging**: Test plugins from development branches before production deployment * **Compliance requirements**: Meet data governance policies requiring internal hosting The `--plugin-repo` option accepts any HTTP/HTTPS URL that serves raw plugin files. You can also set it with the `INFLUXDB3_PLUGIN_REPO` environment variable. See the [plugin-repo configuration option](/influxdb3/enterprise/reference/config-options/#plugin-repo) for more details. Plugins have various functions such as: * Receive plugin-specific arguments (such as written data, call time, or an HTTP request) * Access keyword arguments (as `args`) passed from *trigger arguments* configurations * Access the `influxdb3_local` shared API to write data, query data, and managing state between executions For more information about available functions, arguments, and how plugins interact with InfluxDB, see how to [Extend plugins](/influxdb3/enterprise/extend-plugin/). ### Create a custom plugin To build custom functionality, you can create your own Processing Engine plugin. #### Prerequisites Before you begin, make sure: * The Processing Engine is enabled on your InfluxDB 3 Enterprise instance. * You’ve configured the `--plugin-dir` where plugin files are stored. * You have access to that plugin directory. #### Steps to create a plugin: * [Choose your plugin type](#choose-your-plugin-type) * [Create your plugin file](#create-your-plugin-file) * [Next Steps](#next-steps) #### Choose your plugin type Choose a plugin type based on your automation goals: | Plugin Type | Best For | |----------------|-------------------------------------------| | **Data write** | Processing data as it arrives | | **Scheduled** |Running code at specific intervals or times| |**HTTP request**| Running code on demand via API endpoints | #### Create your plugin file Plugins now support both single-file and multifile architectures: **Single-file plugins:** * Create a `.py` file in your plugins directory * Add the appropriate function signature based on your chosen plugin type * Write your processing logic inside the function **Multifile plugins:** * Create a directory in your plugins directory * Add an `__init__.py` file as the entry point (required) * Organize supporting modules in additional `.py` files * Import and use modules within your plugin code ##### Example multifile plugin structure ``` my_plugin/ ├── __init__.py # Required - entry point with trigger function ├── utils.py # Supporting module ├── processors.py # Data processing functions └── config.py # Configuration helpers ``` The `__init__.py` file must contain your trigger function: ```python # my_plugin/__init__.py from .processors import process_data from .config import get_settings def process_writes(influxdb3_local, table_batches, args=None): settings = get_settings() for table_batch in table_batches: process_data(influxdb3_local, table_batch, settings) ``` Supporting modules can contain helper functions: ```python # my_plugin/processors.py def process_data(influxdb3_local, table_batch, settings): # Processing logic here pass ``` After writing your plugin, [create a trigger](#create-a-trigger) to connect it to a database event and define when it runs. #### Create a data write plugin Use a data write plugin to process data as it’s written to the database. These plugins use [`table:` or `all_tables:`](#trigger-on-data-writes) trigger specifications. Ideal use cases include: * Data transformation and enrichment * Alerting on incoming values * Creating derived metrics ``` def process_writes(influxdb3_local, table_batches, args=None): # Process data as it's written to the database for table_batch in table_batches: table_name = table_batch["table_name"] rows = table_batch["rows"] # Log information about the write influxdb3_local.info(f"Processing {len(rows)} rows from {table_name}") # Write derived data back to the database line = LineBuilder("processed_data") line.tag("source_table", table_name) line.int64_field("row_count", len(rows)) influxdb3_local.write(line) ``` #### Create a scheduled plugin Scheduled plugins run at defined intervals using [`every:` or `cron:`](#trigger-on-a-schedule) trigger specifications. Use them for: * Periodic data aggregation * Report generation * System health checks ``` def process_scheduled_call(influxdb3_local, call_time, args=None): # Run code on a schedule # Query recent data results = influxdb3_local.query("SELECT * FROM metrics WHERE time > now() - INTERVAL '1 hour'") # Process the results if results: influxdb3_local.info(f"Found {len(results)} recent metrics") else: influxdb3_local.warn("No recent metrics found") ``` #### Create an HTTP request plugin HTTP request plugins respond to API calls using [`request:`](#trigger-on-http-requests) trigger specifications. Use them for: * Creating custom API endpoints * Webhooks for external integrations * User interfaces for data interaction ``` def process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None): # Handle HTTP requests to a custom endpoint # Log the request parameters influxdb3_local.info(f"Received request with parameters: {query_parameters}") # Process the request body if request_body: import json data = json.loads(request_body) influxdb3_local.info(f"Request data: {data}") # Return a response (automatically converted to JSON) return {"status": "success", "message": "Request processed"} ``` #### Next steps After writing your plugin: * [Create a trigger](#create-a-trigger) to connect your plugin to database events * [Install any Python dependencies](#manage-plugin-dependencies) your plugin requires * Learn how to [extend plugins with the API](/influxdb3/enterprise/extend-plugin/) ### Upload plugins from local machine For local development and testing, you can upload plugin files directly from your machine when creating triggers. This eliminates the need to manually copy files to the server’s plugin directory. * [Upload a plugin using the influxdb3 CLI](#upload-a-plugin-using-the-influxdb3-cli) * [Upload a plugin using the HTTP API](#upload-a-plugin-using-the-http-api) #### Upload a plugin using the influxdb3 CLI Use the `--upload` flag with `--path` to transfer local files or directories: ```bash # Upload single-file plugin influxdb3 create trigger \ --trigger-spec "every:10s" \ --path "/local/path/to/plugin.py" \ --upload \ --database metrics \ my_trigger # Upload multifile plugin directory influxdb3 create trigger \ --trigger-spec "every:30s" \ --path "/local/path/to/plugin-dir" \ --upload \ --database metrics \ complex_trigger ``` For more information, see the [`influxdb3 create trigger` CLI reference](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/). #### Upload a plugin using the HTTP API To create a plugin file using the HTTP API, send a `POST` request to the `/api/v3/plugins/files` endpoint: ``` POST http://localhost:8181/api/v3/plugins/files ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your admin token * `Content-Type: application/json` * **Request body** (JSON): * `plugin_name` *(string, required)*: Name of the plugin file to create, for example, `plugin.py` * `content` *(string, required)*: Source code of the plugin file The endpoint doesn’t require an existing trigger. Create the file first, then create a trigger that references it. The endpoint rejects a request with `Content-Type: application/octet-stream` and returns HTTP status `415`. To build the JSON body from a local file, use `jq`: ```bash # Create a single-file plugin jq --null-input \ --arg plugin_name "plugin.py" \ --rawfile content "/local/path/to/plugin.py" \ '{plugin_name: $plugin_name, content: $content}' | curl --request POST "http://localhost:8181/api/v3/plugins/files" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data @- ``` Replace `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/admin) > [!Important] > #### Admin privileges required > > Plugin uploads require an admin token. This security measure prevents unauthorized code execution on the server. **When to use plugin upload:** * Local plugin development and testing * Deploying plugins without SSH access to the server * Rapid iteration on plugin code * Automating plugin deployment in CI/CD pipelines ### Update existing plugins Modify plugin code for running triggers without recreating them. This allows you to iterate on plugin development while preserving trigger configuration and history. * [Update a plugin using the influxdb3 CLI](#update-a-plugin-using-the-influxdb3-cli) * [Update a plugin using the HTTP API](#update-a-plugin-using-the-http-api) > [!Important] > Plugin updates through the CLI and HTTP API do not update triggers that reference plugins using the `gh:` prefix. > To fetch updated code for a `gh:` trigger, disable and re-enable the trigger with [`influxdb3 disable trigger`](/influxdb3/enterprise/reference/cli/influxdb3/disable/trigger/) and [`influxdb3 enable trigger`](/influxdb3/enterprise/reference/cli/influxdb3/enable/trigger/). #### Update a plugin using the influxdb3 CLI Use the `influxdb3 update trigger` command: ```bash # Update single-file plugin influxdb3 update trigger \ --database metrics \ --trigger-name my_trigger \ --path "/path/to/updated/plugin.py" # Update multifile plugin influxdb3 update trigger \ --database metrics \ --trigger-name complex_trigger \ --path "/path/to/updated/plugin-dir" ``` For complete reference, see [`influxdb3 update trigger`](/influxdb3/enterprise/reference/cli/influxdb3/update/trigger/). #### Update a plugin using the HTTP API To update a plugin file using the HTTP API, send a `PUT` request to the `/api/v3/plugins/files` endpoint: ``` PUT http://localhost:8181/api/v3/plugins/files ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your admin token * `Content-Type: application/json` * **Request body** (JSON): * `plugin_name` *(string, required)*: Name of the existing trigger whose plugin file you want to update * `content` *(string, required)*: Updated source code of the plugin file The server resolves the trigger’s plugin file from `plugin_name` and overwrites that file. If `plugin_name` doesn’t match a registered trigger, the request returns HTTP status `500`. To add a plugin file before you create a trigger, [create a plugin file](#upload-a-plugin-using-the-http-api) instead. ```bash # Update a plugin file jq --null-input \ --arg plugin_name "TRIGGER_NAME" \ --rawfile content "/path/to/updated/plugin.py" \ '{plugin_name: $plugin_name, content: $content}' | curl --request PUT "http://localhost:8181/api/v3/plugins/files" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data @- ``` Replace the following: * `TRIGGER_NAME`: the name of the trigger that uses the plugin file * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/admin) **The update operation:** * Replaces plugin files immediately * Preserves trigger configuration (spec, schedule, arguments) * Requires admin token for security * Works with both local paths and uploaded files ### View loaded plugins Monitor which plugins are loaded in your system for operational visibility and troubleshooting. **Option 1: Use the CLI command** ```bash # List all plugins influxdb3 show plugins --token $ADMIN_TOKEN # JSON format for programmatic access influxdb3 show plugins --format json --token $ADMIN_TOKEN ``` **Option 2: Query the system table** The `system.plugin_files` table in the `_internal` database provides detailed plugin file information: ```bash influxdb3 query \ -d _internal \ "SELECT * FROM system.plugin_files ORDER BY plugin_name" \ --token $ADMIN_TOKEN ``` **Available columns:** * `plugin_name` (String): Trigger name * `file_name` (String): Plugin file name * `file_path` (String): Full server path * `size_bytes` (Int64): File size * `last_modified` (Int64): Modification timestamp (milliseconds) **Example queries:** ```sql -- Find plugins by name SELECT * FROM system.plugin_files WHERE plugin_name = 'my_trigger'; -- Find large plugins SELECT plugin_name, size_bytes FROM system.plugin_files WHERE size_bytes > 10000; -- Check modification times SELECT plugin_name, file_name, last_modified FROM system.plugin_files ORDER BY last_modified DESC; ``` For more information, see the [`influxdb3 show plugins` reference](/influxdb3/enterprise/reference/cli/influxdb3/show/plugins/) and [Query system data](/influxdb3/enterprise/admin/query-system-data/#query-plugin-files). ## Create a trigger A trigger connects your plugin code to database events. When the specified event occurs, the processing engine executes your plugin. * [Understand trigger types](#understand-trigger-types) * [Create a trigger using the influxdb3 CLI](#create-a-trigger-using-the-influxdb3-cli) * [Create a trigger using the HTTP API](#create-a-trigger-using-the-http-api) * [Trigger specification examples](#trigger-specification-examples) ### Understand trigger types |Plugin Type | Trigger Specification | When Plugin Runs | |------------|-----------------------------------------|-------------------------------| | Data write | `table:` or `all_tables` |When data is written to tables | | Scheduled |`every:` or `cron:`| At specified time intervals | |HTTP request| `request:` |When HTTP requests are received| ### Create a trigger using the influxdb3 CLI Use the `influxdb3 create trigger` command with the appropriate trigger specification: ```bash influxdb3 create trigger \ --trigger-spec SPECIFICATION \ --path PLUGIN_FILE \ --database DATABASE_NAME \ TRIGGER_NAME ``` In the example above, replace the following: * `SPECIFICATION`: Trigger specification * `PLUGIN_FILE`: Plugin filename relative to your configured plugin directory * `DATABASE_NAME`: Name of the database * `TRIGGER_NAME`: Name of the new trigger > [!Note] > #### Plugin paths > > * For **single-file plugins**, provide just the `.py` filename to `--path` (for example, `test_plugin.py`). > * For **multi-file plugins**, provide the directory name containing `__init__.py`. > > When not using `--upload`, the server resolves paths relative to the configured `--plugin-dir`. > For details about multi-file plugin structure, see [Create your plugin file](#create-your-plugin-file). For complete reference, see [`influxdb3 create trigger`](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/). ### Create a trigger using the HTTP API To create a trigger using the HTTP API, send a `POST` request to the `/api/v3/configure/processing_engine_trigger` endpoint: ``` POST http://localhost:8181/api/v3/configure/processing_engine_trigger ``` Include the following in your request: * **Headers**: * `Authorization: Bearer` with your authentication token * `Content-Type: application/json` * **Request body**: JSON object with trigger configuration * `db` *(string, required)*: Database name * `trigger_name` *(string, required)*: Trigger name * `plugin_filename` *(string, required)*: Plugin filename relative to the plugin directory * `trigger_specification` *(string, required)*: When the plugin runs (see [trigger types](#understand-trigger-types)) * `trigger_settings` *(object, optional)*: Configuration for error handling and execution. If you omit `trigger_settings` or any of its fields, each omitted field uses its default. * `run_async` *(boolean)*: Whether to run asynchronously (default: `false`) * `error_behavior` *(string)*: How to handle errors: `log`, `retry`, or `disable` (default: `log`) * `disabled` *(boolean, optional)*: Whether the trigger is disabled (default: `false`) * `trigger_arguments` *(object, optional)*: Arguments passed to the plugin ```bash # Create a basic trigger curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "TRIGGER_NAME", "plugin_filename": "PLUGIN_FILE", "trigger_specification": "TRIGGER_SPEC", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` In the example above, replace the following: * `DATABASE_NAME`: Name of the database * `TRIGGER_NAME`: Name of the new trigger * `PLUGIN_FILE`: Plugin filename relative to your configured plugin directory * `TRIGGER_SPEC`: Trigger specification (see [examples](#trigger-specification-examples)) * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database ### Trigger specification examples The following examples demonstrate how to create triggers for different event types. #### Trigger on data writes [influxdb3 CLI](#influxdb3-cli)[HTTP API](#http-api) ```bash # Trigger on writes to a specific table # The plugin file must be in your configured plugin directory influxdb3 create trigger \ --trigger-spec "table:sensor_data" \ --path "process_sensors.py" \ --database my_database \ sensor_processor # Trigger on writes to all tables influxdb3 create trigger \ --trigger-spec "all_tables" \ --path "process_all_data.py" \ --database my_database \ all_data_processor ``` ```bash # Trigger on writes to a specific table curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "sensor_processor", "plugin_filename": "process_sensors.py", "trigger_specification": "table:sensor_data", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' # Trigger on writes to all tables curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "all_data_processor", "plugin_filename": "process_all_data.py", "trigger_specification": "all_tables", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database The trigger runs when the database flushes ingested data for the specified tables to the Write-Ahead Log (WAL) in the Object store (default is every second). The plugin receives the written data and table information. #### Trigger on data writes with table exclusion If you want to use a single trigger for all tables but exclude specific tables, you can use trigger arguments and your plugin code to filter out unwanted tables–for example: ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --path processor.py \ --trigger-spec "all_tables" \ --trigger-arguments "exclude_tables=temp_data,debug_info,system_logs" \ data_processor ``` Replace the following: * DATABASE\_NAME: the name of the database * AUTH\_TOKEN: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database Then, in your plugin: ```python # processor.py def on_write(self, database, table_name, batch): # Get excluded tables from trigger arguments excluded_tables = set(self.args.get('exclude_tables', '').split(',')) if table_name in excluded_tables: return # Process allowed tables self.process_data(database, table_name, batch) ``` ##### Recommendations * **Early return**: Check exclusions as early as possible in your plugin. * **Efficient lookups**: Use sets for O(1) lookup performance with large exclusion lists. * **Performance**: Log skipped tables for debugging but avoid excessive logging in production. * **Multiple triggers**: For few tables, consider creating separate table-specific triggers instead of filtering within plugin code. See HTTP API [Processing engine endpoints](/influxdb3/enterprise/api/v3/#tag/Processing-engine) for managing triggers. #### Trigger on a schedule [influxdb3 CLI](#influxdb3-cli-2)[HTTP API](#http-api-2) ```bash # Run every 5 minutes influxdb3 create trigger \ --trigger-spec "every:5m" \ --path "periodic_check.py" \ --database my_database \ regular_check # Run on a cron schedule (8am daily) # Supports extended cron format with seconds influxdb3 create trigger \ --trigger-spec "cron:0 0 8 * * *" \ --path "daily_report.py" \ --database my_database \ daily_report ``` ```bash # Run every 5 minutes curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "regular_check", "plugin_filename": "periodic_check.py", "trigger_specification": "every:5m", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' # Run on a cron schedule (8am daily) # Supports extended cron format with seconds curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "daily_report", "plugin_filename": "daily_report.py", "trigger_specification": "cron:0 0 8 * * *", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database The plugin receives the scheduled call time. #### Trigger on HTTP requests [influxdb3 CLI](#influxdb3-cli-3)[HTTP API](#http-api-3) ```bash # Create an endpoint at /api/v3/engine/webhook influxdb3 create trigger \ --trigger-spec "request:webhook" \ --path "webhook_handler.py" \ --database my_database \ webhook_processor ``` ```bash # Create an endpoint at /api/v3/engine/webhook curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "webhook_processor", "plugin_filename": "webhook_handler.py", "trigger_specification": "request:webhook", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database Access your endpoint at `/api/v3/engine/{REQUEST_PATH}` (in this example, `/api/v3/engine/webhook`). The trigger is enabled by default and runs when an HTTP request is received at the specified path. To run the plugin, send a `GET` or `POST` request to the endpoint–for example: ```bash curl http://localhost:8181/api/v3/engine/webhook ``` The plugin receives the HTTP request object with methods, headers, and body. To view triggers associated with a database, use the `influxdb3 show summary` command: ```bash influxdb3 show summary --database my_database --token AUTH_TOKEN ``` ### Pass arguments to plugins Use trigger arguments to pass configuration from a trigger to the plugin it runs. You can use this for: * Threshold values for monitoring * Connection properties for external services * Configuration settings for plugin behavior [influxdb3 CLI](#influxdb3-cli-4)[HTTP API](#http-api-4) ```bash influxdb3 create trigger \ --trigger-spec "every:1h" \ --path "threshold_check.py" \ --trigger-arguments threshold=90,notify_email=admin@example.com \ --database my_database \ threshold_monitor ``` ```bash curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "threshold_monitor", "plugin_filename": "threshold_check.py", "trigger_specification": "every:1h", "trigger_settings": { "run_async": false, "error_behavior": "log" }, "trigger_arguments": { "threshold": "90", "notify_email": "admin@example.com" }, "disabled": false }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database The arguments are passed to the plugin as a `Dict[str, str]` where the key is the argument name and the value is the argument value: ``` def process_scheduled_call(influxdb3_local, call_time, args=None): if args and "threshold" in args: threshold = float(args["threshold"]) email = args.get("notify_email", "default@example.com") # Use the arguments in your logic influxdb3_local.info(f"Checking threshold {threshold}, will notify {email}") ``` ### Control trigger execution By default, triggers run synchronously—each instance waits for previous instances to complete before executing. To allow multiple instances of the same trigger to run simultaneously, configure triggers to run asynchronously: [influxdb3 CLI](#influxdb3-cli-5)[HTTP API](#http-api-5) ```bash # Allow multiple trigger instances to run simultaneously influxdb3 create trigger \ --trigger-spec "table:metrics" \ --path "heavy_process.py" \ --run-asynchronous \ --database my_database \ async_processor ``` ```bash # Allow multiple trigger instances to run simultaneously curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "async_processor", "plugin_filename": "heavy_process.py", "trigger_specification": "table:metrics", "trigger_settings": { "run_async": true, "error_behavior": "log" }, "disabled": false }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database ### Configure error handling for a trigger To configure error handling behavior for a trigger, specify one of the following values: * `log` (default): Log all plugin errors to stdout and the `system.processing_engine_logs` table in the trigger’s database. * `retry`: Attempt to run the plugin again immediately after an error. * `disable`: Automatically disable the plugin when an error occurs (can be re-enabled later). [influxdb3 CLI](#influxdb3-cli-6)[HTTP API](#http-api-6) For more information, see how to [Query trigger logs](/influxdb3/enterprise/admin/query-system-data/#query-trigger-logs). ```bash # Automatically retry on error influxdb3 create trigger \ --trigger-spec "table:important_data" \ --path "critical_process.py" \ --error-behavior retry \ --database my_database \ critical_processor # Disable the trigger on error influxdb3 create trigger \ --trigger-spec "request:webhook" \ --path "webhook_handler.py" \ --error-behavior disable \ --database my_database \ auto_disable_processor ``` ```bash # Automatically retry on error curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "critical_processor", "plugin_filename": "critical_process.py", "trigger_specification": "table:important_data", "trigger_settings": { "run_async": false, "error_behavior": "retry" }, "disabled": false }' # Disable the trigger on error curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "trigger_name": "auto_disable_processor", "plugin_filename": "webhook_handler.py", "trigger_specification": "request:webhook", "trigger_settings": { "run_async": false, "error_behavior": "disable" }, "disabled": false }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with write permissions on the specified database ## Manage plugin dependencies Use the `influxdb3 install package` command to add third-party libraries (like `pandas`, `requests`, or `influxdb3-python`) to your plugin environment. This installs packages into the Processing Engine’s embedded Python environment to ensure compatibility with your InfluxDB instance. [influxdb3 CLI](#influxdb3-cli-7)[Docker](#docker)[HTTP API](#http-api-7) ```bash # Use the CLI to install a Python package influxdb3 install package pandas ``` ```bash # Use the CLI to install a Python package in a Docker container docker exec -it CONTAINER_NAME influxdb3 install package pandas ``` ```bash # Use the HTTP API to install Python packages curl -X POST "http://localhost:8181/api/v3/configure/plugin_environment/install_packages" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "packages": ["pandas", "requests", "numpy"] }' ``` Replace `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/admin) For complete reference, see [Install plugin packages](/influxdb3/enterprise/api/v3/#operation/PostInstallPluginPackages). These examples install the specified Python packages (for example, pandas) into the Processing Engine’s embedded virtual environment. * Use the CLI command when running InfluxDB directly on your system. * Use the Docker variant if you’re running InfluxDB in a containerized environment. * Use the HTTP API for programmatic package installation or CI/CD workflows. > [!Important] > #### Use bundled Python for plugins > > When you start the server with the `--plugin-dir` option, InfluxDB 3 creates a Python virtual environment (`/.venv`) for your plugins. > If you need to create a custom virtual environment, use the Python interpreter bundled with InfluxDB 3. Don’t use the system Python. > Creating a virtual environment with the system Python (for example, using `python -m venv`) can lead to runtime errors and plugin failures. > > For more information, see the [processing engine README](https://github.com/influxdata/influxdb/blob/main/README_processing_engine.md). InfluxDB creates a Python virtual environment in your plugins directory with the specified packages installed. ### Disable package installation for secure environments For air-gapped deployments or environments with strict security requirements, you can disable Python package installation while maintaining Processing Engine functionality. Start the server with `--package-manager disabled`: ```bash influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins \ --package-manager disabled ``` When package installation is disabled: * The Processing Engine continues to function normally for triggers * Plugin code executes without restrictions * Package installation commands are blocked * Pre-installed dependencies in the virtual environment remain available **Pre-install required dependencies:** Before disabling the package manager, install all required Python packages: ```bash # Install packages first influxdb3 install package pandas requests numpy # Then start with disabled package manager influxdb3 serve \ --plugin-dir ~/.plugins \ --package-manager disabled ``` **Use cases for disabled package management:** * Air-gapped environments without internet access * Compliance requirements prohibiting runtime package installation * Centrally managed dependency environments * Security policies requiring pre-approved packages only For more configuration options, see [–package-manager](/influxdb3/enterprise/reference/config-options/#package-manager). ## Plugin security The Processing Engine includes security features to protect your InfluxDB 3 Enterprise instance from unauthorized code execution and file system attacks. ### Plugin path validation All plugin file paths are validated to prevent directory traversal attacks. The system blocks: * **Relative paths with parent directory references** (`../`, `../../`) * **Absolute paths** (`/etc/passwd`, `/usr/bin/script.py`) * **Symlinks that escape the plugin directory** When creating or updating triggers, plugin paths must resolve within the configured `--plugin-dir`. **Example of blocked paths:** ```bash # These will be rejected influxdb3 create trigger \ --path "../../../etc/passwd" \ # Blocked: parent directory traversal ... influxdb3 create trigger \ --path "/tmp/malicious.py" \ # Blocked: absolute path ... ``` **Valid plugin paths:** ```bash # These are allowed influxdb3 create trigger \ --path "myapp/plugin.py" \ # Relative to plugin-dir ... influxdb3 create trigger \ --path "transforms/data.py" \ # Subdirectory in plugin-dir ... ``` ### Upload and update permissions Plugin upload and update operations require admin tokens to prevent unauthorized code deployment: * `--upload` flag requires admin privileges * `update trigger` command requires admin token * Standard resource tokens cannot upload or modify plugin code This security model ensures only administrators can introduce or modify executable code in your database. ### Best practices **For development:** * Use the `--upload` flag to deploy plugins during development * Test plugins in non-production environments first * Review plugin code before deployment **For production:** * Pre-deploy plugins to the server’s plugin directory via secure file transfer * Use custom plugin repositories for vetted, approved plugins * Disable package installation (`--package-manager disabled`) in locked-down environments * Audit plugin files using the [`system.plugin_files` table](#view-loaded-plugins) * Implement change control processes for plugin updates For more security configuration options, see [Configuration options](/influxdb3/enterprise/reference/config-options/). ## Run plugins in a cluster In a cluster, we recommend running the Processing Engine on a standalone process node: a node started with `--mode=process` and `--plugin-dir`. The process node doesn’t store writes or answer data queries. The process node starts each trigger type differently: * **WAL triggers** (`table:` or `all_tables`): The process node reads each ingest node’s WAL files from object storage and passes new rows to `process_writes()`. * **Schedule triggers** (`every:` or `cron:`): The process node calls `process_scheduled_call()` on the schedule. * **Request triggers** (`request:`): The process node serves `/api/v3/engine/` and calls `process_request()` for each request. For every trigger type, plugins reach data through other nodes over the internode protocol:`influxdb3_local.query()` runs on a query node, and `influxdb3_local.write()` and plugin log messages go to an ingest node. Start every ingest node and every query node with `--internode-bind-addr`. For the complete setup, see [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). ### Pin triggers to the process node Create each trigger on the process node, and set [`--node-spec`](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/#options) to the process node’s ID: ```bash influxdb3 create trigger \ --host http://PROCESS_HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --path schedule_rollup.py \ --trigger-spec "every:5s" \ --node-spec "nodes:PROCESS_NODE_ID" \ hourly_rollup ``` The default, `--node-spec all`, makes every node that runs the Processing Engine own the trigger. ### Route requests to the right node * Send `/api/v3/engine/` requests for request triggers to the process node. Other nodes return `404` or `405`. * Connect external query tools, such as Grafana, to query nodes, for example `https://query.example.com:8181`. * Send writes to ingest nodes. #### Related * [influxdb3 test wal\_plugin](/influxdb3/enterprise/reference/cli/influxdb3/test/wal_plugin/) * [influxdb3 create trigger](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/) [processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/) | Deployment | Default state | Configuration | | --- | --- | --- | | Deployment | Default state | Configuration | | Docker images | Enabled | INFLUXDB3_PLUGIN_DIR=/plugins | | DEB/RPM packages | Enabled | plugin-dir="/var/lib/influxdb3/plugins" | | Binary/source | Disabled | No plugin-dir configured | | Plugin Type | Best For | | --- | --- | | Plugin Type | Best For | | Data write | Processing data as it arrives | | Scheduled | Running code at specific intervals or times | | HTTP request | Running code on demand via API endpoints | | Plugin Type | Trigger Specification | When Plugin Runs | | --- | --- | --- | | Plugin Type | Trigger Specification | When Plugin Runs | | Data write | table: or all_tables | When data is written to tables | | Scheduled | every: or cron: | At specified time intervals | | HTTP request | request: | When HTTP requests are received | --- # Extend plugins with API features and state management Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/extend-plugin/ The Processing Engine includes a shared API that your plugins can use to interact with data, write new records in line protocol format, and maintain state between executions. These capabilities let you build plugins that transform, analyze, and respond to time series data as it flows through your database. The plugin API lets you: * [Write data](#write-data) * [Query data](#query-data) * [Log messages for monitoring and debugging](#log-messages-for-monitoring-and-debugging) * [Maintain state with the in-memory cache](#maintain-state-with-the-in-memory-cache) * [Store and retrieve cached data](#store-and-retrieve-cached-data) * [Use TTL appropriately](#use-ttl-appropriately) * [Share data across plugins](#share-data-across-plugins) * [Build a counter](#building-a-counter) * [Guidelines for in-memory caching](#guidelines-for-in-memory-caching) * [Consider cache limitations](#consider-cache-limitations) ## Get started with the shared API Each plugin automatically has access to the shared API through the `influxdb3_local` object. You don’t need to import any libraries. The API becomes available as soon as your plugin runs. If your plugin requires third-party Python packages (like `pandas`, `requests`, or `influxdb3-python`), see [Manage plugin dependencies](/influxdb3/enterprise/plugins/#manage-plugin-dependencies) for installation instructions. ## Write data To write data into your database, use the `LineBuilder` API to create line protocol data: ```python # Create a line protocol entry line = LineBuilder("weather") line.tag("location", "us-midwest") line.float64_field("temperature", 82.5) line.time_ns(1627680000000000000) # Write the data to the database influxdb3_local.write(line) ``` InfluxDB 3 buffers your writes while the plugin runs and flushes them when the plugin completes. [](#view-the-linebuilder-python-implementation) View the `LineBuilder` Python implementation ```python from typing import Optional from collections import OrderedDict class InfluxDBError(Exception): """Base exception for InfluxDB-related errors""" pass class InvalidMeasurementError(InfluxDBError): """Raised when measurement name is invalid""" pass class InvalidKeyError(InfluxDBError): """Raised when a tag or field key is invalid""" pass class InvalidLineError(InfluxDBError): """Raised when a line protocol string is invalid""" pass class LineBuilder: def __init__(self, measurement: str): if ' ' in measurement: raise InvalidMeasurementError("Measurement name cannot contain spaces") self.measurement = measurement self.tags: OrderedDict[str, str] = OrderedDict() self.fields: OrderedDict[str, str] = OrderedDict() self._timestamp_ns: Optional[int] = None def _validate_key(self, key: str, key_type: str) -> None: """Validate that a key does not contain spaces, commas, or equals signs.""" if not key: raise InvalidKeyError(f"{key_type} key cannot be empty") if ' ' in key: raise InvalidKeyError(f"{key_type} key '{key}' cannot contain spaces") if ',' in key: raise InvalidKeyError(f"{key_type} key '{key}' cannot contain commas") if '=' in key: raise InvalidKeyError(f"{key_type} key '{key}' cannot contain equals signs") def tag(self, key: str, value: str) -> 'LineBuilder': """Add a tag to the line protocol.""" self._validate_key(key, "tag") self.tags[key] = str(value) return self def uint64_field(self, key: str, value: int) -> 'LineBuilder': """Add an unsigned integer field to the line protocol.""" self._validate_key(key, "field") if value < 0: raise ValueError(f"uint64 field '{key}' cannot be negative") self.fields[key] = f"{value}u" return self def int64_field(self, key: str, value: int) -> 'LineBuilder': """Add an integer field to the line protocol.""" self._validate_key(key, "field") self.fields[key] = f"{value}i" return self def float64_field(self, key: str, value: float) -> 'LineBuilder': """Add a float field to the line protocol.""" self._validate_key(key, "field") # Check if value has no decimal component self.fields[key] = f"{int(value)}.0" if value % 1 == 0 else str(value) return self def string_field(self, key: str, value: str) -> 'LineBuilder': """Add a string field to the line protocol.""" self._validate_key(key, "field") # Escape quotes and backslashes in string values escaped_value = value.replace('"', '\\"').replace('\\', '\\\\') self.fields[key] = f'"{escaped_value}"' return self def bool_field(self, key: str, value: bool) -> 'LineBuilder': """Add a boolean field to the line protocol.""" self._validate_key(key, "field") self.fields[key] = 't' if value else 'f' return self def time_ns(self, timestamp_ns: int) -> 'LineBuilder': """Set the timestamp in nanoseconds.""" self._timestamp_ns = timestamp_ns return self def build(self) -> str: """Build the line protocol string.""" # Start with measurement name (escape commas only) line = self.measurement.replace(',', '\\,') # Add tags if present if self.tags: tags_str = ','.join( f"{k}={v}" for k, v in self.tags.items() ) line += f",{tags_str}" # Add fields (required) if not self.fields: raise InvalidLineError(f"At least one field is required: {line}") fields_str = ','.join( f"{k}={v}" for k, v in self.fields.items() ) line += f" {fields_str}" # Add timestamp if present if self._timestamp_ns is not None: line += f" {self._timestamp_ns}" return line ``` ## Query data Your plugins can execute SQL queries and process results directly: ```python # Simple query results = influxdb3_local.query("SELECT * FROM metrics WHERE time > now() - INTERVAL '1 hour'") # Parameterized query for safer execution params = {"table": "metrics", "threshold": 90} results = influxdb3_local.query("SELECT * FROM $table WHERE value > $threshold", params) ``` Query results are a `List` of `Dict[String, Any]`, where each dictionary represents a row. Column names are keys, and column values are the corresponding values. ## Log messages for monitoring and debugging Use the shared API’s `info`, `warn`, and `error` functions to log messages from your plugin. Each function accepts one or more arguments, converts them to strings, and logs them as a space-separated message. Add logging to monitor plugin execution and assist with debugging: ```python influxdb3_local.info("Starting data processing") influxdb3_local.warn("Could not process some records") influxdb3_local.error("Failed to connect to external API") # Log structured data obj_to_log = {"records": 157, "errors": 3} influxdb3_local.info("Processing complete", obj_to_log) ``` The system writes all log messages to the server logs and stores them in [system tables](/influxdb3/enterprise/reference/cli/influxdb3/show/system/summary/), where you can query them using SQL. ## Maintain state with the in-memory cache The Processing Engine provides an in-memory cache that enables your plugins to persist and retrieve data between executions. Access the cache using the `cache` property of the shared API: ```python # Basic usage pattern influxdb3_local.cache.METHOD(PARAMETERS) ``` `cache` provides the following methods to retrieve and manage cached values: | Method | Parameters |Returns| Description | |--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------|---------------------------------------------------------------------------| | `put` |`key` (str): The key to store the value under
`value` (Any): Any Python object to cache
`ttl` (Optional[float], default=None): Time in seconds before expiration
`use_global` (bool, default=False): If True, uses global namespace| None | Stores a value in the cache with an optional time-to-live | | `get` | `key` (str): The key to retrieve
`default` (Any, default=None): Value to return if key not found
`use_global` (bool, default=False): If True, uses global namespace | Any | Retrieves a value from the cache or returns default if not found | |`delete`| `key` (str): The key to delete
`use_global` (bool, default=False): If True, uses global namespace | bool |Deletes a value from the cache. Returns True if deleted, False if not found| ### Understanding cache namespaces The cache system offers two distinct namespaces: | Namespace | Scope | Best For | |------------------------------|----------------------------|------------------------------------------------------------------------------------| |**Trigger-specific** (default)|Isolated to a single trigger| Plugin state, counters, timestamps specific to one plugin | | **Global** | Shared across all triggers |Configuration, lookup tables, service states that should be available to all plugins| ### Common cache operations * [Store and retrieve cached data](#store-and-retrieve-cached-data) * [Store cached data with expiration](#store-cached-data-with-expiration) * [Share data across plugins](#share-data-across-plugins) * [Build a counter](#building-a-counter) ### Store and retrieve cached data ```python # Store a value influxdb3_local.cache.put("last_run_time", time.time()) # Retrieve a value with a default if not found last_time = influxdb3_local.cache.get("last_run_time", default=0) # Delete a cached value influxdb3_local.cache.delete("temporary_data") ``` ### Store cached data with expiration ```python # Cache with a 5-minute TTL (time-to-live) influxdb3_local.cache.put("api_response", response_data, ttl=300) ``` ### Share data across plugins ```python # Store in the global namespace influxdb3_local.cache.put("config", {"version": "1.0"}, use_global=True) # Retrieve from the global namespace config = influxdb3_local.cache.get("config", use_global=True) ``` ### Building a counter You can track how many times a plugin has run: ```python # Get current counter or default to 0 counter = influxdb3_local.cache.get("execution_count", default=0) # Increment counter counter += 1 # Store the updated value influxdb3_local.cache.put("execution_count", counter) influxdb3_local.info(f"This plugin has run {counter} times") ``` ## Guidelines for in-memory caching To get the most out of the in-memory cache, follow these guidelines: * [Use the trigger-specific namespace](#use-the-trigger-specific-namespace) * [Use TTL appropriately](#use-ttl-appropriately) * [Cache computation results](#cache-computation-results) * [Warm the cache](#warm-the-cache) * [Consider cache limitations](#consider-cache-limitations) ### Use the trigger-specific namespace The Processing Engine provides a cache that supports stateful operations while maintaining isolation between different triggers. For most use cases, use the trigger-specific namespace to keep plugin state isolated. Use the global namespace only when you need to share data across triggers. ### Use TTL appropriately Set appropriate expiration times based on how frequently your data changes: ```python # Cache external API responses for 5 minutes influxdb3_local.cache.put("weather_data", api_response, ttl=300) ``` ### Cache computation results Store the results of expensive calculations that you frequently utilize: ```python # Cache aggregated statistics influxdb3_local.cache.put("daily_stats", calculate_statistics(data), ttl=3600) ``` ### Warm the cache For critical data, prime the cache at startup. This can be especially useful for global namespace data where multiple triggers need the data: ```python # Check if cache needs to be initialized if not influxdb3_local.cache.get("lookup_table"): influxdb3_local.cache.put("lookup_table", load_lookup_data()) ``` ### Consider cache limitations * **Memory Usage**: Since the system stores cache contents in memory, monitor your memory usage when caching large datasets. * **Server Restarts**: Because the server clears the cache on restart, design your plugins to handle cache initialization (as noted above). * **Concurrency**: Be cautious of accessing inaccurate or out-of-date data when multiple trigger instances might simultaneously update the same cache key. ## Next Steps With an understanding of the InfluxDB 3 Shared Plugin API, you can start building data workflows that transform, analyze, and respond to your time series data. For official plugins and examples that you can extend, see the [plugin library](/influxdb3/enterprise/plugins/library/). #### Related * [influxdb3 create trigger](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/) * [influxdb3 test](/influxdb3/enterprise/reference/cli/influxdb3/test/) * [Processing engine reference](/influxdb3/enterprise/reference/processing-engine/) [processing engine](/influxdb3/enterprise/tags/processing-engine/)[plugins](/influxdb3/enterprise/tags/plugins/)[API](/influxdb3/enterprise/tags/api/)[python](/influxdb3/enterprise/tags/python/) | Method | Parameters | Returns | Description | | --- | --- | --- | --- | | Method | Parameters | Returns | Description | | put | key (str): The key to store the value under value (Any): Any Python object to cache ttl (Optional[float], default=None): Time in seconds before expiration use_global (bool, default=False): If True, uses global namespace | None | Stores a value in the cache with an optional time-to-live | | get | key (str): The key to retrieve default (Any, default=None): Value to return if key not found use_global (bool, default=False): If True, uses global namespace | Any | Retrieves a value from the cache or returns default if not found | | delete | key (str): The key to delete use_global (bool, default=False): If True, uses global namespace | bool | Deletes a value from the cache. Returns True if deleted, False if not found | | Namespace | Scope | Best For | | --- | --- | --- | | Namespace | Scope | Best For | | Trigger-specific (default) | Isolated to a single trigger | Plugin state, counters, timestamps specific to one plugin | | Global | Shared across all triggers | Configuration, lookup tables, service states that should be available to all plugins | --- # Plugin library Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/ Browse plugins for InfluxDB 3 Enterprise. Use these plugins to extend your database functionality with custom Python code that runs on write events, schedules, or HTTP requests. ### [Example plugins](/influxdb3/enterprise/plugins/library/examples/) Start with example plugins that demonstrate common use cases. ### [Official plugins](/influxdb3/enterprise/plugins/library/official/) Production-ready plugins developed and maintained by InfluxData. ## Requirements All plugins require: * InfluxDB 3 Core or InfluxDB 3 Enterprise with Processing Engine enabled * Python environment (managed automatically by InfluxDB 3) * Appropriate trigger configuration ## Plugin metadata Plugins in this library include a JSON metadata schema in a docstring header that defines supported trigger types and configuration parameters. This metadata enables: * the [InfluxDB 3 Explorer UI](/influxdb3/explorer/) to display and configure the plugin * automated testing and validation of plugins in the repository ## Using TOML Configuration Files Many plugins in this library support using TOML configuration files to specify all plugin arguments. This is useful for complex configurations or when you want to version control your plugin settings. ### Important Requirements **To use TOML configuration files, you must set the `PLUGIN_DIR` environment variable in the InfluxDB 3 Enterprise host environment.** This is required in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise: * `--plugin-dir` tells InfluxDB 3 Enterprise where to find plugin Python files * `PLUGIN_DIR` environment variable tells the plugins where to find TOML configuration files ### Set up TOML Configuration 1. **Start InfluxDB 3 Enterprise with the PLUGIN\_DIR environment variable set**: ``` PLUGIN_DIR=~/.plugins influxdb3 serve --node-id node0 --object-store file --data-dir ~/.influxdb3 --plugin-dir ~/.plugins ``` 2. **Copy or create a TOML configuration file in your plugin directory**: ``` # Example: copy a plugin's configuration template cp plugin_config_example.toml ~/.plugins/my_config.toml ``` 3. **Edit the TOML file** to match your requirements. The TOML file should contain all the arguments defined in the plugin’s argument schema. 4. **Create a trigger with the `config_file_path` argument**: When creating a trigger, specify the `config_file_path` argument to point to your TOML configuration file. * Specify only the filename (not the full path) * The file must be located under `PLUGIN_DIR` ``` influxdb3 create trigger \ --database mydb \ --plugin-filename plugin_name.py \ --trigger-spec "every:1d" \ --trigger-arguments config_file_path=my_config.toml \ my_trigger_name ``` For more information on using TOML configuration files, see the project [README](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/) --- # Example plugins Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/examples/ Example plugins demonstrate common use cases and patterns for extending InfluxDB 3 Enterprise with custom Python code. ### [WAL plugin](/influxdb3/enterprise/plugins/library/examples/wal-plugin/) Example Write-Ahead Log (WAL) plugin that demonstrates processing data as it’s written to the database. [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[examples](/influxdb3/enterprise/tags/examples/) --- # WAL plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/examples/wal-plugin/ The example WAL plugin monitors data write operations in InfluxDB 3 by tracking row counts for each table during WAL (Write-Ahead Log) flush events. It creates summary reports in a `write_reports` table to help analyze data ingestion patterns and rates. The plugin can optionally double-count rows for a specified table to demonstrate configurable behavior. ## Configuration ### Optional parameters | Parameter | Type |Default| Description | |--------------------|------|-------|-----------------------------------------------------------------------------------------| |`double_count_table`|string| none |Table name for which to double the row count in write reports (for testing/demonstration)| ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`) ``` influxdb3 serve \ --cluster-id cluster0 \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` ## Trigger setup ### Write monitoring Monitor all table writes and generate write reports: ```bash influxdb3 create trigger \ --database monitoring \ --plugin-filename examples/wal_plugin/wal_plugin.py \ --trigger-spec "all_tables" \ wal_monitoring ``` ### Write monitoring with special handling Monitor writes with special handling for a specific table: ```bash influxdb3 create trigger \ --database monitoring \ --plugin-filename examples/wal_plugin/wal_plugin.py \ --trigger-spec "all_tables" \ --trigger-arguments 'double_count_table=temperature' \ wal_monitoring_special ``` ## Example usage ### Example: Basic write monitoring Set up write monitoring to track data ingestion: ``` # Create the monitoring trigger influxdb3 create trigger \ --database testdb \ --plugin-filename examples/wal_plugin/wal_plugin.py \ --trigger-spec "all_tables" \ write_monitor # Write test data to various tables influxdb3 write \ --database testdb \ "temperature,location=office value=22.5" influxdb3 write \ --database testdb \ "humidity,location=office value=45.2" influxdb3 write \ --database testdb \ "pressure,location=office value=1013.25" # The plugin automatically generates write reports in the `write_reports` measurement. # Query the write reports influxdb3 query \ --database testdb \ "SELECT * FROM write_reports ORDER BY time DESC" ``` ### Expected output ``` table_name | row_count | time ------------|-----------|----- pressure | 1 | 2024-01-01T12:02:00Z humidity | 1 | 2024-01-01T12:01:00Z temperature | 1 | 2024-01-01T12:00:00Z ``` ### Example: Monitoring with special table handling Monitor writes with doubled counting for temperature data: ``` # Create trigger with special handling influxdb3 create trigger \ --database testdb \ --plugin-filename examples/wal_plugin/wal_plugin.py \ --trigger-spec "all_tables" \ --trigger-arguments 'double_count_table=temperature' \ write_monitor_special # Write test data influxdb3 write \ --database testdb \ "temperature,location=office value=22.5" influxdb3 write \ --database testdb \ "humidity,location=office value=45.2" # Query the write reports influxdb3 query \ --database testdb \ "SELECT * FROM write_reports ORDER BY time DESC" ``` ### Expected output ``` table_name | row_count | time ------------|-----------|----- humidity | 1 | 2024-01-01T12:01:00Z temperature | 2 | 2024-01-01T12:00:00Z ``` **Note**: The temperature table shows a row count of 2 despite only writing 1 row, demonstrating the `double_count_table` parameter. ## Generated Measurements ### write\_reports Tracks the number of rows written to each table during WAL flush events. **Tags:** * `table_name`: Name of the table that received writes **Fields:** * `row_count`: Number of rows written in this WAL flush (integer) **Special behavior:** * If `double_count_table` parameter matches the table name, the row count will be doubled * The plugin automatically skips the `write_reports` table to avoid infinite recursion ## Code overview ### Files * `wal_plugin.py`: Main plugin code that processes write batches and generates reports ### Main functions #### `process_writes(influxdb3_local, table_batches, args)` Entry point for processing write batches. Called each time data is written to the database. **Parameters:** * `influxdb3_local`: InfluxDB client for writing and logging * `table_batches`: List of table batches containing written data * `args`: Configuration arguments from trigger setup **Processing logic:** 1. Iterates through each table batch in the write operation 2. Skips the `write_reports` table to prevent recursion 3. Counts rows in each batch 4. Applies special handling if `double_count_table` matches 5. Writes report record to `write_reports` measurement ### Logging Logs are stored in the `_internal` database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query \ --database _internal \ --token AUTH_TOKEN \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'wal_monitoring'" ``` Replace `AUTH_TOKEN` with your [admin token](/influxdb3/enterprise/admin/tokens/). ## Troubleshooting ### Common issues #### Issue: No write reports appearing **Solution**: 1. Verify the trigger was created successfully: ``` influxdb3 show summary --database DATABASE_NAME --token AUTH_TOKEN ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with read permissions on the specified database 2. Check that data is actually being written to tables other than `write_reports` 3. Review logs for errors #### Issue: Infinite recursion with write\_reports **Solution**: This shouldn’t happen as the plugin automatically skips the `write_reports` table, but if you see this: 1. Check that you haven’t modified the plugin to remove the skip logic 2. Verify the table name comparison is working correctly #### Issue: Row counts seem incorrect **Solution**: 1. Remember that row counts represent WAL flush batches, not individual write operations 2. Multiple write operations may be batched together before the plugin processes them 3. Check if `double_count_table` is set and affecting specific tables ### Performance considerations * This plugin processes every write operation, so it adds minimal overhead * The plugin generates one additional write per table per WAL flush batch * Consider the storage impact of write reports for high-volume systems ### Use cases * **Write monitoring**: Track data ingestion patterns and volumes * **Debugging**: Identify which tables are receiving writes * **Performance analysis**: Monitor write batch sizes and patterns * **Data validation**: Verify expected write volumes * **Testing**: Use `double_count_table` parameter for testing scenarios ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [WAL plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/examples/wal-plugin) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[wal](/influxdb3/enterprise/tags/wal/)[data-write](/influxdb3/enterprise/tags/data-write/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | double_count_table | string | none | Table name for which to double the row count in write reports (for testing/demonstration) | --- # Official plugins Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/ Official plugins developed and maintained by InfluxData for production use with InfluxDB 3 Enterprise. ### [Amqp subscriber plugin](/influxdb3/enterprise/plugins/library/official/amqp-subscriber/) Enables real-time ingestion of AMQP messages into InfluxDB 3. Subscribe to queues and transform messages into time-series data with support for JSON, Line Protocol, and custom text formats. ### [Basic transformation plugin](/influxdb3/enterprise/plugins/library/official/basic-transformation/) Provides common data transformation functions for modifying and enriching time series data. ### [Bird data simulator plugin](/influxdb3/enterprise/plugins/library/official/bird-data-simulator/) Generates synthetic bird tracking telemetry with persistent birds, ranges, simulated movement, speed, heading, and body temperature. Exposes only simple volume controls. ### [Chronos forecasting plugin](/influxdb3/enterprise/plugins/library/official/chronos-forecasting/) Enables zero-shot time-series forecasting with Amazon Chronos models from HuggingFace, supporting scheduled batch forecasts and on-demand HTTP forecasts. ### [Downsampler plugin](/influxdb3/enterprise/plugins/library/official/downsampler/) Automatically downsample and aggregate time series data at configurable intervals. ### [Earthquake sampler plugin](/influxdb3/enterprise/plugins/library/official/earthquake-sampler/) Ingests USGS earthquake events on a schedule and writes normalized records for dashboards and alerting, with optional canonical quake-schema output. ### [Forecast error evaluator plugin](/influxdb3/enterprise/plugins/library/official/forecast-error-evaluator/) Evaluate forecast accuracy by comparing predicted values against actual measurements. ### [Gapfill plugin](/influxdb3/enterprise/plugins/library/official/gapfill/) Detects gaps in time series and fills them with imputed values using configurable interpolation methods, with optional fill marking and per-gap quality reports. Supports scheduled and HTTP backfill. ### [Geo enrichment plugin](/influxdb3/enterprise/plugins/library/official/geo-enrichment/) Resolves lat/lon into location attributes—country and city, a zone or site you define, or a grid cell—merged into the source rows or written to a target table. HTTP endpoint backfills history. ### [Import plugin](/influxdb3/enterprise/plugins/library/official/import/) Enables seamless data import from InfluxDB v1, v2, or v3 instances to InfluxDB 3 Core/Enterprise ### [InfluxDB to Iceberg plugin](/influxdb3/enterprise/plugins/library/official/influxdb-to-iceberg/) Export time series data from InfluxDB to Apache Iceberg table format for data lake integration. ### [Kafka subscriber plugin](/influxdb3/enterprise/plugins/library/official/kafka-subscriber/) Ingests Kafka messages into InfluxDB 3 with JSON, Line Protocol, text, Avro, JSON Schema, and Protobuf decoding, plus Schema Registry and local schema support. ### [MAD anomaly detection plugin](/influxdb3/enterprise/plugins/library/official/mad-anomaly-detection/) Detect anomalies using Median Absolute Deviation (MAD) statistical analysis. ### [Mqtt publisher plugin](/influxdb3/enterprise/plugins/library/official/mqtt-publisher/) Publishes InfluxDB 3 rows to an MQTT broker on write, on a schedule, or over HTTP for a backfill. JSON, line protocol or template payloads; per-row, per-field or batch; filtering, TLS, buffered queue. ### [Mqtt sparkplug publisher plugin](/influxdb3/enterprise/plugins/library/official/mqtt-sparkplug-publisher/) Publishes InfluxDB 3 rows to an MQTT broker as a Sparkplug B Edge Node on write: spBv1.0 topics, Protobuf payloads, NBIRTH/NDATA/NDEATH, bdSeq/seq, aliases, rebirth, STATE and store-and-forward. ### [Mqtt subscriber plugin](/influxdb3/enterprise/plugins/library/official/mqtt-subscriber/) Enables real-time ingestion of MQTT messages into InfluxDB 3. Subscribe to MQTT topics and transform messages into time-series data with support for JSON, Line Protocol, and custom text formats. ### [Notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) Send notifications and alerts to various channels including email, Slack, and webhooks. ### [NWS weather plugin](/influxdb3/enterprise/plugins/library/official/nws-weather/) Provides real-time weather data from the National Weather Service API for demonstration and sample data purposes. Fetches live observations from multiple weather stations across the United States. ### [Opcua plugin](/influxdb3/enterprise/plugins/library/official/opcua/) Enables periodic ingestion of OPC UA node values into InfluxDB 3. Connect to OPC UA servers and read current values with auto-discovery browse mode, data type detection, and namespace resolution. ### [Prophet forecasting plugin](/influxdb3/enterprise/plugins/library/official/prophet-forecasting/) Generate time series forecasts using Facebook Prophet for predictive analytics. ### [Resampler plugin](/influxdb3/enterprise/plugins/library/official/resampler/) Resamples time series with non-uniform timestamps onto a uniform time grid via configurable interpolation, or snaps timestamps to grid nodes keeping values and types unchanged. ### [River anomaly detector plugin](/influxdb3/enterprise/plugins/library/official/river-anomaly-detector/) Detects time-series anomalies on writes using River ML models, combining rolling statistics with HalfSpaceTrees for per-series anomaly scoring and optional auto-tuning from River Auto Profiler output. ### [River auto profiler plugin](/influxdb3/enterprise/plugins/library/official/river-auto-profiler/) Incrementally profiles time-series data with River ML streaming statistics and writes per-series recommendations for anomaly detection tuning, including thresholds, fading factors, and maturity. ### [River forecaster plugin](/influxdb3/enterprise/plugins/library/official/river-forecaster/) Provides online time-series forecasting with River ML SNARIMAX models, learning from writes and producing scheduled multi-step forecasts with per-series model state and optional auto-horizon support. ### [Sagemaker plugin](/influxdb3/enterprise/plugins/library/official/sagemaker/) Runs scheduled inference against Amazon SageMaker endpoints using recent InfluxDB 3 rows and writes prediction results back to InfluxDB. ### [Schema validator plugin](/influxdb3/enterprise/plugins/library/official/schema-validator/) Validates incoming line protocol data against a JSON schema. ### [Signal filter plugin](/influxdb3/enterprise/plugins/library/official/signal-filter/) Applies streaming digital IIR filters to numeric fields ### [Signal generator plugin](/influxdb3/enterprise/plugins/library/official/signal-generator/) Generates scheduled, configurable waveform time-series data for demos, testing, dashboards, alerts, and plugin validation without external sources. Simulates outage gaps by default. ### [Simple data replicator plugin](/influxdb3/enterprise/plugins/library/official/simple-data-replicator/) Replicates data between InfluxDB 3 instances over HTTP with table and field filtering, renaming, scheduler/write triggers, and compressed retry queue buffering. ### [State change plugin](/influxdb3/enterprise/plugins/library/official/state-change/) Detect and track state changes in time series data for event monitoring. ### [Stateless ADTK detector plugin](/influxdb3/enterprise/plugins/library/official/stateless-adtk-detector/) Perform anomaly detection using the Anomaly Detection Toolkit (ADTK) without maintaining state. ### [Stock plugin](/influxdb3/enterprise/plugins/library/official/stock-plugin/) Tracks stock, ETF, and mutual fund portfolio values from Yahoo Finance with market-hours gating and rollups. ### [Synthefy forecasting plugin](/influxdb3/enterprise/plugins/library/official/synthefy-forecasting/) Integrates Synthefy Forecasting API with InfluxDB 3 for on-demand time series forecasting via HTTP. Reads data from InfluxDB, generates forecasts with Synthefy models, and writes results back. ### [Synthefy nori regression plugin](/influxdb3/enterprise/plugins/library/official/synthefy-nori-regression/) Predict a numeric InfluxDB 3 field from other columns with Synthefy’s Nori in-context tabular regression model, via the Synthefy inference gateway. Imputes rows where the target field is null. ### [System metrics plugin](/influxdb3/enterprise/plugins/library/official/system-metrics/) Collects comprehensive system performance metrics including CPU, memory, disk, and network statistics. ### [Threshold deadman checks plugin](/influxdb3/enterprise/plugins/library/official/threshold-deadman-checks/) Monitor data thresholds and detect missing data with deadman checks for alerting. ### [Valuecounter plugin](/influxdb3/enterprise/plugins/library/official/valuecounter/) Counts unique field values from data-write or scheduled inputs and writes rollup measurements with per-value counts. [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) --- # Amqp subscriber plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/amqp-subscriber/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The AMQP Subscriber Plugin enables real-time ingestion of AMQP messages (RabbitMQ and other AMQP-compatible brokers) into InfluxDB 3 Enterprise. Subscribe to queues and automatically transform messages into time-series data with support for JSON, Line Protocol, and custom text formats. The plugin provides flexible message acknowledgement policies and comprehensive error tracking with statistics. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. This plugin supports TOML configuration files for complex mapping scenarios, which can be specified using the `config_file_path` parameter. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type | Default | Description | |------------------|------|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `host` |string| required | AMQP broker hostname or IP address | | `queues` |string| required | Space-separated list of queue names to consume messages from (for example, “sensor\_data alerts”) | | `table_name` |string|required (json/text only)| Static InfluxDB measurement name. Required for `json`/`text` formats if `table_name_field` is not set. Not required for `lineprotocol`. | |`table_name_field`|string| none |Field path to extract table name dynamically from message data. JSON: field name (auto-prefixed with `$.`). Text: regex pattern with capture group. Alternative to `table_name`.| ### Connection parameters | Parameter | Type |Default| Description | |--------------|-------|-------|-------------------------------------------------| | `port` |integer| 5672 |AMQP broker port (5672 for non-TLS, 5671 for TLS)| |`virtual_host`|string | “/” | AMQP virtual host | ### Authentication parameters | Parameter | Type |Default| Description | |----------------|------|-------|---------------------------------------------------------------| |`auth_mechanism`|string|“plain”| Authentication mechanism: `plain` or `external` | | `username` |string| none |AMQP broker username. Required when `auth_mechanism` is `plain`| | `password` |string| none |AMQP broker password. Required when `auth_mechanism` is `plain`| **Authentication mechanisms:** * `plain` - Username/password authentication (SASL PLAIN). Both `username` and `password` are required; the plugin no longer falls back to the RabbitMQ default `guest`/`guest` credentials. * `external` - Client TLS certificate authentication (SASL EXTERNAL). Requires `ssl_ca_cert`, `ssl_client_cert`, and `ssl_client_key`. `username`/`password` are ignored. ### TLS/SSL parameters | Parameter (CLI) | Parameter (TOML) | Type |Default| Description | |-----------------|--------------------------|------|-------|-------------------------------------------------------| | `ssl_ca_cert` | `[amqp.ssl]` `ca_cert` |string| none |Path to CA certificate file. Enables SSL when provided.| |`ssl_client_cert`|`[amqp.ssl]` `client_cert`|string| none | Path to client certificate for mutual TLS | |`ssl_client_key` |`[amqp.ssl]` `client_key` |string| none | Path to client private key for mutual TLS | **Note:** For mutual TLS, both `ssl_client_cert` and `ssl_client_key` must be provided together. SSL is automatically enabled when `ssl_ca_cert` is provided. ### Message handling parameters | Parameter | Type | Default | Description | |--------------------|-------|-------------|-------------------------------------------------------------------------------------------------------------------| | `ack_policy` |string |“on\_success”| When to acknowledge messages: `on_success` or `always` | |`requeue_on_failure`|string | “false” |Whether to requeue messages that fail processing. When `true`, failed messages are returned to the queue for retry.| | `max_messages` |integer| 500 | Maximum number of messages to retrieve per scheduled call | **Acknowledgement policies:** * `on_success` - Acknowledge only after successful processing. Failed messages are rejected (requeue controlled by `requeue_on_failure`). * `always` - Acknowledge all messages at the end of processing, regardless of success or failure. ### Logging parameters In TOML configuration, `enable_full_logging` is placed directly under the `[amqp]` section. | Parameter | Type |Default| Description | |---------------------|-------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`enable_full_logging`|boolean| false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths) into log output. Enable temporarily for debugging.| ### Message format parameters | Parameter | Type |Default| Description | |-----------------|------|-------|-------------------------------------------------------------| | `format` |string|“json” | Message format: json, lineprotocol, or text | |`timestamp_field`|string| none |Field containing timestamp (format depends on message format)| **Format-specific timestamp\_field syntax:** The timestamp field format differs between JSON and Text formats: |Format| Syntax | Split Method | Example (CLI) | Example (TOML) | |------|-------------------|------------------|----------------|------------------| | JSON |`field_name:format`|Split by first `:`|`"timestamp:ms"`|`"$.timestamp:ms"`| | Text | `regex:format` |Split by last `:` |`"ts:(\\d+):ms"`| `"ts:(\\d+):ms"` | **Note:** In CLI arguments, JSON paths are specified without `$.` prefix (added automatically). In TOML configuration, use full JSONPath syntax with `$.` prefix. **Note:** Text format uses the last colon to split, allowing regex patterns to contain colons (for example, time patterns). **Supported timestamp formats:** * `ns` - nanoseconds (Unix timestamp) * `ms` - milliseconds (Unix timestamp) * `s` - seconds (Unix timestamp) * `datetime` - ISO 8601 string (for example, “2021-12-01T12:00:00Z”) ### JSON format parameters |Parameter| Type |Default | Description | |---------|------|--------|-------------------------------------------------------------------------| | `tags` |string| none | Space-separated tag names. Example: “room sensor location” | |`fields` |string|required|Space-separated field mappings. Format: “name:type=jsonpath” without `$.`| **Field specification format:** `"temp:float=temperature hum:int=humidity status:bool=online"` **Supported field types:** `int`, `uint`, `float`, `string`, `bool` ### Text format parameters |Parameter| Type |Default | Description | |---------|------|--------|------------------------------------------------------------------| | `tags` |string| none | Space-separated tag patterns. Format: “name=regex\_pattern” | |`fields` |string|required|Space-separated field patterns. Format: “name:type=regex\_pattern”| **Tag specification format:** `"room=room:([^,\\s]+) sensor=sensor:(\\w+)"` **Field specification format:** `"temp:float=temp:([\\d.]+) status:bool=(true|false)"` ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|-----------------------------------------------| |`config_file_path`|string| none |Path to TOML config file (absolute or relative)| *To use a TOML configuration file, specify the `config_file_path` in the trigger arguments.* ### File path resolution All file paths in the plugin (configuration file, TLS certificates) follow the same resolution logic: * **Absolute paths** (for example, `/etc/amqp/config.toml`) are used as-is * **Relative paths** (for example, `config.toml`, `certs/ca.crt`) are resolved from `PLUGIN_DIR` environment variable If a relative path is specified and `PLUGIN_DIR` is not set, the plugin will return an error. #### Example TOML configuration [amqp\_config\_example.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/amqp_subscriber/amqp_config_example.toml) - comprehensive configuration example with all three message formats ## Data requirements The plugin automatically creates the target measurement table on first write. Field mappings are required for JSON and Text formats to specify which fields to extract and their data types. ### Message encoding requirements * **UTF-8 text only**: The plugin only processes UTF-8 encoded text messages. Binary messages are automatically skipped with a warning logged. * **Non-empty payloads**: Empty or whitespace-only messages are automatically skipped. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: * `pika` (AMQP client library) * `jsonpath-ng` (JSON path parsing for JSON format) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package pika influxdb3 install package jsonpath-ng ``` ## Trigger setup ### Scheduled ingestion with TOML configuration Recommended for production use with complex mappings: ```bash # 1. Set PLUGIN_DIR environment variable export PLUGIN_DIR=~/.plugins # 2. Copy and edit configuration file cp amqp_config_example.toml $PLUGIN_DIR/my_amqp_config.toml # Edit my_amqp_config.toml with your broker and mapping settings # 3. Create the trigger influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/amqp_subscriber/amqp_subscriber.py \ --trigger-spec "every:10s" \ --trigger-arguments config_file_path=my_amqp_config.toml \ amqp_ingestion ``` ### Scheduled ingestion with command-line arguments For simple JSON message ingestion: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/amqp_subscriber/amqp_subscriber.py \ --trigger-spec "every:5s" \ --trigger-arguments 'host=localhost,queues=sensor_data alerts,format=json,table_name=sensor_data,fields=temp:float=temperature hum:int=humidity,tags=location sensor_id' \ amqp_sensors ``` ### Secure AMQP connection with TLS ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/amqp_subscriber/amqp_subscriber.py \ --trigger-spec "every:10s" \ --trigger-arguments 'host=secure-broker.example.com,port=5671,queues=secure_data,format=json,table_name=secure_data,ssl_ca_cert=certs/ca.crt,username=myuser,password=mypass,fields=value:float=value' \ secure_amqp ``` ## Message Formats ### JSON Format The primary use case for structured IoT data. Supports nested fields using JSONPath expressions. #### TOML Configuration ```toml [amqp] host = "localhost" port = 5672 queues = ["sensor_data", "alerts"] format = "json" [mapping.json] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.json.tags] location = "$.location" sensor_id = "$.sensor.id" [mapping.json.fields] temperature = ["$.temp", "float"] humidity = ["$.humidity", "int"] status = ["$.online", "bool"] ``` #### Example Message ```json { "timestamp": 1638360000000, "location": "warehouse_a", "sensor": { "id": "sensor_001" }, "temp": 22.5, "humidity": 65, "online": true } ``` #### Resulting Data ``` sensor_data,location=warehouse_a,sensor_id=sensor_001 temperature=22.5,humidity=65i,status=true 1638360000000000000 ``` #### JSON Array Support Process batch messages containing arrays of JSON objects: ```json [ {"timestamp": 1638360000000, "sensor_id": "001", "temperature": 22.5}, {"timestamp": 1638360001000, "sensor_id": "002", "temperature": 23.1}, {"timestamp": 1638360002000, "sensor_id": "003", "temperature": 21.8} ] ``` **Array processing behavior:** * Each array element is processed independently as a separate data point * If one element fails to parse, the others continue processing (partial success) * Parse errors for individual elements are logged to `amqp_exceptions` table * Statistics count each AMQP message as one unit (messages\_received+=1, messages\_processed+=1), regardless of array size ### Line Protocol Format For messages already in InfluxDB line protocol format, use passthrough mode. No mapping configuration needed - messages are validated and written directly. #### TOML Configuration ```toml [amqp] host = "rabbitmq.example.com" queues = ["influxdb_metrics"] format = "lineprotocol" ``` #### CLI Configuration ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename amqp_subscriber.py \ --trigger-spec "every:10s" \ --trigger-arguments 'host=rabbitmq.example.com,queues=influxdb_metrics,format=lineprotocol' \ amqp_lineprotocol ``` #### Example Message ``` sensor_data,location=warehouse_a,sensor_id=001 temperature=22.5,humidity=65i 1638360000000000000 ``` #### Supported Line Protocol Types | Type | Suffix | Example | |----------------|--------------|------------------| | Float | none |`temperature=22.5`| | Integer | `i` | `count=100i` | |Unsigned Integer| `u` | `bytes=1024u` | | String | `"..."` |`status="running"`| | Boolean |`true`/`false`| `active=true` | ### Text Format Parse plain text messages using regular expressions: #### TOML Configuration ```toml [amqp] format = "text" [mapping.text] table_name = "sensor_logs" timestamp_field = "ts:(\\d+):ms" [mapping.text.tags] location = "location=([^,\\s]+)" [mapping.text.fields] temperature = ["temp:([\\d.]+)", "float"] humidity = ["hum:(\\d+)", "int"] status = ["status:(true|false)", "bool"] ``` #### Example Message ``` location=warehouse_a,temp:22.5,hum:65,status:true,ts:1638360000000 ``` ## Statistics and Monitoring The plugin tracks comprehensive statistics and writes them to the `amqp_stats` table on every plugin invocation. **Important notes:** * Statistics are written **on every plugin invocation** * Each queue is tracked separately with independent statistics * Statistics persist across plugin restarts using the InfluxDB cache ### amqp\_stats Table | Field |Type | Description | |---------------------|-----|------------------------------------| | `queue` (tag) | tag | AMQP queue name | | `host` (tag) | tag | AMQP broker address | |`virtual_host` (tag) | tag | AMQP virtual host | | `messages_received` | int | Total messages received (all time) | |`messages_processed` | int |Total messages processed (all time) | | `messages_failed` | int | Total messages failed (all time) | | `success_rate` |float| Total success rate (all time, %) | | `period_received` | int |Messages received in current period | | `period_processed` | int |Messages processed in current period| | `period_failed` | int | Messages failed in current period | |`period_success_rate`|float|Success rate for current period (%) | ### Querying Statistics ```bash # Get latest statistics influxdb3 query --database mydb \ "SELECT * FROM amqp_stats ORDER BY time DESC LIMIT 10" # Success rate over time influxdb3 query --database mydb \ "SELECT queue, success_rate, messages_processed, messages_failed FROM amqp_stats WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC" ``` ## Error Handling Parse errors and message processing failures are logged to the `amqp_exceptions` table: ### amqp\_exceptions Table | Field | Type | Description | |------------------|------|--------------------------------------------| | `queue` (tag) | tag | AMQP queue where error occurred | |`error_type` (tag)| tag |Type of error (for example, JSONDecodeError)| | `error_message` |string| Detailed error message | | `raw_message` |string| Original AMQP message (truncated to 1KB) | ### Checking for Errors ```bash influxdb3 query --database mydb \ "SELECT * FROM amqp_exceptions ORDER BY time DESC LIMIT 10" ``` ## Troubleshooting ### Check Plugin Logs ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'amqp_ingestion' ORDER BY time DESC LIMIT 20" ``` ### Common Issues #### “pika library not installed” ```bash influxdb3 install package pika ``` #### “Configuration file not found” * For relative paths, ensure `PLUGIN_DIR` environment variable is set * For absolute paths, verify the file exists at the specified location ```bash # For relative paths export PLUGIN_DIR=~/.plugins ls $PLUGIN_DIR/my_amqp_config.toml # Or use absolute path ls /etc/amqp/my_amqp_config.toml ``` #### “Failed to connect to AMQP broker” * Verify broker address and port * Check network connectivity * For TLS connections, verify certificate paths * Verify authentication credentials * Verify `auth_mechanism` matches the broker configuration (`plain` requires `username`/`password`; `external` requires client certificates) #### “Both ssl\_client\_cert and ssl\_client\_key must be provided for mutual TLS” For mutual TLS authentication, both client certificate and key are required. #### “No fields were mapped from JSON data” * Verify JSONPath expressions in field mappings (use `$.` prefix) * Check that JSON structure matches your paths * Review `amqp_exceptions` table for detailed errors #### Messages not being processed * Check trigger status: `influxdb3 show summary --database mydb` * Verify AMQP connection in plugin logs * Ensure the queue exists and has messages * Check that the queue name is correct #### Failed messages are being discarded By default, failed messages are rejected without requeue (`requeue_on_failure=false`): * Set `requeue_on_failure=true` to return failed messages to the queue for retry * Use `ack_policy=always` to acknowledge all messages regardless of errors * Check `amqp_exceptions` table to diagnose parsing issues * Configure a dead-letter queue in RabbitMQ to capture rejected messages for analysis ## Architecture ### How It Works 1. **Scheduled Trigger**: Plugin runs on schedule (for example, `every:10s`) 2. **Configuration Caching**: Plugin configuration is parsed once and cached between executions 3. **Message Retrieval**: Plugin connects, retrieves up to `max_messages` from queue using `basic_get` 4. **Parse & Write**: Messages parsed according to format and written to InfluxDB 5. **Acknowledgement**: Messages acknowledged based on `ack_policy` setting 6. **Error Tracking**: Parse errors logged to `amqp_exceptions` table 7. **Statistics**: Written to `amqp_stats` table on every plugin invocation 8. **Disconnect**: Connection closed after processing ### Performance Optimization The plugin includes several optimizations for high-throughput scenarios: * **Configuration Caching**: Plugin configuration is parsed once and reused across all trigger executions * **Pre-compiled Patterns**: JSONPath expressions and regex patterns are compiled once during parser initialization, not per-message * **Batch Retrieval**: Multiple messages retrieved in a single connection session * **QoS Control**: Prefetch count limits memory usage on consumer side ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Amqp subscriber plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/amqp_subscriber) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | host | string | required | AMQP broker hostname or IP address | | queues | string | required | Space-separated list of queue names to consume messages from (for example, “sensor_data alerts”) | | table_name | string | required (json/text only) | Static InfluxDB measurement name. Required for json / text formats if table_name_field is not set. Not required for lineprotocol . | | table_name_field | string | none | Field path to extract table name dynamically from message data. JSON: field name (auto-prefixed with $. ). Text: regex pattern with capture group. Alternative to table_name . | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | port | integer | 5672 | AMQP broker port (5672 for non-TLS, 5671 for TLS) | | virtual_host | string | “/” | AMQP virtual host | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | auth_mechanism | string | “plain” | Authentication mechanism: plain or external | | username | string | none | AMQP broker username. Required when auth_mechanism is plain | | password | string | none | AMQP broker password. Required when auth_mechanism is plain | | Parameter (CLI) | Parameter (TOML) | Type | Default | Description | | --- | --- | --- | --- | --- | | Parameter (CLI) | Parameter (TOML) | Type | Default | Description | | ssl_ca_cert | [amqp.ssl] ca_cert | string | none | Path to CA certificate file. Enables SSL when provided. | | ssl_client_cert | [amqp.ssl] client_cert | string | none | Path to client certificate for mutual TLS | | ssl_client_key | [amqp.ssl] client_key | string | none | Path to client private key for mutual TLS | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | ack_policy | string | “on_success” | When to acknowledge messages: on_success or always | | requeue_on_failure | string | “false” | Whether to requeue messages that fail processing. When true , failed messages are returned to the queue for retry. | | max_messages | integer | 500 | Maximum number of messages to retrieve per scheduled call | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths) into log output. Enable temporarily for debugging. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | format | string | “json” | Message format: json, lineprotocol, or text | | timestamp_field | string | none | Field containing timestamp (format depends on message format) | | Format | Syntax | Split Method | Example (CLI) | Example (TOML) | | --- | --- | --- | --- | --- | | Format | Syntax | Split Method | Example (CLI) | Example (TOML) | | JSON | field_name:format | Split by first : | "timestamp:ms" | "$.timestamp:ms" | | Text | regex:format | Split by last : | "ts:(\\d+):ms" | "ts:(\\d+):ms" | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tags | string | none | Space-separated tag names. Example: “room sensor location” | | fields | string | required | Space-separated field mappings. Format: “name:type=jsonpath” without $. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tags | string | none | Space-separated tag patterns. Format: “name=regex_pattern” | | fields | string | required | Space-separated field patterns. Format: “name:type=regex_pattern” | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | Path to TOML config file (absolute or relative) | | Type | Suffix | Example | | --- | --- | --- | | Type | Suffix | Example | | Float | none | temperature=22.5 | | Integer | i | count=100i | | Unsigned Integer | u | bytes=1024u | | String | "..." | status="running" | | Boolean | true / false | active=true | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | queue (tag) | tag | AMQP queue name | | host (tag) | tag | AMQP broker address | | virtual_host (tag) | tag | AMQP virtual host | | messages_received | int | Total messages received (all time) | | messages_processed | int | Total messages processed (all time) | | messages_failed | int | Total messages failed (all time) | | success_rate | float | Total success rate (all time, %) | | period_received | int | Messages received in current period | | period_processed | int | Messages processed in current period | | period_failed | int | Messages failed in current period | | period_success_rate | float | Success rate for current period (%) | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | queue (tag) | tag | AMQP queue where error occurred | | error_type (tag) | tag | Type of error (for example, JSONDecodeError) | | error_message | string | Detailed error message | | raw_message | string | Original AMQP message (truncated to 1KB) | --- # Basic transformation plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/basic-transformation/ The Basic Transformation Plugin enables real-time and scheduled transformation of time series data in InfluxDB 3 Enterprise. Transform field and tag names, convert values between units, and apply custom string replacements to standardize or clean your data. The plugin supports both scheduled batch processing of historical data and real-time transformation as data is written. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type | Default | Description | |---------------------|-------|----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `measurement` |string | required | Source measurement containing data to transform | |`target_measurement` |string | required | Destination measurement for transformed data | | `target_database` |string |current database| Database for storing transformed data | | `dry_run` |string | “false” | When “true”, logs transformations without writing | |`enable_full_logging`|boolean| false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging.| ### Transformation parameters | Parameter | Type |Default| Description | |------------------------|------|-------|---------------------------------------------------------------------------------------------------| |`names_transformations` |string| none |Field/tag name transformation rules. Format: `'field1:"transform1 transform2".field2:"transform3"'`| |`values_transformations`|string| none | Field value transformation rules. Format: `'field1:"transform1".field2:"transform2"'` | | `custom_replacements` |string| none | Custom string replacements. Format: `'rule_name:"find=replace"'` | | `custom_regex` |string| none | Regex patterns for field matching. Format: `'pattern_name:"temp%"'` | ### Data selection parameters | Parameter | Type | Default | Description | |-----------------|------|-------------------------|----------------------------------------------------------------------------------------| | `window` |string|required (scheduled only)| Historical data window. Format: `` (for example, “30d”, “1h”) | |`included_fields`|string| all fields and tags |Dot-separated list of fields and tags to include (for example, “temp.humidity.location”)| |`excluded_fields`|string| none | Dot-separated list of fields and tags to exclude | | `filters` |string| none | Query filters. Format: `'field:"operator value"'` | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. #### Example TOML configurations * [basic\_transformation\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/basic_transformation/basic_transformation_config_scheduler.toml) - for scheduled triggers * [basic\_transformation\_config\_data\_writes.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/basic_transformation/basic_transformation_config_data_writes.toml) - for data write triggers For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Data requirements The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: * `pint` (for unit conversions) ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package pint ``` ## Trigger setup ### Scheduled transformation Run transformations periodically on historical data: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/basic_transformation/basic_transformation.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'measurement=temperature,window=24h,target_measurement=temperature_normalized,names_transformations=temp:"snake",values_transformations=temp:"convert_degC_to_degF"' \ hourly_temp_transform ``` ### Real-time transformation Transform data as it’s written: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/basic_transformation/basic_transformation.py" \ --trigger-spec "all_tables" \ --trigger-arguments 'measurement=sensor_data,target_measurement=sensor_data_clean,names_transformations=.*:"snake remove_special_chars normalize_underscores"' \ realtime_clean ``` ## Example usage ### Example 1: Temperature unit conversion Convert temperature readings from Celsius to Fahrenheit while standardizing field names: ```bash # Create the trigger influxdb3 create trigger \ --database weather \ --path "gh:influxdata/basic_transformation/basic_transformation.py" \ --trigger-spec "every:30m" \ --trigger-arguments 'measurement=raw_temps,window=1h,target_measurement=temps_fahrenheit,names_transformations=Temperature:"snake",values_transformations=temperature:"convert_degC_to_degF"' \ temp_converter # Write test data influxdb3 write \ --database weather \ "raw_temps,location=office Temperature=22.5" # Query transformed data (after trigger runs) influxdb3 query \ --database weather \ "SELECT * FROM temps_fahrenheit" ``` ### Expected output |location|temperature| time | |--------|-----------|--------------------| | office | 72.5 |2024-01-01T00:00:00Z| **Transformation details:** * Before: `Temperature=22.5` (Celsius) * After: `temperature=72.5` (Fahrenheit, field name converted to snake\_case) ### Example 2: Field name standardization Clean and standardize field names from various sensors: ```bash # Create trigger with multiple transformations influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/basic_transformation/basic_transformation.py" \ --trigger-spec "all_tables" \ --trigger-arguments 'measurement=raw_sensors,target_measurement=clean_sensors,names_transformations=.*:"remove_special_chars snake collapse_underscore trim_underscore"' \ field_cleaner # Write data with inconsistent field names influxdb3 write \ --database sensors \ "raw_sensors,device=sensor1 \"Room Temperature\"=20.1,\"__Humidity_%\"=45.2" # Query cleaned data influxdb3 query \ --database sensors \ "SELECT * FROM clean_sensors" ``` ### Expected output |device |room\_temperature|humidity| time | |-------|-----------------|--------|--------------------| |sensor1| 20.1 | 45.2 |2024-01-01T00:00:00Z| **Transformation details:** * Before: `"Room Temperature"=20.1`, `"__Humidity_%"=45.2` * After: `room_temperature=20.1`, `humidity=45.2` (field names standardized) ### Example 3: Custom string replacements Replace specific strings in field values: ```bash # Create trigger with custom replacements influxdb3 create trigger \ --database inventory \ --path "gh:influxdata/basic_transformation/basic_transformation.py" \ --trigger-spec "every:1d" \ --trigger-arguments 'measurement=products,window=7d,target_measurement=products_updated,values_transformations=status:"status_replace",custom_replacements=status_replace:"In Stock=available.Out of Stock=unavailable"' \ status_updater ``` ## Using TOML Configuration Files This plugin supports using TOML configuration files to specify all plugin arguments. This is useful for complex configurations or when you want to version control your plugin settings. ### Important Requirements **To use TOML configuration files, you must set the `PLUGIN_DIR` environment variable in the InfluxDB 3 Enterprise host environment.** This is required in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise: * `--plugin-dir` tells InfluxDB 3 Enterprise where to find plugin Python files * `PLUGIN_DIR` environment variable tells the plugins where to find TOML configuration files ### Setting Up TOML Configuration 1. **Start InfluxDB 3 Enterprise with the PLUGIN\_DIR environment variable set**: ``` PLUGIN_DIR=~/.plugins influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. **Copy the example TOML configuration file to your plugin directory**: ``` cp basic_transformation_config_scheduler.toml ~/.plugins/ # or for data writes: cp basic_transformation_config_data_writes.toml ~/.plugins/ ``` 3. **Edit the TOML file** to match your requirements. The TOML file contains all the arguments defined in the plugin’s argument schema (see the JSON schema in the docstring at the top of basic\_transformation.py). 4. **Create a trigger using the `config_file_path` argument**: ``` influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/basic_transformation/basic_transformation.py" \ --trigger-spec "every:1d" \ --trigger-arguments config_file_path=basic_transformation_config_scheduler.toml \ basic_transform_trigger ``` ## Code overview ### Files * `basic_transformation.py`: The main plugin code containing handlers for scheduled tasks and data write transformations * `basic_transformation_config_data_writes.toml`: Example TOML configuration file for data write triggers * `basic_transformation_config_scheduler.toml`: Example TOML configuration file for scheduled triggers ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled transformation tasks. Queries historical data within the specified window and applies transformations. Key operations: 1. Parses configuration from arguments 2. Queries source measurement with filters 3. Applies name and value transformations 4. Writes transformed data to target measurement #### `process_writes(influxdb3_local, table_batches, args)` Handles real-time transformation during data writes. Processes incoming data batches and applies transformations before writing. Key operations: 1. Filters relevant table batches 2. Applies transformations to each row 3. Writes to target measurement immediately #### `apply_transformations(value, transformations)` Core transformation engine that applies a chain of transformations to a value. Supported transformations: **Case conversions:** * `lower` - Convert to lowercase * `upper` - Convert to uppercase * `snake` - Convert to snake\_case * `camel` - Convert to camelCase * `pascal` - Convert to PascalCase * `kebab` - Convert to kebab-case * `title` - Convert to Title Case * `capitalize_first` - Capitalize first letter only * `capitalize_words` - Capitalize each word **String cleaning and normalization:** * `space_to_underscore` - Replace spaces with underscores * `remove_space` - Remove all spaces * `alnum_underscore_only` - Keep only alphanumeric and underscore characters * `collapse_underscore` - Collapse multiple underscores into one * `trim_underscore` - Remove leading/trailing underscores * `normalize_whitespace` - Normalize whitespace to single spaces * `normalize_dashes` - Normalize dashes and underscores to dashes * `normalize_underscores` - Normalize dashes and spaces to underscores **Character filtering:** * `remove_digits` - Remove all digits * `remove_punctuation` - Remove punctuation marks * `keep_alphanumeric` - Keep only letters and numbers * `remove_special_chars` - Remove special characters (keep letters, numbers, spaces, \_, -) **String extraction and filtering:** * `extract_numbers_only` - Extract only numeric characters * `extract_letters_only` - Extract only alphabetic characters **Mathematical operations (for numeric values):** * `abs` - Absolute value * `round2` - Round to 2 decimal places * `sqrt` - Square root * `ln` - Natural logarithm * `floor` - Round down to nearest integer * `ceil` - Round up to nearest integer **Value conversion and clamping:** * `to_percentage` - Multiply by 100 (convert to percentage) * `from_percentage` - Divide by 100 (convert from percentage) * `clamp_min_zero` - Limit minimum value to zero * `clamp_max_hundred` - Limit maximum value to 100 * `boolean_to_int` - Convert boolean values to 1/0 **Other operations:** * `reverse` - Reverse the string * Unit conversions: `convert__to_` * Custom replacements: User-defined string substitutions ## Troubleshooting ### Common issues #### Issue: Transformations not applying **Solution**: Check that field names match exactly (case-sensitive). Use regex patterns for flexible matching: ```bash --trigger-arguments 'custom_regex=temp_fields:"temp%",values_transformations=temp_fields:"convert_degC_to_degF"' ``` #### Issue: “Permission denied” errors in logs **Solution**: Ensure the plugin file has execute permissions: ```bash chmod +x ~/.plugins/basic_transformation.py ``` #### Issue: Unit conversion failing **Solution**: Verify unit names are valid pint units. Common units: * Temperature: `degC`, `degF`, `degK` * Length: `meter`, `foot`, `inch` * Time: `second`, `minute`, `hour` #### Issue: No data in target measurement **Solution**: 1. Check dry\_run is not set to “true” 2. Verify source measurement contains data 3. Check logs for errors: ``` influxdb3 query \ --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` ### Debugging tips 1. **Enable dry run** to test transformations: ``` --trigger-arguments 'dry_run=true,...' ``` 2. **Use specific time windows** for testing: ``` --trigger-arguments 'window=1h,...' ``` 3. **Check field names** in source data: ``` influxdb3 query --database mydb "SHOW FIELD KEYS FROM measurement" ``` ### Performance considerations * Field name caching reduces query overhead (1-hour cache) * Batch processing for scheduled tasks improves throughput * Retry mechanism (3 attempts) handles transient write failures * Use filters to process only relevant data ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. ## Schema requirements The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. > [!Warning] > #### Requires existing schema > > By design, the plugin returns an error if the schema doesn’t exist or doesn’t contain the expected columns. #### Related * [Basic transformation plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/basic_transformation) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[transformation](/influxdb3/enterprise/tags/transformation/)[data-processing](/influxdb3/enterprise/tags/data-processing/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Source measurement containing data to transform | | target_measurement | string | required | Destination measurement for transformed data | | target_database | string | current database | Database for storing transformed data | | dry_run | string | “false” | When “true”, logs transformations without writing | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | names_transformations | string | none | Field/tag name transformation rules. Format: 'field1:"transform1 transform2".field2:"transform3"' | | values_transformations | string | none | Field value transformation rules. Format: 'field1:"transform1".field2:"transform2"' | | custom_replacements | string | none | Custom string replacements. Format: 'rule_name:"find=replace"' | | custom_regex | string | none | Regex patterns for field matching. Format: 'pattern_name:"temp%"' | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | window | string | required (scheduled only) | Historical data window. Format: (for example, “30d”, “1h”) | | included_fields | string | all fields and tags | Dot-separated list of fields and tags to include (for example, “temp.humidity.location”) | | excluded_fields | string | none | Dot-separated list of fields and tags to exclude | | filters | string | none | Query filters. Format: 'field:"operator value"' | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | | location | temperature | time | | --- | --- | --- | | location | temperature | time | | office | 72.5 | 2024-01-01T00:00:00Z | | device | room_temperature | humidity | time | | --- | --- | --- | --- | | device | room_temperature | humidity | time | | sensor1 | 20.1 | 45.2 | 2024-01-01T00:00:00Z | --- # Bird data simulator plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/bird-data-simulator/ The Bird Tracking Simulator Plugin generates a stream of synthetic bird telemetry for demos, testing, and sample-data workflows. On its first run, it creates a persistent flock of named birds, assigns each bird a species, natural range, starting location, heading, and healthy body temperature, then stores that flock in the Processing Engine cache. Each scheduled call advances the flock with sinusoidal flight speed, gentle heading changes, latitude and longitude updates, and temperature jitter. The plugin intentionally exposes only volume controls. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. The trigger interval plus the options below control how much data the plugin writes. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Optional parameters There are no required parameters. All configuration parameters control data volume only. | Parameter | Type |Default| Description | |-----------------|-------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `bird_count` |integer| `25` | Number of persistent simulated birds to track | |`points_per_bird`|integer| `1` |Number of movement points to emit for each bird on each scheduled call. When greater than `1`, timestamps are evenly spaced across the elapsed time since the previous call.| ### TOML configuration This plugin does not expose TOML configuration. Use the two inline volume options above and the trigger interval to control output volume. ## Requirements ### Data requirements This plugin does not require incoming writes or source measurements. It generates data directly from a scheduled trigger. The generated flock state is stored in the Processing Engine’s trigger-specific cache. Changing `bird_count` creates a new cached flock with the requested size. ### Schema requirements The plugin writes to the `bird_tracking` measurement. Tags: * `species`: common species name, such as `American Robin` * `name`: generated name for the individual bird Fields: * `body_temp`: body temperature in degrees Celsius * `longitude`: current longitude in decimal degrees * `latitude`: current latitude in decimal degrees * `speed`: current speed in miles per hour * `heading`: current heading in degrees, where `0` is north and `90` is east ### Species metadata The plugin embeds 20 United States bird species with simplified ranges, weight ranges, healthy body temperature ranges, and approximate top flight speeds directly in `bird_data_simulator.py`. The species catalog uses [Cornell Lab All About Birds](https://www.allaboutbirds.org/guide/) species accounts and range maps as the primary reference for species presence, range, habitat, and measurements. General healthy body temperature ranges are based on published avian veterinary reference values such as the [Merck Veterinary Manual normal temperature table](https://www.merckvetmanual.com/reference-values-and-conversion-tables/reference-guides/normal-rectal-temperature-ranges). ### Software requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Python packages**: * `Faker` ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install `Faker` into the Processing Engine Python environment: ``` influxdb3 install package Faker ``` ## Trigger setup ### Basic scheduled trigger ```bash influxdb3 create trigger \ --database sample_data \ --path "gh:influxdata/bird_data_simulator/bird_data_simulator.py" \ --trigger-spec "every:1s" \ bird_tracking ``` ### Larger flock ```bash influxdb3 create trigger \ --database sample_data \ --path "gh:influxdata/bird_data_simulator/bird_data_simulator.py" \ --trigger-spec "every:1s" \ --trigger-arguments bird_count=100 \ bird_tracking_large ``` ### More points per bird ```bash influxdb3 create trigger \ --database sample_data \ --path "gh:influxdata/bird_data_simulator/bird_data_simulator.py" \ --trigger-spec "every:10s" \ --trigger-arguments bird_count=50,points_per_bird=10 \ bird_tracking_dense ``` ## Example usage ### Generate bird telemetry ```bash # Create a small flock that writes once per second. influxdb3 create trigger \ --database sample_data \ --path "gh:influxdata/bird_data_simulator/bird_data_simulator.py" \ --trigger-spec "every:1s" \ --trigger-arguments bird_count=10 \ bird_tracking_demo # Query generated points after the trigger runs. influxdb3 query \ --database sample_data \ "SELECT * FROM bird_tracking ORDER BY time DESC LIMIT 5" ``` ### Expected output ```text species | name | body_temp | longitude | latitude | speed | heading | time -----------------|-------|-----------|-------------|-----------|-------|---------|--------------------- American Robin | Willa | 41.822 | -83.182337 | 39.912884 | 21.4 | 83.2 | 2026-04-29T12:00:04Z Cactus Wren | Felix | 42.117 | -111.913552 | 33.382018 | 8.7 | 244.9 | 2026-04-29T12:00:04Z Florida Scrub-Jay| Pearl | 41.603 | -81.224901 | 28.399102 | 12.1 | 11.6 | 2026-04-29T12:00:04Z ``` ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Bird data simulator plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/bird_data_simulator) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | bird_count | integer | 25 | Number of persistent simulated birds to track | | points_per_bird | integer | 1 | Number of movement points to emit for each bird on each scheduled call. When greater than 1 , timestamps are evenly spaced across the elapsed time since the previous call. | --- # Chronos forecasting plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/chronos-forecasting/ ⚡ scheduled, http 🏷️ forecasting, machine-learning, time-series, deep-learning 🔧 InfluxDB 3 Enterprise > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The Chronos Forecasting Plugin forecasts time series stored in InfluxDB 3 Enterprise with Amazon’s pre-trained Chronos models from HuggingFace, with no training. A scheduled trigger forecasts on every run and writes the forecast back to InfluxDB; an HTTP trigger answers each request with the forecast. * **Zero-shot inference**: Chronos-Bolt, Chronos-2 and original Chronos (T5) models forecast any series without training. * **A field or a query**: forecast a numeric field of a measurement, or the result of any SQL query, such as a chart’s aggregating query. * **Per series**: `group_by` splits the data by tags, and every series is forecast in one model call. * **Prediction intervals**: any interval levels the model can produce, as `lower_` / `upper_`. * **Covariates**: Chronos-2 models take extra fields as past covariates. * **Backtest**: on request, the forecast is checked on the latest history in the same model call: the share of points inside each interval and the mean absolute error. * **Shared forecasting interface**: requests, responses, errors and stored rows follow the contract InfluxDB 3 Explorer uses for every forecasting plugin, and [`forecast_interface.json`](forecast_interface.json) describes what this plugin supports. ## Software requirements * **InfluxDB 3 Enterprise**: 3.8.2 or later, with the Processing Engine enabled. * **Python packages**: * `chronos-forecasting` (Amazon Chronos model pipeline) * `torch` (PyTorch for model inference) * `influxdata-plugin-utils` (configuration loading) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package chronos-forecasting influxdb3 install package torch influxdb3 install package influxdata-plugin-utils ``` ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ## Scheduled trigger A scheduled trigger forecasts on every run of its schedule and writes the forecast to `target_measurement`. It returns nothing; a failure goes to the plugin log. ### Scheduled trigger parameters Set these with `--trigger-arguments` (CLI) or `trigger_arguments` (HTTP API) when creating the trigger, or in the[TOML file](#scheduled-trigger-toml-configuration). In trigger arguments every value is a string; see [Value formats](#value-formats). | Parameter | Type | Default | Description | |---------------------|---------------|---------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurement` | string | required¹ | Measurement to read | | `field` | string | required¹ | Numeric field to forecast | | `query` | string | required¹ | SQL whose result is forecast instead of `source_measurement` and `field`; see [Forecasting a query](#forecasting-a-query) | | `source_database` | string | the trigger’s database | Database to read from; see [Security](#security) | | `tag_filter` |string or table| none | Tags the rows must match: `host=a region=eu`, or a TOML table for several values per tag. Not with `query` | | `sql_filter` | string | none |SQL condition on tags and fields, without `WHERE`, AND-ed with `tag_filter` and inserted as is; see [Security](#security). Not with `query`| | `group_by` | list | none | Tags to split by; each distinct combination is its own series; see [Series](#series). Not with `query` | | `series_limit` | integer ≥ 1 | none | Forecast only the first N forecasts (series × fields) in the order of [Series](#series) and skip the rest; at most `max_series` | | `window` | duration | derived | History as a time window ending now; see [History](#history) | | `point_limit` | integer ≥ 10 | `512` without `window` | History as the most recent cells of each series’ grid, at most the model’s context; see [History](#history) | | `sampling_interval` | duration | none | Resample history to the mean per bucket; see [History](#history) | | `horizon_steps` | integer ≥ 1 | required² | Forecast length in points at the data spacing | | `horizon_duration` | duration | required² | Forecast length as a duration, converted to points by the data spacing | | `levels` |list of numbers| `0.5 0.8` | Interval levels, each between 0 and 1; a level the model cannot produce is not written | | `model` | string | `amazon/chronos-bolt-tiny` | HuggingFace model ID or a local path; see [Models](#models) | |`target_measurement` | string |`_forecasts.`| Measurement the forecast is written to; required when `query` reads several measurements | | `target_database` | string | the trigger’s database | Database the forecast is written to | | `forecast_id` | string | none | Tag `forecast_id` in every row, so forecasts of one series with different settings keep their own rows | | `covariate_fields` | list | none | Numeric fields other than `field` passed to a Chronos-2 model as past covariates; see [Models](#models). Not with `query` | | `max_series` | integer ≥ 1 | `100` | Most forecasts one run may produce (series × fields); more is refused unless `series_limit` keeps fewer | |`max_forecast_points`| integer ≥ 1 | `1024` | Most forecast points per series; a longer horizon is refused | | `config_file_path` | string | none | TOML file with more arguments; see [Scheduled trigger TOML configuration](#scheduled-trigger-toml-configuration) | ¹ Either `source_measurement` with `field`, or `query`. ² Exactly one of `horizon_steps` and `horizon_duration`. A scheduled trigger always writes and returns no response, so it refuses `write_results` and `backtest_points`. An unknown argument or an invalid value fails every run with `bad_request` in the log. ### Scheduled trigger environment variables Every parameter can also come from an environment variable of the InfluxDB server named`INFLUXDB3_CHRONOS_FORECASTING_` in upper case, for example `INFLUXDB3_CHRONOS_FORECASTING_MODEL` sets`model` for every trigger of the plugin. * The environment is the lowest layer: a trigger argument overrides it, and the TOML file overrides both. * `INFLUXDB3_CHRONOS_FORECASTING_CONFIG_FILE_PATH` names the TOML file when the trigger has no `config_file_path`argument. * Variables of the HTTP-only keys (`..._WRITE_RESULTS`, `..._BACKTEST_POINTS`) are ignored by a scheduled trigger. ### Scheduled trigger TOML configuration `config_file_path`, a trigger argument or the environment variable above, names a `.toml` file with more arguments. * The path is absolute or relative to the plugin directory: the first set of `PLUGIN_DIR`, `INFLUXDB3_PLUGIN_DIR` and the parent of the engine’s virtual environment. * The file’s keys override the trigger arguments; an argument the file does not set still applies. * Lists and tables are native: `group_by = ["host"]`, and `[tag_filter]` with several values per tag (`host = ["a", "b"]`). Keep the `[tag_filter]` table last: keys after a table header belong to the table. * A `query` or `sql_filter` with a comma goes here: the CLI splits `--trigger-arguments` at every comma. * An unknown key, `config_file_path` inside the file, `write_results` or `backtest_points` fails every run with`bad_request`. #### Example scheduled TOML configuration [chronos\_forecasting\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/chronos_forecasting/chronos_forecasting_scheduler.toml) lists every key a scheduled trigger accepts, with its default. A query, which has commas, set through a file: ```toml query = "SELECT date_bin(INTERVAL '5 minutes', time) AS time, host, avg(usage) AS usage FROM cpu WHERE time > now() - INTERVAL '6 hours' GROUP BY 1, host" horizon_steps = 12 series_limit = 5 ``` ### Create a scheduled trigger With trigger arguments: ```bash influxdb3 create trigger \ --database mydb \ --path chronos_forecasting.py \ --trigger-spec "every:5m" \ --trigger-arguments "source_measurement=cpu,field=usage,group_by=host,window=6h,sampling_interval=1min,horizon_steps=60" \ chronos_forecast_scheduled ``` With the TOML file: ```bash influxdb3 create trigger \ --database mydb \ --path chronos_forecasting.py \ --trigger-spec "every:5m" \ --trigger-arguments config_file_path=chronos_forecasting_scheduler.toml \ chronos_forecast_scheduled ``` A `cron:` spec takes six fields, seconds first (`cron:20 */15 * * * *`), which spreads the runs of several triggers. Tune the settings with on-demand requests to an [HTTP trigger](#http-trigger) first: the response shows `points_read`and `step_ms` for every series. ### Scheduled trigger example The trigger above writes 60 one-minute points per host to `_forecasts.cpu` on every run. Read the latest run of each series: ```sql SELECT time, host, forecast, lower_80, upper_80, step, run_time FROM ( SELECT *, max (run_time) OVER (PARTITION BY field, host) AS latest FROM "_forecasts.cpu" ) AS f WHERE run_time = latest ORDER BY host, time ``` Expected output (first rows): | time |host|forecast|lower\_80|upper\_80|step| run\_time | |-------------------|----|--------|---------|---------|----|--------------------| |2026-10-29T12:01:00| a | 48.92 | 46.50 | 51.37 | 1 |2026-10-29T12:00:05Z| |2026-10-29T12:02:00| a | 49.10 | 46.31 | 51.88 | 2 |2026-10-29T12:00:05Z| The rows are described in [Output data structure](#output-data-structure). With `forecast_id`, add `WHERE forecast_id = ''` to the inner query. ### Scheduled trigger errors A failed run writes nothing and logs `[] : ` at `ERROR`, with the codes of the[HTTP errors](#http-trigger-errors). A series with too few points logs `[] too_few_points for : ` at `WARN`, and the other series are written. Read the log: ```sql SELECT event_time, log_level, log_text FROM system.processing_engine_logs WHERE trigger_name = 'chronos_forecast_scheduled' ORDER BY event_time DESC LIMIT 20 ``` ## HTTP trigger An HTTP trigger answers each request with the forecast and writes it only with `write_results`. Only an admin token can call it. ### HTTP request parameters Send these as a JSON object in the body of a POST request. The trigger’s arguments and TOML file set defaults for the same keys; see[HTTP trigger arguments](#http-trigger-arguments). | Parameter | Type | Default | Description | |---------------------|-------------------------------|---------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurement` | string | required¹ | Measurement to read | | `field` | string | required¹ | Numeric field to forecast | | `query` | string | required¹ | SQL whose result is forecast instead of `source_measurement` and `field`; see [Forecasting a query](#forecasting-a-query) | | `source_database` | string | the trigger’s database | Database to read from; see [Security](#security) | | `tag_filter` | object or string | none | Tags the rows must match: `host=a region=eu`, or `{"host": ["a", "b"]}` in JSON and TOML. Not with `query` | | `sql_filter` | string | none |SQL condition on tags and fields, without `WHERE`, AND-ed with `tag_filter` and inserted as is; see [Security](#security). Not with `query`| | `group_by` | list | none | Tags to split by; each distinct combination is its own series; see [Series](#series). Not with `query` | | `series_limit` | integer ≥ 1 | none | Forecast only the first N forecasts (series × fields) in the order of [Series](#series) and skip the rest; at most `max_series` | | `window` | duration | derived | History as a time window ending now; see [History](#history) | | `point_limit` | integer ≥ 10 | `512` without `window` | History as the most recent cells of each series’ grid, at most the model’s context; see [History](#history) | | `sampling_interval` | duration | none | Resample history to the mean per bucket; see [History](#history) | | `horizon_steps` | integer ≥ 1 | required² | Forecast length in points at the data spacing | | `horizon_duration` | duration | required² | Forecast length as a duration, converted to points by the data spacing | | `levels` | list of numbers | `[0.5, 0.8]` | Interval levels, each between 0 and 1; a level the model cannot produce is `null` in the response and not written | | `model` | string | `amazon/chronos-bolt-tiny` | HuggingFace model ID or a local path; see [Models](#models) | | `write_results` | boolean | `false` | Also write the forecast, as the scheduled trigger does; see [Output data structure](#output-data-structure) | |`target_measurement` | string |`_forecasts.`| Measurement the forecast is written to; required when `query` reads several measurements | | `target_database` | string | the trigger’s database | Database the forecast is written to | | `forecast_id` | string | none | Tag `forecast_id` in every written row, so forecasts of one series with different settings keep their own rows | | `covariate_fields` | list | none | Numeric fields other than `field` passed to a Chronos-2 model as past covariates; see [Models](#models). Not with `query` | | `max_series` | integer ≥ 1 | `100` | Most forecasts one request may produce (series × fields); more is refused unless `series_limit` keeps fewer | |`max_forecast_points`| integer ≥ 1 | `1024` | Most forecast points per series; a longer horizon is refused | | `backtest_points` |integer 1…`max_forecast_points`| none | Hold out the last N points of each series, forecast them from the points before and return `backtest`; see [Backtest](#backtest) | ¹ Either `source_measurement` with `field`, or `query`. ² Exactly one of `horizon_steps` and `horizon_duration`. An unknown key, an invalid value, a body that is not a JSON object, or `config_file_path` answers `400 bad_request`. ### HTTP headers and query string Every request parameter can also come from a header `X-Influxdb3-Chronos-Forecasting-` with underscores written as hyphens (`X-Influxdb3-Chronos-Forecasting-Horizon-Steps` sets `horizon_steps`), or from the query string under its own name (`?horizon_steps=12`). Their values are strings; see [Value formats](#value-formats). * Header names are matched regardless of case. * A header with the plugin’s prefix that names no parameter answers `bad_request`; other headers, such as the client’s own, are ignored. * An unknown query-string parameter answers `bad_request`. * The body, then the headers, then the query string: each overrides the one before. ```bash curl -X POST "http://localhost:8181/api/v3/engine/forecast?horizon_steps=12" \ -H "Authorization: Bearer $INFLUXDB3_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -H "X-Influxdb3-Chronos-Forecasting-Levels: 0.5 0.9" \ -d '{"source_measurement": "cpu", "field": "usage", "tag_filter": {"host": "a"}}' ``` ### HTTP trigger arguments The trigger’s arguments hold defaults for every request: the same keys as the request, plus `config_file_path`. A request overrides an argument it names; an argument it does not name still applies. Two choices are made as a whole by the highest layer that sets either side: * `query` against `source_measurement`, `field`, `tag_filter`, `sql_filter`, `group_by` and `covariate_fields`; * `horizon_steps` against `horizon_duration`. So a trigger created with `source_measurement=cpu field=usage horizon_steps=12` forecasts a query when the body sends`query`, and 1 h ahead when it sends `horizon_duration: "1h"`. Both sides in one layer answer `bad_request`. An unknown trigger argument fails every request with `bad_request`. ### HTTP trigger environment variables Every parameter can also come from an environment variable of the InfluxDB server named`INFLUXDB3_CHRONOS_FORECASTING_` in upper case, for example `INFLUXDB3_CHRONOS_FORECASTING_MAX_SERIES`. * The layers, lowest first: the environment, the trigger arguments, the TOML file, then the request’s body, headers and query string. * `INFLUXDB3_CHRONOS_FORECASTING_CONFIG_FILE_PATH` names the TOML file when the trigger has no `config_file_path`argument. ### HTTP trigger TOML configuration `config_file_path`, a trigger argument or the environment variable above, names a `.toml` file of defaults; a request can never name it. * The path is absolute or relative to the plugin directory: the first set of `PLUGIN_DIR`, `INFLUXDB3_PLUGIN_DIR` and the parent of the engine’s virtual environment. * The file’s keys override the trigger arguments, and the request overrides the file. * Every request key is accepted, with native lists and tables; keep the `[tag_filter]` table last. * An unknown key or `config_file_path` inside the file fails every request with `bad_request`. #### Example HTTP TOML configuration [chronos\_forecasting\_http.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/chronos_forecasting/chronos_forecasting_http.toml) lists every key an HTTP trigger accepts, with its default: the endpoint’s model, levels and limits are set, the rest is left for each request. ```bash influxdb3 create trigger \ --database mydb \ --path chronos_forecasting.py \ --trigger-spec "request:forecast" \ --trigger-arguments config_file_path=chronos_forecasting_http.toml \ chronos_forecast_http ``` ### Create an HTTP trigger ```bash influxdb3 create trigger \ --database mydb \ --path chronos_forecasting.py \ --trigger-spec "request:forecast" \ chronos_forecast_http ``` The trigger answers at `/api/v3/engine/forecast`. ### HTTP trigger examples #### Forecast a field, per host ```bash curl -X POST "http://localhost:8181/api/v3/engine/forecast" \ -H "Authorization: Bearer $INFLUXDB3_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "source_measurement": "cpu", "field": "usage", "tag_filter": {"region": "eu"}, "group_by": ["host"], "window": "6h", "sampling_interval": "30s", "horizon_duration": "1h", "levels": [0.5, 0.8] }' ``` Expected output: HTTP `200`, one result per series × field. A series without enough history carries its own error instead of a forecast. ```json { "status": "ok", "model": "amazon/chronos-bolt-tiny", "levels": [ 0.5, 0.8 ], "series_total": 2, "results": [ { "field": "usage", "tags": { "host": "a", "region": "eu" }, "points_read": 720, "step_ms": 30000, "forecast": [ { "timestamp": 1700003630000, "forecast": 1.3, "lower_50": 1.1, "upper_50": 1.5, "lower_80": 0.9, "upper_80": 1.7 } ] }, { "field": "usage", "tags": { "host": "c", "region": "eu" }, "status": "error", "code": "too_few_points", "message": "need >= 10 points, got 4" } ] } ``` | Field | Description | |-------------------------------------|----------------------------------------------------------------------------------------------------------------------------| | `status` | `"ok"` | | `model` | The model that ran | | `levels` | The returned levels: each point carries `lower_` and `upper_` for each | | `series_total` | Forecasts (series × fields) the request selects before `series_limit`; equals the number of results without it | | `results[].field`, `results[].tags` | The forecast field and the series tags; see [Series](#series) | | `results[].points_read` | Observed history points the model used: cells of the grid that hold data | | `results[].step_ms` | Spacing of the history and the forecast, in milliseconds | | `results[].forecast[]` |Points: `timestamp` (epoch ms), `forecast` (the median) and the interval bounds, `null` for a level the model cannot produce| | `results[].backtest` | With `backtest_points`; see [Backtest](#backtest) | |`results[].status`, `code`, `message`| Instead of a forecast, when the series cannot be forecast (`too_few_points`) | The forecast starts one step after the last history point. #### Forecast a query ```bash curl -X POST "http://localhost:8181/api/v3/engine/forecast" \ -H "Authorization: Bearer $INFLUXDB3_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "query": "SELECT date_bin(INTERVAL '"'"'5 minutes'"'"', time) AS time, host, avg(usage) AS usage FROM cpu WHERE time > now() - INTERVAL '"'"'6 hours'"'"' GROUP BY 1, host", "horizon_steps": 12, "series_limit": 5 }' ``` #### Multivariate forecast with Chronos-2 ```bash curl -X POST "http://localhost:8181/api/v3/engine/forecast" \ -H "Authorization: Bearer $INFLUXDB3_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{"source_measurement": "sensor_data", "field": "temperature", "model": "amazon/chronos-2", "covariate_fields": ["humidity", "pressure"], "horizon_steps": 64}' ``` ### Backtest With `backtest_points: N`, the plugin also checks the forecast on the latest history, in the same model call. It holds out the last N points of each series, forecasts them from the points before them with the same model, levels and history length, and compares. Each successful result then carries `backtest`: ```json { "backtest": { "points": 11, "coverage": { "50": 0.545, "80": 0.818 }, "mae": 2.21 } } ``` * `points`: the measured points compared; an empty bucket is skipped. * `coverage`: per level, the share of those points inside its interval, `null` for a level the model cannot produce. A share close to the level means the intervals are calibrated. * `mae`: the mean absolute difference between the central forecast and the measured points, in the field’s units. `backtest` is `null` when fewer than 10 points precede the held-out ones, or none of these is measured; the forecast itself is the same as without the check. The check reads N more points per series and takes about one more forecast per series. With N equal to the horizon, it measures accuracy over the same distance as the forecast; a small N gives a noisy estimate. N above `max_forecast_points` answers `bad_request`. ### HTTP trigger errors A request that fails as a whole answers with an HTTP error status and a JSON body: ```json { "status": "error", "code": "bad_request", "message": "set exactly one of horizon_steps and horizon_duration" } ``` | Code |HTTP | When | |-------------------|-----|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `bad_request` |`400`|Unknown or invalid parameter, unknown measurement, field or database, a query that does not plan, too many forecasts or forecast points, a series mixing tag sets without `sampling_interval`| |`model_unavailable`|`503`| The model failed to download or load | | `internal` |`500`| A query, the inference or the write failed, or a forecast row cannot be built | | `too_few_points` | — | Per series, in `results`, with the request still `200` | The plugin logs a `bad_request` at `WARN` and the other codes at `ERROR`. ## Forecasting details ### Value formats * In trigger arguments, headers and the query string every value is a string: lists are space-separated (`group_by=host region`, `levels=0.5 0.8`), and `tag_filter` is space-separated `tag=value` pairs with one value per tag. * The JSON body and the TOML file take native lists and objects. * **Durations** are a whole number and a unit: `s`, `min`, `h`, `d` or `w` (`30s`, `5min`, `6h`, `7d`). * The CLI splits `--trigger-arguments` at every comma, so a `query` or `sql_filter` with a comma (`host IN ('a', 'b')`) goes in the TOML file or the request. ### Models | Model |Interval levels|History used|Native horizon| |------------------------------------------------------|---------------|------------|--------------| |`amazon/chronos-bolt-tiny`, `-mini`, `-small`, `-base`| up to 0.8 |2048 points | 64 steps | | `amazon/chronos-2` | up to 0.98 |8192 points | 1024 steps | | original Chronos (`amazon/chronos-t5-*`) | any | 512 points | 64 steps | * A horizon past the native one is extended autoregressively, at lower quality, and the plugin logs a warning. * A level whose bounds fall outside the quantiles the model was trained on is left out rather than extrapolated: Chronos-Bolt is trained on quantiles 0.1–0.9, Chronos-2 on 0.01–0.99. * Chronos-T5 builds its intervals from 20 samples, so levels above 0.9 are close to the samples’ minimum and maximum. * `covariate_fields` needs a Chronos-2 model, checked once the model loads; a gap in a covariate is passed as missing. * The model loads on the first forecast that needs it and stays loaded for later runs; a request whose series all have too few points loads no model. ### Series * Without `group_by` all rows matching the filters form one series; with it, each distinct combination of the `group_by`tags is its own series. With `query`, each distinct combination of the key columns is. * A series’ `tags` hold the `group_by` tags and every other tag that all its rows carry with one value; a tag missing from some rows is left out. When the data changes which tags a series carries, for example a second host appears, the forecast goes to a new series. * `results` are ordered by the series’ `group_by` tags, or the query’s key columns, compared as text with a missing value last, then by field name.`series_limit: N` keeps the first N forecasts in this order and skips the rest without an error; `series_total` in the response counts them all, so a chart can say “the first 5 of 12”. In measurement mode skipped series are neither read nor checked for mixed tag sets; with `query` the result is read whole, then cut. The scheduled trigger logs how many it kept. * Without `sampling_interval`, a series whose rows come from several tag sets writing at the same time within its last`point_limit` points (all of `window` when only it is set) is refused with `bad_request`, for example several hosts without `group_by`, or the per-core rows of Telegraf’s `cpu` measurement: interleaved, they would form a saw-tooth. Add the tags the message names to `group_by`, pin them with `tag_filter`, or set `sampling_interval` to forecast their mean per bucket. A tag that changes value over time, such as a version after an upgrade, does not make a series mixed; one that flips back and forth, such as a status, does, so set `sampling_interval` for it. With `query` the plugin cannot see tag sets, so the query has to keep series apart. * When the forecast is written, a `group_by` tag or a query key column named like a column of the[stored rows](#output-data-structure) is refused with `bad_request`: as a tag it would break the row or merge series. ### History * The plugin reads at most the model’s context per series (the History used column of [Models](#models)); `window`and `point_limit` narrow it further. With neither, it keeps the last 512 cells. * Without `window`, a read in measurement mode covers `point_limit` buckets of `sampling_interval` back from now (at most 365 days, plus `backtest_points` and the bucket still filling), or the last day of raw points. Pick `sampling_interval` so that this covers several seasonal cycles: `1h` with the default 512 buckets is 21 days. With `query` there is no default window; see [Forecasting a query](#forecasting-a-query). * Chronos reads points as equally spaced, so history reaches the model on a grid. With `sampling_interval` the cells are its buckets, aligned like `date_bin`, and the bucket still filling is left out. Without it, the step is the median spacing of the points, rounded to a round unit about 1% of it (so 10 s, 1 h or 1 week stay exact), and each point goes to the nearest cell of a grid that keeps the rhythm most points follow. Points in one cell are averaged, and an empty cell is passed as a gap, which Chronos treats as missing. * A series needs at least 10 observed points; fewer answers `too_few_points` for that series, naming the window read. ### Forecasting a query With `query`, the plugin runs the SQL against `source_database` and forecasts what it returns: * **time**: the column named `time`, so the query aliases its timestamp `AS time`. No `time` column answers `bad_request`. * **series keys**: the text and boolean columns; each distinct combination is one series. * **values**: the numeric columns; each is forecast on its own, per series. A result without a numeric column answers `bad_request`. A timestamp column other than `time` (`max(time) AS last_seen`) reaches the plugin as an integer and is forecast as a number, so leave such columns out of the query. * **column types**: the Processing Engine passes `BIGINT`, `BIGINT UNSIGNED`, `DOUBLE`, `BOOLEAN`, text and timestamp columns to plugins. A query that returns another type (`INT`, `REAL`, `DECIMAL`, an array or a struct) fails with `internal`, so cast such a column, for example `CAST(x AS DOUBLE)`. * **source measurement**: taken from the query’s plan (`EXPLAIN`). When the query reads one measurement, it is the `source_measurement` tag and names the default `target_measurement`; with several, writing needs `target_measurement`. `window`, `point_limit` and `sampling_interval` apply to the query’s result. The plugin reads the whole result into memory before `point_limit` cuts it, so the query must bound its own time range (`WHERE time > now() - INTERVAL '6 hours'`), above all in a scheduled trigger, which runs it on every run. With `window`, the server filters the result by its `time` column, as`SELECT * FROM () WHERE time > now() - `; a query that cannot be wrapped this way, such as one with an integer `time`, is read whole and filtered by the plugin, which logs a warning. ### Output data structure The scheduled trigger, and an HTTP request with `write_results`, write one row per forecast point to`target_measurement`. Tags: * `field`: the forecast field * `model`: the full model ID, as in the response * `source_measurement`: the source measurement; with `query`, only when it reads one * `forecast_id`: when set * the series tags, as in the response `tags`, except one named like a tag or field of the row, which is left out Fields: * `forecast` (float): the central forecast, the median * `lower_`, `upper_` (float): interval bounds, for example `lower_80`, `upper_97.5` * `step` (integer): the forecast step, `1` for the first point after the history * `points_read` (integer): observed history points the model used, that is cells of the grid that hold data * `run_time` (string): when the run happened, UTC to the second, for example `2026-10-26T12:00:00Z` Timestamp: `time`, the forecast point’s time in nanoseconds. A later run overwrites the points it repeats: buckets of `sampling_interval` and the grid of raw points keep their timestamps from run to run. A longer horizon from an earlier run remains, so read the latest `run_time`. Two triggers that forecast one series with different settings, for example different horizons, write to the same rows; give each its own `forecast_id` and read by it. ### Security * `sql_filter` is inserted into the query as is. It can reference any table of the source database, for example in a subquery. * The plugin reads `source_database` and writes `target_database` with the Processing Engine’s rights, not the caller’s. Only an admin token can call an HTTP trigger, so whoever calls it can already read every database. ### Forecast interface manifest [`forecast_interface.json`](forecast_interface.json) describes this plugin under the shared forecasting interface, contract version 1: the engine’s name and description, its modes, filters, history and horizon controls, models with their labels and limits, and advanced parameters. InfluxDB 3 Explorer reads it from the plugin’s registry package to configure its forecasting UI. The plugin never reads it: it validates every request on its own and runs any model the library can load. ## Code overview ### Key functions * **`process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None)`**: HTTP entry point; returns the response body and its HTTP status. * **`process_scheduled_call(influxdb3_local, call_time, args=None)`**: scheduled entry point; writes every run and logs errors with their code. * **`_load_settings(args, names, request_layers)`** / **`_resolve(cfg, scheduled)`**: merge every settings source with`influxdata-plugin-utils`, resolve the choices between keys by the highest source, validate each value, and check the parameters against each other, including the horizon against `max_forecast_points` when its length in points does not depend on the data. * **`_measurement_plan(influxdb3_local, req)`** / **`_read_measurement(influxdb3_local, req, plan, limit)`**: check the names against the schema, find the series and their single-valued tags with one aggregate query (which also enforces`max_series` and `series_limit`, keeping only the chosen series with `_chosen_series`), and refuse a series mixing tag sets that write at the same time, before the model loads; then read the history of every series in one query, at most`limit` points each, averaging rows that share a timestamp. * **`_query_measurements(influxdb3_local, req)`** / **`_read_query(influxdb3_local, req)`**: plan the query with`EXPLAIN` for its source measurement, run it before the model loads (with `window` as a filter the server applies when the query can be wrapped), split the result into series by column type, sort them, and apply `series_limit`. * **`_history_limit(pipeline, req)`** / **`_regularize(series, req, limit)`**: cap the history at `point_limit` and the model’s context, and lay it on a grid: the `sampling_interval` buckets without the one still filling, or, for raw points, a grid at their median spacing that keeps their rhythm (`_grid_anchor`); empty cells are gaps. * **`_pipeline_for(influxdb3_local, req, task_id)`**: loads the model when a series has enough points, either from the row counts or, when rows arrived between two queries, from the history read, and checks it for covariates. * **`_quantile_plan(pipeline, levels)`** / **`_predict(pipeline, inputs, horizon, quantiles)`**: turn each level `L`into the quantiles `(1 − L) / 2` and `(1 + L) / 2`, and forecast every series in one `predict_quantiles` call. * **`_split_backtest(series, limit, held_out)`** / **`_backtest_result(matrix, actual, level_columns, median)`**: hold out the last `backtest_points` cells as the backtest’s input of the same length as the forecast’s, and score its forecast against them: the share inside each level and the MAE. * **`_load_model(model_id, influxdb3_local, task_id)`**: loads a model once and keeps the two most recently used ones for later runs. The module is reloaded on every run, so the loaded models and a load lock live in a module registered in`sys.modules`. Loads are serialized: transformers’ `from_pretrained` swaps process-wide functions while it builds a model, and overlapping loads could leave weight tying switched off for good. ### Docker environment notes The plugin sets environment variables at import time for Docker compatibility: * `USER=influxdb3` — HuggingFace Hub requires a username * `HF_HOME=/tmp/hf_cache` — writable cache directory for model downloads * `HOME=/tmp` — fallback home directory for unmapped UIDs * `TORCHDYNAMO_DISABLE=1` — disables `torch.compile` to avoid conflicts in embedded Python * `HF_HUB_DISABLE_PROGRESS_BARS=1`, `TQDM_DISABLE=1` — disable tqdm/HuggingFace progress bars These are no-ops when running outside Docker with a normal user environment. ## Troubleshooting ### Common issues **`model_unavailable`** * The first run downloads the model from HuggingFace, so the InfluxDB host needs internet access and a writable`HF_HOME`; in Docker the plugin sets `HF_HOME=/tmp/hf_cache`. * For air-gapped environments, pre-download the model and set `model` to its local path. * If the message says the model loaded with weights missing, another plugin loaded a transformers model at the same moment; restart InfluxDB. **`too_few_points`** * A series needs at least 10 observed points in the window the message names; see [History](#history) for the default window. Widen `window`, set `sampling_interval` for sparse data, or raise `point_limit`. **`bad_request: rows of the series ... come from several tag sets writing at the same time`** * Without `sampling_interval`, rows of several tag sets of one series would interleave. Add the tags the message names to `group_by`, pin them with `tag_filter`, or set `sampling_interval` to forecast their mean per bucket. **`bad_request: ... more forecasts than max_series`** * `group_by` or a query produced more series × fields than `max_series`. Set `series_limit` to forecast the first ones, narrow the request with `tag_filter` or `sql_filter`, or raise`max_series`. **`bad_request: ... more than max_forecast_points`** * The horizon in points is longer than `max_forecast_points`; `horizon_duration` is converted by the data spacing, so a fine spacing makes many points. Shorten the horizon, set `sampling_interval`, or raise `max_forecast_points`. **`bad_request: ... clashes with a column of the stored forecast rows`** * A `group_by` tag or a query key column is named like a column the plugin writes, such as `step` or `model`. Forecast with a query that renames it, for example `SELECT time, step AS process_step, usage FROM process`, or forecast without writing. **`internal: the query failed: unsupported Arrow type for Python conversion`** * The query returns a column type the Processing Engine does not pass to plugins, such as `INT`, `REAL`, `DECIMAL`, an array or a struct. Cast it to `BIGINT`, `DOUBLE` or `VARCHAR` in the query. **`bad_request: no time column in query result`** * Alias the query’s timestamp `AS time`, for example `date_bin(INTERVAL '5 minutes', time) AS time`. **`internal: forecast computed but its rows cannot be built`** * The engine’s `LineBuilder` refuses some names, such as a measurement, tag or column name with a space. Rename it in the query with an alias, or forecast without writing. **`bad_request` for covariates** * `covariate_fields` needs a Chronos-2 model, checked once the model loads, and does not apply to `query`. **A scheduled trigger fails at the first comma** * `--trigger-arguments` splits at every comma, so a `query` or `sql_filter` with one is cut or refused; set it in the TOML file. **Slow inference** * `chronos-bolt-tiny` (\~8M parameters) forecasts in milliseconds on CPU; `amazon/chronos-2` takes seconds. * Only the first run after a restart loads the model; later runs reuse it. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Chronos forecasting plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/chronos_forecasting) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_measurement | string | required¹ | Measurement to read | | field | string | required¹ | Numeric field to forecast | | query | string | required¹ | SQL whose result is forecast instead of source_measurement and field ; see Forecasting a query | | source_database | string | the trigger’s database | Database to read from; see Security | | tag_filter | string or table | none | Tags the rows must match: host=a region=eu , or a TOML table for several values per tag. Not with query | | sql_filter | string | none | SQL condition on tags and fields, without WHERE , AND-ed with tag_filter and inserted as is; see Security . Not with query | | group_by | list | none | Tags to split by; each distinct combination is its own series; see Series . Not with query | | series_limit | integer ≥ 1 | none | Forecast only the first N forecasts (series × fields) in the order of Series and skip the rest; at most max_series | | window | duration | derived | History as a time window ending now; see History | | point_limit | integer ≥ 10 | 512 without window | History as the most recent cells of each series’ grid, at most the model’s context; see History | | sampling_interval | duration | none | Resample history to the mean per bucket; see History | | horizon_steps | integer ≥ 1 | required² | Forecast length in points at the data spacing | | horizon_duration | duration | required² | Forecast length as a duration, converted to points by the data spacing | | levels | list of numbers | 0.5 0.8 | Interval levels, each between 0 and 1; a level the model cannot produce is not written | | model | string | amazon/chronos-bolt-tiny | HuggingFace model ID or a local path; see Models | | target_measurement | string | _forecasts. | Measurement the forecast is written to; required when query reads several measurements | | target_database | string | the trigger’s database | Database the forecast is written to | | forecast_id | string | none | Tag forecast_id in every row, so forecasts of one series with different settings keep their own rows | | covariate_fields | list | none | Numeric fields other than field passed to a Chronos-2 model as past covariates; see Models . Not with query | | max_series | integer ≥ 1 | 100 | Most forecasts one run may produce (series × fields); more is refused unless series_limit keeps fewer | | max_forecast_points | integer ≥ 1 | 1024 | Most forecast points per series; a longer horizon is refused | | config_file_path | string | none | TOML file with more arguments; see Scheduled trigger TOML configuration | | time | host | forecast | lower_80 | upper_80 | step | run_time | | --- | --- | --- | --- | --- | --- | --- | | time | host | forecast | lower_80 | upper_80 | step | run_time | | 2026-10-29T12:01:00 | a | 48.92 | 46.50 | 51.37 | 1 | 2026-10-29T12:00:05Z | | 2026-10-29T12:02:00 | a | 49.10 | 46.31 | 51.88 | 2 | 2026-10-29T12:00:05Z | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_measurement | string | required¹ | Measurement to read | | field | string | required¹ | Numeric field to forecast | | query | string | required¹ | SQL whose result is forecast instead of source_measurement and field ; see Forecasting a query | | source_database | string | the trigger’s database | Database to read from; see Security | | tag_filter | object or string | none | Tags the rows must match: host=a region=eu , or {"host": ["a", "b"]} in JSON and TOML. Not with query | | sql_filter | string | none | SQL condition on tags and fields, without WHERE , AND-ed with tag_filter and inserted as is; see Security . Not with query | | group_by | list | none | Tags to split by; each distinct combination is its own series; see Series . Not with query | | series_limit | integer ≥ 1 | none | Forecast only the first N forecasts (series × fields) in the order of Series and skip the rest; at most max_series | | window | duration | derived | History as a time window ending now; see History | | point_limit | integer ≥ 10 | 512 without window | History as the most recent cells of each series’ grid, at most the model’s context; see History | | sampling_interval | duration | none | Resample history to the mean per bucket; see History | | horizon_steps | integer ≥ 1 | required² | Forecast length in points at the data spacing | | horizon_duration | duration | required² | Forecast length as a duration, converted to points by the data spacing | | levels | list of numbers | [0.5, 0.8] | Interval levels, each between 0 and 1; a level the model cannot produce is null in the response and not written | | model | string | amazon/chronos-bolt-tiny | HuggingFace model ID or a local path; see Models | | write_results | boolean | false | Also write the forecast, as the scheduled trigger does; see Output data structure | | target_measurement | string | _forecasts. | Measurement the forecast is written to; required when query reads several measurements | | target_database | string | the trigger’s database | Database the forecast is written to | | forecast_id | string | none | Tag forecast_id in every written row, so forecasts of one series with different settings keep their own rows | | covariate_fields | list | none | Numeric fields other than field passed to a Chronos-2 model as past covariates; see Models . Not with query | | max_series | integer ≥ 1 | 100 | Most forecasts one request may produce (series × fields); more is refused unless series_limit keeps fewer | | max_forecast_points | integer ≥ 1 | 1024 | Most forecast points per series; a longer horizon is refused | | backtest_points | integer 1… max_forecast_points | none | Hold out the last N points of each series, forecast them from the points before and return backtest ; see Backtest | | Field | Description | | --- | --- | | Field | Description | | status | "ok" | | model | The model that ran | | levels | The returned levels: each point carries lower_ and upper_ for each | | series_total | Forecasts (series × fields) the request selects before series_limit ; equals the number of results without it | | results[].field , results[].tags | The forecast field and the series tags; see Series | | results[].points_read | Observed history points the model used: cells of the grid that hold data | | results[].step_ms | Spacing of the history and the forecast, in milliseconds | | results[].forecast[] | Points: timestamp (epoch ms), forecast (the median) and the interval bounds, null for a level the model cannot produce | | results[].backtest | With backtest_points ; see Backtest | | results[].status , code , message | Instead of a forecast, when the series cannot be forecast ( too_few_points ) | | Code | HTTP | When | | --- | --- | --- | | Code | HTTP | When | | bad_request | 400 | Unknown or invalid parameter, unknown measurement, field or database, a query that does not plan, too many forecasts or forecast points, a series mixing tag sets without sampling_interval | | model_unavailable | 503 | The model failed to download or load | | internal | 500 | A query, the inference or the write failed, or a forecast row cannot be built | | too_few_points | — | Per series, in results , with the request still 200 | | Model | Interval levels | History used | Native horizon | | --- | --- | --- | --- | | Model | Interval levels | History used | Native horizon | | amazon/chronos-bolt-tiny , -mini , -small , -base | up to 0.8 | 2048 points | 64 steps | | amazon/chronos-2 | up to 0.98 | 8192 points | 1024 steps | | original Chronos ( amazon/chronos-t5-* ) | any | 512 points | 64 steps | --- # Downsampler plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/downsampler/ The Downsampler Plugin enables time-based data aggregation and downsampling in InfluxDB 3 Enterprise. Reduce data volume by aggregating measurements over specified time intervals using functions like avg, sum, min, max, median, count, stddev, first\_value, last\_value, var, or approx\_median. The plugin supports both scheduled batch processing of historical data and on-demand downsampling through HTTP requests. Each downsampled record includes metadata about the original data points compressed. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Scheduled trigger parameters | Parameter | Type | Default | Description | |---------------------|-------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurement` |string | required | Source measurement containing data to downsample | |`target_measurement` |string | required | Destination measurement for downsampled data | | `window` |string | required | Time window for each downsampling job. Format: `` (for example, “1h”, “1d”) | | `interval` |string | “10min” | Time interval for downsampling. Format: `` (for example, “10min”, “2h”, “1d”) | | `calculations` |string | “avg” | Aggregation functions. Single function or dot-separated field:aggregation pairs | | `specific_fields` |string |all fields| Dot-separated list of fields to downsample (for example, “co.temperature”) | | `excluded_fields` |string | none | Dot-separated list of fields and tags to exclude from downsampling results | | `tag_values` |string | none | Tag filters. Format: `tag:value1@value2@value3` for multiple values | | `offset` |string | “0” | Time offset to apply to the window | | `target_database` |string |“default” | Database for storing downsampled data | | `max_retries` |integer| 5 | Maximum number of retries for write operations | |`enable_full_logging`|boolean| false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging.| ### HTTP request parameters Send these parameters as JSON in the HTTP POST request body: | Parameter | Type | Default | Description | |---------------------|------------|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurement` | string | required | Source measurement containing data to downsample | |`target_measurement` | string | required | Destination measurement for downsampled data | | `interval` | string | “10min” | Time interval for downsampling. Format: `` (for example, “10min”, “2h”, “1d”) | | `batch_size` | string | “30d” | Time interval for each HTTP backfill batch | | `calculations` |string/array| “avg” | Aggregation functions. Use `"avg"` for all fields, or an array of `[field, aggregation]` pairs | | `specific_fields` | array | all fields | List of fields to downsample | | `excluded_fields` | array | none | List of fields and tags to exclude from downsampling results | | `tag_values` | object | none | Tag filters as an object mapping tag names to lists of values | | `target_database` | string | “default” | Database for storing downsampled data | | `max_retries` | integer | 5 | Maximum number of retries for write operations | | `backfill_start` | string |oldest point| ISO 8601 datetime with timezone for the start of the HTTP backfill window | | `backfill_end` | string |current time| ISO 8601 datetime with timezone for the end of the HTTP backfill window | |`enable_full_logging`| boolean | false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging.| ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. #### Example TOML configuration [downsampling\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/downsampler/downsampling_config_scheduler.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Schema management Each downsampled record includes three additional metadata columns: * `record_count`: the number of original points compressed into this single downsampled row * `time_from`: the minimum timestamp among the original points in the interval * `time_to`: the maximum timestamp among the original points in the interval ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. No additional Python packages required for this plugin. ## Trigger setup ### Scheduled downsampling Run downsampling periodically on historical data: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/downsampler/downsampler.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'source_measurement=cpu_metrics,target_measurement=cpu_hourly,interval=1h,window=6h,calculations=avg,specific_fields=usage_user.usage_system' \ cpu_hourly_downsample ``` ### On-demand downsampling Trigger downsampling via HTTP requests: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/downsampler/downsampler.py" \ --trigger-spec "request:downsample" \ downsample_api ``` ## Example usage ### Example 1: CPU metrics hourly aggregation Downsample CPU usage data from 1-minute intervals to hourly averages: ```bash # Create the trigger influxdb3 create trigger \ --database system_metrics \ --path "gh:influxdata/downsampler/downsampler.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'source_measurement=cpu,target_measurement=cpu_hourly,interval=1h,window=6h,calculations=avg,specific_fields=usage_user.usage_system.usage_idle' \ cpu_hourly_downsample # Write test data influxdb3 write \ --database system_metrics \ "cpu,host=server1 usage_user=45.2,usage_system=12.1,usage_idle=42.7" # Query downsampled data (after trigger runs) influxdb3 query \ --database system_metrics \ "SELECT * FROM cpu_hourly WHERE time >= now() - 1d" ``` ### Expected output | host |usage\_user|usage\_system|usage\_idle|record\_count| time\_from | time\_to | time | |-------|-----------|-------------|-----------|-------------|--------------------|--------------------|--------------------| |server1| 44.8 | 11.9 | 43.3 | 60 |2024-01-01T00:00:00Z|2024-01-01T00:59:59Z|2024-01-01T01:00:00Z| **Aggregation details:** * Before: 60 individual CPU measurements over 1 hour * After: 1 aggregated measurement with averages and metadata * Metadata shows original record count and time range ### Example 2: Multi-field aggregation with different functions Apply different aggregation functions to different fields: ```bash # Create trigger with field-specific aggregations influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/downsampler/downsampler.py" \ --trigger-spec "every:10min" \ --trigger-arguments 'source_measurement=environment,target_measurement=environment_10min,interval=10min,window=30min,calculations=temperature:avg.humidity:avg.pressure:max' \ env_multi_agg # Write data with various sensor readings influxdb3 write \ --database sensors \ "environment,location=office temperature=22.5,humidity=45.2,pressure=1013.25" # Query aggregated data influxdb3 query \ --database sensors \ "SELECT * FROM environment_10min WHERE time >= now() - 1h" ``` ### Expected output |location|temperature|humidity|pressure|record\_count| time | |--------|-----------|--------|--------|-------------|--------------------| | office | 22.3 | 44.8 | 1015.1 | 10 |2024-01-01T00:10:00Z| ### Example 3: HTTP API downsampling with backfill Use HTTP API for on-demand downsampling with historical data: ```bash # Send HTTP request for backfill downsampling curl -X POST http://localhost:8181/api/v3/engine/downsample \ --header "Authorization: Bearer YOUR_TOKEN" \ --data '{ "source_measurement": "metrics", "target_measurement": "metrics_daily", "target_database": "analytics", "interval": "1d", "batch_size": "7d", "calculations": [["cpu_usage", "avg"], ["memory_usage", "max"], ["disk_usage", "avg"]], "backfill_start": "2024-01-01T00:00:00Z", "backfill_end": "2024-01-31T00:00:00Z", "max_retries": 3 }' ``` ## Code overview ### Files * `downsampler.py`: The main plugin code containing handlers for scheduled and HTTP-triggered downsampling * `downsampling_config_scheduler.toml`: Example TOML configuration file for scheduled triggers ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event (with nanosecond precision) * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error with unique task\_id for traceability ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled downsampling tasks. Queries historical data within the specified window and applies aggregation functions. Key operations: 1. Parses configuration from arguments or TOML file 2. Queries source measurement with optional tag filters 3. Applies time-based aggregation with specified functions 4. Writes downsampled data with metadata columns #### `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)` Handles HTTP-triggered on-demand downsampling. Processes batch downsampling with configurable time ranges for backfill scenarios. Key operations: 1. Parses JSON request body parameters 2. Processes data in configurable time batches 3. Applies aggregation functions to historical data 4. Returns processing statistics and results #### `build_downsample_query(fields_list, measurement, tags_list, interval, tag_values, start_time, end_time)` Builds the SQL query that applies time bucketing, tag filters, grouping, and aggregation functions to source data. Supported aggregation functions: * `avg`: Average value * `sum`: Sum of values * `min`: Minimum value * `max`: Maximum value * `median`: Median value * `count`: Count of values * `stddev`: Standard deviation * `first_value`: First value in time interval * `last_value`: Last value in time interval * `var`: Variance of values * `approx_median`: Approximate median (faster than exact median) ## Troubleshooting ### Common issues #### Issue: No data in target measurement **Solution**: Check that source measurement exists and contains data in the specified time window: ```bash influxdb3 query --database mydb "SELECT COUNT(*) FROM source_measurement WHERE time >= now() - 1h" ``` #### Issue: Aggregation function not working **Solution**: Verify field names and aggregation syntax. Use SHOW FIELD KEYS to check available fields: ```bash influxdb3 query --database mydb "SHOW FIELD KEYS FROM source_measurement" ``` #### Issue: Tag filters not applied **Solution**: Check tag value format. Use @ separator for multiple values: ```bash --trigger-arguments 'tag_values=host:server1@server2@server3' ``` #### Issue: HTTP endpoint not accessible **Solution**: Verify the trigger was created with correct request specification: ```bash influxdb3 list triggers --database mydb ``` ### Debugging tips 1. **Check execution logs** with task ID filtering: ``` influxdb3 query --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%task_id%' ORDER BY event_time DESC LIMIT 10" ``` 2. **Test with smaller time windows** for debugging: ``` --trigger-arguments 'window=5min,interval=1min' ``` 3. **Verify field types** before aggregation: ``` influxdb3 query --database mydb "SELECT * FROM source_measurement LIMIT 1" ``` ### Performance considerations #### Consolidate calculations in fewer triggers For best performance, define a single trigger per measurement that performs all necessary field calculations. Avoid creating multiple separate triggers that each handle only one field or calculation. Internal testing showed significant performance differences based on trigger design: * **Many triggers** (one calculation each): When 134 triggers were created, each handling a single calculation for a measurement, the cluster showed degraded performance with high CPU and memory usage. * **Consolidated triggers** (all calculations per measurement): When triggers were restructured so each one performed all necessary field calculations for a measurement, CPU usage dropped to approximately 4% and memory remained stable. #### Recommended Combine all field calculations for a measurement in one trigger: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/downsampler/downsampler.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'source_measurement=temperature,target_measurement=temperature_hourly,interval=1h,window=6h,calculations=temp:avg.temp:max.temp:min,specific_fields=temp' \ temperature_hourly_downsample ``` #### Not recommended Multiple triggers for the same measurement creates unnecessary overhead: ```bash # Avoid creating multiple triggers for calculations on the same measurement influxdb3 create trigger ... --trigger-arguments 'calculations=temp:avg' avg_trigger influxdb3 create trigger ... --trigger-arguments 'calculations=temp:max' max_trigger influxdb3 create trigger ... --trigger-arguments 'calculations=temp:min' min_trigger ``` #### Use specific\_fields to limit processing If your measurement contains fields that you don’t need to downsample, use the `specific_fields` parameter to specify only the relevant ones. Without this parameter, the downsampler processes all fields and applies the default aggregation (such as `avg`) to fields not listed in your calculations, which can lead to unnecessary processing and storage. ```bash # Only downsample the 'temp' field, ignore other fields in the measurement --trigger-arguments 'specific_fields=temp' # Downsample multiple specific fields --trigger-arguments 'specific_fields=temp.humidity.pressure' ``` #### Additional performance tips * **Batch processing**: Use appropriate `batch_size` for HTTP requests to balance memory usage and performance * **Retry logic**: Configure `max_retries` based on network reliability * **Metadata overhead**: Metadata columns add approximately 20% storage overhead but provide valuable debugging information * **Index optimization**: Tag filters are more efficient than field filters for large datasets ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Downsampler plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/downsampler) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[downsampling](/influxdb3/enterprise/tags/downsampling/)[aggregation](/influxdb3/enterprise/tags/aggregation/)[performance](/influxdb3/enterprise/tags/performance/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_measurement | string | required | Source measurement containing data to downsample | | target_measurement | string | required | Destination measurement for downsampled data | | window | string | required | Time window for each downsampling job. Format: (for example, “1h”, “1d”) | | interval | string | “10min” | Time interval for downsampling. Format: (for example, “10min”, “2h”, “1d”) | | calculations | string | “avg” | Aggregation functions. Single function or dot-separated field:aggregation pairs | | specific_fields | string | all fields | Dot-separated list of fields to downsample (for example, “co.temperature”) | | excluded_fields | string | none | Dot-separated list of fields and tags to exclude from downsampling results | | tag_values | string | none | Tag filters. Format: tag:value1@value2@value3 for multiple values | | offset | string | “0” | Time offset to apply to the window | | target_database | string | “default” | Database for storing downsampled data | | max_retries | integer | 5 | Maximum number of retries for write operations | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_measurement | string | required | Source measurement containing data to downsample | | target_measurement | string | required | Destination measurement for downsampled data | | interval | string | “10min” | Time interval for downsampling. Format: (for example, “10min”, “2h”, “1d”) | | batch_size | string | “30d” | Time interval for each HTTP backfill batch | | calculations | string/array | “avg” | Aggregation functions. Use "avg" for all fields, or an array of [field, aggregation] pairs | | specific_fields | array | all fields | List of fields to downsample | | excluded_fields | array | none | List of fields and tags to exclude from downsampling results | | tag_values | object | none | Tag filters as an object mapping tag names to lists of values | | target_database | string | “default” | Database for storing downsampled data | | max_retries | integer | 5 | Maximum number of retries for write operations | | backfill_start | string | oldest point | ISO 8601 datetime with timezone for the start of the HTTP backfill window | | backfill_end | string | current time | ISO 8601 datetime with timezone for the end of the HTTP backfill window | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | | host | usage_user | usage_system | usage_idle | record_count | time_from | time_to | time | | --- | --- | --- | --- | --- | --- | --- | --- | | host | usage_user | usage_system | usage_idle | record_count | time_from | time_to | time | | server1 | 44.8 | 11.9 | 43.3 | 60 | 2024-01-01T00:00:00Z | 2024-01-01T00:59:59Z | 2024-01-01T01:00:00Z | | location | temperature | humidity | pressure | record_count | time | | --- | --- | --- | --- | --- | --- | | location | temperature | humidity | pressure | record_count | time | | office | 22.3 | 44.8 | 1015.1 | 10 | 2024-01-01T00:10:00Z | --- # Earthquake sampler plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/earthquake-sampler/ The Earthquake Sampler Plugin ingests earthquake events from the USGS GeoJSON feeds on a schedule and writes normalized points for dashboards and alerting. It can also read from a custom JSON endpoint or from an existing InfluxDB table, and optionally writes directly into an existing canonical `quake` table using that table’s column names (`write_quake_schema=true`). Events are deduplicated with a per-event update-marker cache, so reruns only write new or updated earthquakes. * **Zero authentication**: USGS feeds require no API keys or signup * **Two source modes**: HTTP (USGS or custom JSON) and `influxdb_table` (transform rows from an existing table) * **Per-event deduplication**: update markers are cached per event id; updated events are re-written, unchanged events are skipped * **Canonical quake schema**: optional direct writes into an existing `quake` table ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. All parameters are optional; invalid values abort the run with an error log rather than silently falling back to defaults. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Data selection parameters | Parameter | Type | Default | Description | |------------------|-------|-------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `feed` |string | all\_hour | USGS GeoJSON feed key: `all`, `significant`, `4.5`, `2.5`, or `1.0` combined with `_hour`, `_day`, `_week`, or `_month` (for example `significant_day`) | | `source_type` |string | http | Data source type: `http` fetches JSON from `source_url` or `feed`; `influxdb_table` queries an existing table in the trigger database | | `source_url` |string | none | Custom source URL (`http` or `https` only). When provided, overrides `feed` and uses `source_format` parsing | | `source_format` |string |usgs\_geojson| Source parser for HTTP mode: `usgs_geojson` or `flat_json` (for records like `{id, latitude, longitude, mag, time, ...}`) | | `source_table` |string | quake | Source table name when `source_type=influxdb_table` | | `source_query` |string | none |Optional SQL override for `influxdb_table` mode; disables watermark paging and the source-table existence check. A query containing commas must come from a TOML file — see [TOML configuration](#toml-configuration)| |`lookback_minutes`|integer| 15 | Initial lookback window for `influxdb_table` mode. Later runs page forward from the cached fetch watermark while `skip_unchanged=true` | ### Optional parameters | Parameter | Type | Default | Description | |---------------------|-------|-------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------| | `measurement` |string | earthquakes | Destination measurement name | |`write_quake_schema` |boolean| false | Write events using the canonical `quake` table’s column names; no tags or normalized-only columns are written. Use with `measurement=quake` | | `min_magnitude` | float | none |Optional minimum magnitude. When omitted, nothing is filtered (USGS feeds include negative-magnitude microseisms and events without a magnitude)| | `max_events` |integer| 250 | Maximum events written per run, applied after filtering. Events deferred by the cap remain uncached and are written on a later run | |`use_event_timestamp`|boolean| true | Use the event time as the point timestamp; `false` uses the trigger execution time. Ignored (forced `true`) when `write_quake_schema=true` | | `skip_unchanged` |boolean| true | Skip events whose update marker is not newer than the last written copy of the same event. Events without an id are always written | | `user_agent` |string |InfluxDB3-Earthquake-Plugin/1.0| Custom User-Agent header for API requests | |`enable_full_logging`|boolean| false | When `true`, full exception messages are logged. When `false` (default), only exception types are logged | | `config_file_path` |string | none | Path to a TOML configuration file, relative to the plugin directory. See [TOML configuration](#toml-configuration) | ### TOML configuration Trigger arguments are a comma-separated `key=value` list, so a value that itself contains a comma — most notably `source_query` — cannot be passed inline. Put those parameters in a TOML file instead and point `config_file_path` at it. The path is resolved relative to the plugin directory (`PLUGIN_DIR`, `INFLUXDB3_PLUGIN_DIR`, or the processing-engine virtualenv). Values in the file override the inline trigger arguments. A file that is missing, malformed, or fails validation is reported in the logs and skipped, and the run continues with the inline arguments; a path that does not end in `.toml` is rejected the same way. `earthquake_sampler_config_scheduler.toml` ships alongside the plugin with every parameter documented and commented out. Copy it into your plugin directory and uncomment what you need: ```toml measurement = "earthquakes" feed = "all_day" min_magnitude = 2.5 max_events = 500 skip_unchanged = true # A query containing commas is only expressible here, not in --trigger-arguments. # Use a single-quoted TOML string when column names need double quotes. source_type = "influxdb_table" source_table = "quake" source_query = 'SELECT time, id, mag, depth, "magType", net, updated FROM quake ORDER BY time DESC LIMIT 500' ``` ```bash influxdb3 create trigger \ --database quakes \ --path "gh:influxdata/earthquake_sampler/earthquake_sampler.py" \ --trigger-spec "every:5m" \ --trigger-arguments "config_file_path=earthquake_sampler_config_scheduler.toml" \ earthquakes_from_toml ``` Types are native in TOML: `min_magnitude = 2.5` and `skip_unchanged = true` need no quoting, unlike the string values that inline arguments always deliver. ## Schema requirements `write_quake_schema=true` targets an existing `quake` table with the USGS CSV column layout (`depth`, `dmin`, `gap`, `id`, `latitude`, `longitude`, `mag`, `magType`, `net`, `nst`, `place`, `rms`, `status`, `time`, `type`, and the CSV-only columns below). * All numeric columns (including `nst` and `magNst`) are written as float64. This matches quake tables created by CSV import, where numeric columns containing blanks are inferred as doubles. If your existing table stores these columns as int64, the writes are rejected with a type conflict. * `depthError`, `horizontalError`, `magError`, `magNst`, `locationSource`, and `magSource` exist only in USGS CSV feeds, not GeoJSON, so they are written only when a `flat_json` or `influxdb_table` source supplies them. * Quake-schema rows carry no tags, so point identity rests entirely on the timestamp. USGS supplies millisecond-precision times; the plugin fills the unused sub-millisecond bits with a stable per-event offset so two earthquakes in the same millisecond do not overwrite each other. Millisecond-level time is unchanged. ## Software requirements * **InfluxDB 3 Enterprise**: 3.8.2 or later (the plugin writes with `write_sync`), with the Processing Engine enabled * **Python packages**: `influxdata-plugin-utils>=0.3.0` * **Network access**: Outbound HTTPS access to `earthquake.usgs.gov` (HTTP mode) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install the required Python package: ``` influxdb3 install package influxdata-plugin-utils ``` ## Trigger setup ### Scheduled trigger Create a trigger that ingests the USGS `all_hour` feed every two minutes: ```bash influxdb3 create trigger \ --database quakes \ --path "gh:influxdata/earthquake_sampler/earthquake_sampler.py" \ --trigger-spec "every:2m" \ earthquake_sampler_trigger # Enable the trigger influxdb3 enable trigger --database quakes earthquake_sampler_trigger ``` ## Example usage ### Example 1: Normalized earthquake ingestion Ingest the daily feed into the normalized `earthquakes` measurement: ```bash # Create the trigger influxdb3 create trigger \ --database quakes \ --path "gh:influxdata/earthquake_sampler/earthquake_sampler.py" \ --trigger-spec "every:5m" \ --trigger-arguments "feed=all_day" \ earthquakes_all_day # Enable the trigger influxdb3 enable trigger --database quakes earthquakes_all_day # Query events (after a few minutes) influxdb3 query \ --database quakes \ "SELECT event_id, magnitude, place, latitude, longitude, depth_km, time FROM earthquakes ORDER BY time DESC LIMIT 5" ``` ### Expected output |event\_id |magnitude| place |latitude|longitude|depth\_km| time | |----------|---------|----------------------------|--------|---------|---------|------------------------| |us7000abcd| 4.6 |100 km SSW of Sand Point, AK| 54.5 | \-161.2 | 32.4 |2026-08-21T17:58:12.421Z| |ak0261abcd| 1.4 | 12 km NNE of Palmer, AK | 61.7 | \-148.9 | 27.9 |2026-08-21T17:55:03.118Z| |nc75abcdef| 0.9 | 8 km WNW of Cobb, CA | 38.8 | \-122.8 | 1.6 |2026-08-21T17:51:47.902Z| ### Example 2: Write directly into an existing quake table Write USGS events into an existing `quake` table using its original column names (see [Schema requirements](#schema-requirements)): ```bash influxdb3 create trigger \ --database usgs \ --path "gh:influxdata/earthquake_sampler/earthquake_sampler.py" \ --trigger-spec "every:2m" \ --trigger-arguments "feed=all_hour,measurement=quake,write_quake_schema=true" \ usgs_to_quake ``` ### Example 3: Transform rows from an existing table Read rows from an existing `quake` table and write them as normalized events. The plugin pages forward from a cached fetch watermark, so each run picks up where the previous one stopped: ```bash influxdb3 create trigger \ --database usgs \ --path "gh:influxdata/earthquake_sampler/earthquake_sampler.py" \ --trigger-spec "every:1m" \ --trigger-arguments "source_type=influxdb_table,source_table=quake,measurement=earthquakes,lookback_minutes=60" \ quake_to_earthquakes ``` ### Example 4: Magnitude filtering Only ingest events at or above magnitude 2.5: ```bash influxdb3 create trigger \ --database quakes \ --path "gh:influxdata/earthquake_sampler/earthquake_sampler.py" \ --trigger-spec "every:5m" \ --trigger-arguments "feed=all_day,min_magnitude=2.5" \ earthquakes_m25 ``` ## Code overview ### Files * `earthquake_sampler.py`: The main plugin code containing the scheduled handler for earthquake ingestion * `requirements.txt`: Python dependencies (`influxdata-plugin-utils`, used for configuration loading, validation, and writes) * `earthquake_sampler_config_scheduler.toml`: Example TOML configuration with every parameter documented ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT event_time, log_level, log_text FROM system.processing_engine_logs WHERE trigger_name = 'earthquake_sampler_trigger' ORDER BY event_time DESC LIMIT 20" ``` Log lines are prefixed with a per-run task id. Logged source names strip URL credentials/query strings and truncate custom SQL. ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles a scheduled run end to end: validates configuration (aborting with an error on invalid values), fetches events from the configured source, normalizes and filters them, deduplicates against the per-event marker cache, and writes points with `write_sync` so write errors surface during the run. #### `_fetch_payload(url, user_agent)` / `_fetch_table_rows(...)` HTTP fetching (scheme-validated, `http`/`https` only) and table reading. Table reads page oldest-first from the cached fetch watermark (`WHERE time > watermark ORDER BY time ASC LIMIT max_events`), falling back to the `lookback_minutes` window on the first run. #### `_normalize_usgs_feature(feature)` / `_normalize_flat_event(item)` Normalize a USGS GeoJSON feature or a flat JSON/table record into the common internal event shape. Timestamps of unknown shape (ISO strings, epoch seconds/ms/us/ns) are coerced by magnitude. #### `_write_event(...)` / `_write_quake_event(...)` Build and write a point in the normalized schema (tags: `event_type`, `status`, `alert`, `net`, `mag_type`; `event_id` is a string field to avoid unbounded series cardinality) or in the canonical quake schema (no tags). Fields the source does not supply are left out of the point rather than written as empty values, and an event with no usable field at all is skipped and logged. Millisecond-aligned timestamps get a stable per-event sub-millisecond offset so same-millisecond events stay distinct. ## Troubleshooting ### Common issues #### Issue: No data appearing **Solution**: Check trigger status, review plugin logs, and verify network connectivity: ```bash # Check trigger status influxdb3 show summary --database quakes --token YOUR_TOKEN # Check plugin logs influxdb3 query --database quakes "SELECT event_time, log_level, log_text FROM system.processing_engine_logs WHERE log_text LIKE '%Earthquake%' ORDER BY event_time DESC LIMIT 10" # Verify the feed is reachable curl -H "User-Agent: test" https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_hour.geojson ``` #### Issue: `Invalid configuration` error in logs **Solution**: A trigger argument has an invalid value (unknown `feed` key, non-numeric `min_magnitude`, unrecognized boolean, `max_events` below 1, and so on). The message quotes the offending value; the run aborts rather than proceeding with a silent fallback. Booleans accept `true/false`, `yes/no`, `on/off`, and `1/0`. #### Issue: `Source table ... not found in the trigger database` **Solution**: `source_type=influxdb_table` checks that `source_table` exists before querying it. Verify the name with `influxdb3 query --database YOUR_DATABASE "SHOW TABLES"`, or supply an explicit `source_query`, which bypasses the check. #### Issue: Log summary shows `written=0` with a nonzero `skipped` count **Solution**: This is normal steady-state behavior: `skip_unchanged=true` skips events already written at their current update marker. New and updated events are still written. Set `skip_unchanged=false` to re-write everything the source returns. #### Issue: Field type conflict when writing with `write_quake_schema=true` **Solution**: The plugin writes all numeric quake columns as float64 (see [Schema requirements](#schema-requirements)). If your existing table stores `nst` or `magNst` as int64, rename the destination via `measurement` or recreate the table with float64 columns. #### Issue: Table mode does not re-read older rows **Solution**: With `skip_unchanged=true`, table reads page forward from the cached fetch watermark and do not revisit rows behind it. Set `skip_unchanged=false` to re-read the full `lookback_minutes` window, or provide an explicit `source_query`. ### Debugging tips 1. **Check trigger status**: ``` influxdb3 show summary --database quakes --token YOUR_TOKEN ``` 2. **Enable/Disable trigger**: ``` influxdb3 disable trigger earthquake_sampler_trigger --database quakes --token YOUR_TOKEN influxdb3 enable trigger earthquake_sampler_trigger --database quakes --token YOUR_TOKEN ``` 3. **Enable full exception logging temporarily**: add `enable_full_logging=true` to the trigger arguments. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Earthquake sampler plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/earthquake_sampler) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | feed | string | all_hour | USGS GeoJSON feed key: all , significant , 4.5 , 2.5 , or 1.0 combined with _hour , _day , _week , or _month (for example significant_day ) | | source_type | string | http | Data source type: http fetches JSON from source_url or feed ; influxdb_table queries an existing table in the trigger database | | source_url | string | none | Custom source URL ( http or https only). When provided, overrides feed and uses source_format parsing | | source_format | string | usgs_geojson | Source parser for HTTP mode: usgs_geojson or flat_json (for records like {id, latitude, longitude, mag, time, ...} ) | | source_table | string | quake | Source table name when source_type=influxdb_table | | source_query | string | none | Optional SQL override for influxdb_table mode; disables watermark paging and the source-table existence check. A query containing commas must come from a TOML file — see TOML configuration | | lookback_minutes | integer | 15 | Initial lookback window for influxdb_table mode. Later runs page forward from the cached fetch watermark while skip_unchanged=true | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | earthquakes | Destination measurement name | | write_quake_schema | boolean | false | Write events using the canonical quake table’s column names; no tags or normalized-only columns are written. Use with measurement=quake | | min_magnitude | float | none | Optional minimum magnitude. When omitted, nothing is filtered (USGS feeds include negative-magnitude microseisms and events without a magnitude) | | max_events | integer | 250 | Maximum events written per run, applied after filtering. Events deferred by the cap remain uncached and are written on a later run | | use_event_timestamp | boolean | true | Use the event time as the point timestamp; false uses the trigger execution time. Ignored (forced true ) when write_quake_schema=true | | skip_unchanged | boolean | true | Skip events whose update marker is not newer than the last written copy of the same event. Events without an id are always written | | user_agent | string | InfluxDB3-Earthquake-Plugin/1.0 | Custom User-Agent header for API requests | | enable_full_logging | boolean | false | When true , full exception messages are logged. When false (default), only exception types are logged | | config_file_path | string | none | Path to a TOML configuration file, relative to the plugin directory. See TOML configuration | | event_id | magnitude | place | latitude | longitude | depth_km | time | | --- | --- | --- | --- | --- | --- | --- | | event_id | magnitude | place | latitude | longitude | depth_km | time | | us7000abcd | 4.6 | 100 km SSW of Sand Point, AK | 54.5 | -161.2 | 32.4 | 2026-08-21T17:58:12.421Z | | ak0261abcd | 1.4 | 12 km NNE of Palmer, AK | 61.7 | -148.9 | 27.9 | 2026-08-21T17:55:03.118Z | | nc75abcdef | 0.9 | 8 km WNW of Cobb, CA | 38.8 | -122.8 | 1.6 | 2026-08-21T17:51:47.902Z | --- # Forecast error evaluator plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/forecast-error-evaluator/ The Forecast Error Evaluator Plugin validates forecast model accuracy for time series data in InfluxDB 3 Enterprise by comparing predicted values with actual observations. On every scheduled run the plugin matches the two measurements over a time window, computes an error metric (MSE, MAE, RMSE, MAPE, or SMAPE) for each matched timestamp, and notifies for the points that reach a configured threshold. It includes debounce logic to suppress transient anomalies and supports multi-channel notifications via the Notification Sender Plugin. The metric is computed per timestamp rather than aggregated over the window, which is what lets `min_condition_duration` measure how long an elevated error persists. As a consequence `rmse` yields the same value as `mae`: the root of a single squared difference is its absolute value. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |----------------------|------|--------|-----------------------------------------------------------------------------------------------------------------------------------| |`forecast_measurement`|string|required| Measurement containing forecasted values | | `actual_measurement` |string|required| Measurement containing actual (ground truth) values | | `forecast_field` |string|required| Field name for forecasted values | | `actual_field` |string|required| Field name for actual values | | `error_metric` |string|required| Error metric to compute: `mse`, `mae`, `rmse`, `mape`, or `smape` | | `error_thresholds` |string|required|Colon-separated `-` pairs, e.g. `INFO-"0.5":WARN-"0.9":ERROR-"1.2":CRITICAL-"1.5"`. See [Thresholds](#thresholds)| | `window` |string|required| Time window for data analysis. Must be a positive duration. Units: `us`, `ms`, `s`, `min`, `h`, `d`, `w` | | `senders` |string|required| Dot-separated list of notification channels (for example, “slack.discord”) | ### Thresholds Levels are `INFO`, `WARN`, `ERROR` and `CRITICAL`, and each threshold must be above `0`. Every supported metric is non-negative, so a threshold of `0` or below would flag every point of the window; such a level is skipped with a warning. Levels are evaluated independently: a point whose error reaches several thresholds produces one notification per level, each with its own debounce state. Configure only the levels you want to be paged about. Levels are processed from the highest threshold down, so `max_notifications_per_run` is spent on the most severe alerts first. Malformed segments, and a level given twice, are skipped with a warning and the remaining levels still apply. If no level survives parsing, the run logs an error and stops without notifying. ### Notification parameters | Parameter | Type | Default | Description | |---------------------------|-------|----------------|-------------------------------------------------------------------------------------------------------------------------------| | `notification_text` |string |default template|Template for notification message with variables `$measurement`, `$level`, `$field`, `$error`, `$metric`, `$tags`, `$timestamp`| | `notification_path` |string | “notify” | URL path for the notification sending plugin | | `port_override` |integer| 8181 | Port number where InfluxDB accepts requests | |`max_notifications_per_run`|integer| 20 | Maximum notifications sent by a single run. Alerts beyond the limit are counted in a warning and not resent later | ### Timing parameters | Parameter | Type | Default | Description | |------------------------|------|-----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`min_condition_duration`|string| `0s` |Time an error must stay above a threshold before alerting. Units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`. With the default the first point above the threshold alerts| | `rounding_freq` |string|no rounding| Fixed pandas frequency used to round timestamps before matching the two measurements, e.g. `1s`, `500ms`, `5min`, `1h` | ### Authentication parameters | Parameter | Type | Default | Description | |----------------------|------|------------|--------------------------------------------------------------------------| |`influxdb3_auth_token`|string|env variable|API token for InfluxDB 3 Enterprise. Can be set via `INFLUXDB3_AUTH_TOKEN`| ### Sender-specific parameters #### Slack notifications | Parameter | Type |Default | Description | |-------------------|------|--------|---------------------------| |`slack_webhook_url`|string|required| Webhook URL from Slack | | `slack_headers` |string| none |Base64-encoded HTTP headers| #### Discord notifications | Parameter | Type |Default | Description | |---------------------|------|--------|---------------------------| |`discord_webhook_url`|string|required| Webhook URL from Discord | | `discord_headers` |string| none |Base64-encoded HTTP headers| #### HTTP notifications | Parameter | Type |Default | Description | |------------------|------|--------|------------------------------------| |`http_webhook_url`|string|required|Custom webhook URL for POST requests| | `http_headers` |string| none | Base64-encoded HTTP headers | #### SMS notifications (via Twilio) | Parameter | Type | Default | Description | |--------------------|------|------------|-------------------------------------------------| | `twilio_sid` |string|env variable| Twilio Account SID (or `TWILIO_SID` env var) | | `twilio_token` |string|env variable| Twilio Auth Token (or `TWILIO_TOKEN` env var) | |`twilio_from_number`|string| required |Twilio sender number (for example, “+1234567890”)| | `twilio_to_number` |string| required | Recipient number (for example, “+0987654321”) | ### Matching forecast to actual values Forecast and actual rows are matched with an inner join on the timestamp plus the tags that both measurements share. Tags present in only one of them are ignored with a warning, so a forecast table without the tag columns of the actual table still works. Set `rounding_freq` when the two series are written with slightly different timestamps. Rounding coarser than the sampling interval puts several points into the same slot; only the earliest point of each slot is kept, and the number of collapsed rows is logged. Without that, matching would join every forecast of the slot against every actual value in it. Rows where either value is missing are dropped before the metric is computed. For `mape`, rows with `actual = 0` are skipped; for `smape`, rows where both values are `0` are skipped. ### Alert state Debounce and alert state live in the trigger-local cache, keyed by measurement, field, level and tag values: * While an error stays above a threshold for less than `min_condition_duration`, the plugin logs the pending state and waits. The duration is measured in data time, and a pending start that has scrolled out of the window is discarded, so a gap in the data cannot stand in for a persistent error. * After an alert is delivered, its timestamp is recorded and earlier or equal timestamps are never alerted again. Overlapping windows on successive runs therefore do not resend the same point. * If delivery fails after all retries, the state is left untouched so the next run alerts on that point again. * The cache is in-memory and trigger-local: restarting the server clears the debounce and last-alert state. ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Relative paths are resolved against the first directory that is set: `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. Only that directory is used — the file is not looked up in the remaining ones. When `config_file_path` is set, the TOML file provides the whole configuration and inline trigger arguments are ignored. `INFLUXDB3_AUTH_TOKEN` from the environment still applies when `influxdb3_auth_token` is not set in the file. In TOML, `senders` and `error_thresholds` can use native structures (a list and a table) instead of the inline string formats, though the inline strings are also accepted. #### Example TOML configuration [forecast\_error\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/forecast_error_evaluator/forecast_error_config_scheduler.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Notification Sender Plugin for InfluxDB 3 Enterprise**: Required for sending notifications. See the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/). * **Python packages**: * `influxdata-plugin-utils>=0.3.0` (configuration loading, parsing, and schema introspection) * `pandas` (for data processing) * `requests` (for HTTP notifications) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package influxdata-plugin-utils influxdb3 install package pandas influxdb3 install package requests ``` 3. Install the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) (required) ## Trigger setup ### Scheduled forecast validation Run forecast error evaluation periodically: ```bash influxdb3 create trigger \ --database weather_forecasts \ --path "gh:influxdata/forecast_error_evaluator/forecast_error_evaluator.py" \ --trigger-spec "every:30m" \ --trigger-arguments 'forecast_measurement=temperature_forecast,actual_measurement=temperature_actual,forecast_field=predicted_temp,actual_field=temp,error_metric=rmse,error_thresholds=INFO-"0.5":WARN-"1.0":ERROR-"2.0",window=1h,senders=slack,slack_webhook_url="$SLACK_WEBHOOK_URL"' \ forecast_validation ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ## Example usage ### Example 1: Temperature forecast validation with Slack alerts Validate temperature forecast accuracy and send Slack notifications: ```bash # Create the trigger influxdb3 create trigger \ --database weather_db \ --path "gh:influxdata/forecast_error_evaluator/forecast_error_evaluator.py" \ --trigger-spec "every:15m" \ --trigger-arguments 'forecast_measurement=temp_forecast,actual_measurement=temp_actual,forecast_field=predicted,actual_field=temperature,error_metric=rmse,error_thresholds=INFO-"0.5":WARN-"1.0":ERROR-"2.0":CRITICAL-"3.0",window=30min,senders=slack,slack_webhook_url="$SLACK_WEBHOOK_URL",min_condition_duration=10min' \ temp_forecast_check # Write forecast data influxdb3 write \ --database weather_db \ "temp_forecast,location=station1 predicted=22.5" # Write actual data influxdb3 write \ --database weather_db \ "temp_actual,location=station1 temperature=21.8" # Check logs after trigger runs influxdb3 query \ --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'temp_forecast_check'" ``` **Expected output** * Plugin computes the error between forecast and actual values for every matched timestamp * Points with an error of 0.5 or more send an INFO notification, 1.0 or more a WARN notification, and so on for each configured level * A point is only alerted after its error has stayed above the level for 10 minutes (debounce) Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. **Notification example:** [WARN] Forecast error alert in temp\_actual.temperature: rmse=1.2. Tags: location=station1 The message names the actual measurement and field, because that is where the observed value comes from. ### Example 2: Multi-metric validation with multiple channels Monitor multiple forecast metrics with different notification channels: ```bash # Create trigger with Discord and HTTP notifications influxdb3 create trigger \ --database analytics \ --path "gh:influxdata/forecast_error_evaluator/forecast_error_evaluator.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'forecast_measurement=sales_forecast,actual_measurement=sales_actual,forecast_field=predicted_sales,actual_field=sales_amount,error_metric=mae,error_thresholds=WARN-"1000":ERROR-"5000":CRITICAL-"10000",window=6h,senders=discord.http,discord_webhook_url="$DISCORD_WEBHOOK_URL",http_webhook_url="$HTTP_WEBHOOK_URL",notification_text="[$$level] Sales forecast error: $$metric=$$error (threshold exceeded)",rounding_freq=5min' \ sales_forecast_monitor ``` Set `DISCORD_WEBHOOK_URL` and `HTTP_WEBHOOK_URL` to your webhook URLs. ### Example 3: SMS alerts for critical forecast failures Set up SMS notifications for critical forecast accuracy issues: ```bash # Set environment variables (recommended for sensitive data) export TWILIO_SID="your_twilio_sid" export TWILIO_TOKEN="your_twilio_token" # Create trigger with SMS notifications influxdb3 create trigger \ --database production_forecasts \ --path "gh:influxdata/forecast_error_evaluator/forecast_error_evaluator.py" \ --trigger-spec "every:5m" \ --trigger-arguments 'forecast_measurement=demand_forecast,actual_measurement=demand_actual,forecast_field=predicted_demand,actual_field=actual_demand,error_metric=mse,error_thresholds=CRITICAL-"100000",window=15min,senders=sms,twilio_from_number="+1234567890",twilio_to_number="+0987654321",notification_text="CRITICAL: Production demand forecast error exceeded threshold. MSE: $$error",min_condition_duration=2min' \ critical_forecast_alert ``` ## Using TOML Configuration Files This plugin supports using TOML configuration files for complex configurations. ### Important Requirements **To use TOML configuration files, you must set the `PLUGIN_DIR` environment variable in the InfluxDB 3 Enterprise host environment:** ```bash PLUGIN_DIR=~/.plugins influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` ### Example TOML Configuration ```toml # forecast_error_config_scheduler.toml forecast_measurement = "temperature_forecast" actual_measurement = "temperature_actual" forecast_field = "predicted_temp" actual_field = "temperature" error_metric = "rmse" error_thresholds = 'INFO-"0.5":WARN-"1.0":ERROR-"2.0":CRITICAL-"3.0"' window = "1h" senders = "slack" slack_webhook_url = "$SLACK_WEBHOOK_URL" min_condition_duration = "10min" rounding_freq = "1min" notification_text = "[$$level] Forecast validation alert: $$metric=$$error in $$measurement.$$field" # Authentication (use environment variables instead when possible) influxdb3_auth_token = "your_token_here" ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Create trigger using TOML config ```bash influxdb3 create trigger \ --database weather_db \ --path "gh:influxdata/forecast_error_evaluator/forecast_error_evaluator.py" \ --trigger-spec "every:30m" \ --trigger-arguments config_file_path=forecast_error_config_scheduler.toml \ forecast_validation_trigger ``` ## Code overview ### Files * `forecast_error_evaluator.py`: The main plugin code containing scheduler handler for forecast validation * `forecast_error_config_scheduler.toml`: Example TOML configuration file * `test_forecast_error_evaluator.py`: Pytest suite, runs without a live InfluxDB 3 Enterprise server * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`, `pandas`, `requests`) * `requirements-dev.txt`: Development dependencies (`pytest`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing validation results or errors ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled forecast validation tasks. Queries forecast and actual measurements, computes error metrics, and triggers notifications. Key operations: 1. Parses configuration from arguments or TOML file 2. Verifies that both measurements exist and resolves the tags they share 3. Queries forecast and actual measurements within the time window 4. Rounds timestamps, collapses duplicate keys and matches the two series 5. Computes the error metric for every matched timestamp 6. Evaluates each threshold level, applies debounce logic and skips already-alerted points 7. Sends notifications via configured channels, up to `max_notifications_per_run` #### `compute_error(influxdb3_local, merged, error_metric, task_id)` Adds a per-timestamp `error` column to the matched frame. |Metric | Formula per timestamp | Notes | |-------|-----------------------------------------------------|----------------------------------------------------------| | `mse` | `(forecast - actual)²` | Thresholds are in squared units | | `mae` | `|forecast - actual|` | | |`rmse` | `((forecast - actual)²)^0.5` | Equals `mae` for a single point | |`mape` | `|forecast - actual| / |actual| * 100` | Rows with `actual = 0` are skipped | |`smape`|`200 * |forecast - actual| / (|forecast| + |actual|)`|Bounded 0-200%; rows where both values are `0` are skipped| #### `align_frames(influxdb3_local, df_forecast, df_actual, tags, rounding_freq, task_id)` Rounds timestamps, keeps the earliest row per key and inner-joins the two frames on the timestamp and the shared tags. #### `parse_error_thresholds(influxdb3_local, config, task_id)` Parses the inline `-` string or the TOML table into a `{level: threshold}` mapping, skipping unknown levels, non-numeric values and thresholds at or below zero. ## Troubleshooting ### Common issues #### Issue: No overlapping timestamps between forecast and actual data **Solution**: Check that both measurements have data in the specified time window and use `rounding_freq` for alignment: ```bash influxdb3 query --database mydb "SELECT time, field_value FROM forecast_measurement WHERE time >= now() - 1h" influxdb3 query --database mydb "SELECT time, field_value FROM actual_measurement WHERE time >= now() - 1h" ``` #### Issue: Notifications not being sent **Solution**: Verify the Notification Sender Plugin is installed and webhook URLs are correct: ```bash # Check if notifier plugin exists ls ~/.plugins/notifier_plugin.py # Test webhook URL manually curl -X POST "your_webhook_url" -d '{"text": "test message"}' ``` #### Issue: Error threshold format not recognized **Solution**: Use proper threshold format with level prefixes. Note that MAPE and SMAPE thresholds are in percentages: ```bash # For absolute metrics (MSE, MAE, RMSE) --trigger-arguments 'error_thresholds=INFO-"0.5":WARN-"1.0":ERROR-"2.0":CRITICAL-"3.0"' # For percentage metrics (MAPE, SMAPE) --trigger-arguments 'error_thresholds=INFO-"5.0":WARN-"10.0":ERROR-"20.0":CRITICAL-"30.0"' ``` A `Skipping threshold` warning names the level that was dropped and why. `No valid error thresholds configured` means every level was rejected, so nothing was evaluated. #### Issue: Trigger fails with “Failed to load configuration” **Solution**: The message names the offending parameter. Common causes are a duration without a supported unit (use `min`, not `m`), a `window` of `0s`, a `port_override` outside 1-65535 and an `error_metric` outside `mse`, `mae`, `rmse`, `mape`, `smape`. #### Issue: Many rows collapse into one timestamp **Solution**: A `Collapsed N forecast and M actual rows sharing a rounded timestamp` line means `rounding_freq` is coarser than the sampling interval, so only the earliest point of each slot is compared. Lower `rounding_freq` to match how far apart the two series are actually written. #### Issue: Notifications stop mid-run **Solution**: `Suppressed N notifications after reaching max_notifications_per_run` means the per-run cap was hit. Raise `max_notifications_per_run`, raise the thresholds, or set `min_condition_duration` so short spikes are not alerted. #### Issue: MAPE/SMAPE calculation errors with zero values **Solution**: MAPE cannot be calculated when actual values are zero, and SMAPE cannot be calculated when both forecast and actual are zero. The plugin automatically skips such rows and logs warnings. For datasets with frequent zero values, consider using MAE or RMSE instead. #### Issue: Environment variables not loaded **Solution**: Set environment variables before starting InfluxDB: ```bash export INFLUXDB3_AUTH_TOKEN="your_token" export TWILIO_SID="your_sid" influxdb3 serve --plugin-dir ~/.plugins ``` ### Debugging tips 1. **Check data availability** in both measurements: ```bash influxdb3 query --database mydb \ "SELECT COUNT(*) FROM forecast_measurement WHERE time >= now() - window" ``` 1. **Verify timestamp alignment** with rounding frequency: ```bash --trigger-arguments 'rounding_freq=5min' ``` 1. **Test with shorter windows** for faster debugging: ```bash --trigger-arguments 'window=10min,min_condition_duration=1min' ``` 1. **Monitor notification delivery** in logs: ```bash influxdb3 query --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%notification%'" ``` ### Performance considerations * **Data alignment**: Use appropriate `rounding_freq` to balance accuracy and performance * **Window size**: Larger windows evaluate more points per run, and every point is checked against every configured level * **Debounce duration**: Balance between noise suppression and alert responsiveness * **Notification throttling**: Deliveries are sequential with up to three retries each, so `max_notifications_per_run` bounds how long a run can take * **Trigger interval**: An interval shorter than `window` re-reads the overlap on every run; the last-alert state keeps it from resending, but the data is queried again * **Memory usage**: Plugin processes data in pandas DataFrames - consider memory for large datasets ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Forecast error evaluator plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/forecast_error_evaluator) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[forecasting](/influxdb3/enterprise/tags/forecasting/)[evaluation](/influxdb3/enterprise/tags/evaluation/)[analytics](/influxdb3/enterprise/tags/analytics/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | forecast_measurement | string | required | Measurement containing forecasted values | | actual_measurement | string | required | Measurement containing actual (ground truth) values | | forecast_field | string | required | Field name for forecasted values | | actual_field | string | required | Field name for actual values | | error_metric | string | required | Error metric to compute: mse , mae , rmse , mape , or smape | | error_thresholds | string | required | Colon-separated - pairs, e.g. INFO-"0.5":WARN-"0.9":ERROR-"1.2":CRITICAL-"1.5" . See Thresholds | | window | string | required | Time window for data analysis. Must be a positive duration. Units: us , ms , s , min , h , d , w | | senders | string | required | Dot-separated list of notification channels (for example, “slack.discord”) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | notification_text | string | default template | Template for notification message with variables $measurement , $level , $field , $error , $metric , $tags , $timestamp | | notification_path | string | “notify” | URL path for the notification sending plugin | | port_override | integer | 8181 | Port number where InfluxDB accepts requests | | max_notifications_per_run | integer | 20 | Maximum notifications sent by a single run. Alerts beyond the limit are counted in a warning and not resent later | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | min_condition_duration | string | 0s | Time an error must stay above a threshold before alerting. Units: us , ms , s , min , h , d , w . With the default the first point above the threshold alerts | | rounding_freq | string | no rounding | Fixed pandas frequency used to round timestamps before matching the two measurements, e.g. 1s , 500ms , 5min , 1h | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | influxdb3_auth_token | string | env variable | API token for InfluxDB 3 Enterprise. Can be set via INFLUXDB3_AUTH_TOKEN | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | slack_webhook_url | string | required | Webhook URL from Slack | | slack_headers | string | none | Base64-encoded HTTP headers | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | discord_webhook_url | string | required | Webhook URL from Discord | | discord_headers | string | none | Base64-encoded HTTP headers | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | http_webhook_url | string | required | Custom webhook URL for POST requests | | http_headers | string | none | Base64-encoded HTTP headers | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | twilio_sid | string | env variable | Twilio Account SID (or TWILIO_SID env var) | | twilio_token | string | env variable | Twilio Auth Token (or TWILIO_TOKEN env var) | | twilio_from_number | string | required | Twilio sender number (for example, “+1234567890”) | | twilio_to_number | string | required | Recipient number (for example, “+0987654321”) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | | Metric | Formula per timestamp | Notes | | --- | --- | --- | | Metric | Formula per timestamp | Notes | | mse | (forecast - actual)² | Thresholds are in squared units | | mae | |forecast - actual| | | | rmse | ((forecast - actual)²)^0.5 | Equals mae for a single point | | mape | |forecast - actual| / |actual| * 100 | Rows with actual = 0 are skipped | | smape | 200 * |forecast - actual| / (|forecast| + |actual|) | Bounded 0-200%; rows where both values are 0 are skipped | --- # Gapfill plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/gapfill/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later (uses the synchronous write API). > > The Gapfill Plugin detects gaps in time series (spacing between consecutive points wider than a threshold) and fills them with imputed values using standard missing-data imputation methods. Fill points land on an epoch-anchored time grid strictly inside each gap, so overlapping runs are idempotent. It works both on uniform-grid data (for example, the output of the[`resampler`](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/resampler)plugin) and directly on irregular raw series, and supports two modes of operation: continuous scheduled filling and one-off historical repair over HTTP. * **Detects and fills in one pass**: a gap is any inter-point spacing greater than `gap_threshold`; detection runs per numeric field, so a field that goes missing while others keep reporting is still filled — non-numeric fields are carried onto fill points by last known value * **Standard imputation methods**: linear, previous (LOCF), next, nearest, cubic spline, PCHIP (monotone cubic, no overshoot at gap edges), constant * **Two output modes**: materialize the full series into a target measurement (pipeline mode), or append only fill points into the source (in-place repair) * **Bounded imputation**: gaps longer than `max_fill_gap` are never invented, only reported Unlike SQL[`date_bin_gapfill`](https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill), which fills gaps at query time, this plugin **materializes** a continuous series that downstream consumers — dashboards, other plugins such as[`signal_filter`](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/signal_filter)— read as physical points, with more fill methods, optional marking of imputed points, and a per-gap quality report. ## Configuration Plugin parameters may be specified as key-value pairs in the`--trigger-arguments` flag (`influxdb3 create trigger`) or in the`trigger_arguments` field of the API. Values are strings; the plugin coerces them. Alternatively, supply every parameter from a TOML file via`config_file_path` — see [TOML configuration](#toml-configuration). For the HTTP trigger, parameters are passed as a JSON request body. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters, enabling the[InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Shared parameters (scheduled args and HTTP body) | Parameter | Type | Default | Description | |--------------------|-------|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurement`|string | required | Source measurement to scan for gaps. | |`target_measurement`|string | *(in-place)* | Target measurement: the full series (existing points plus fills) is written there. Omit to append only the fill points into the source measurement. Must differ from the source. | | `interval` |string | `1s` | Expected series cadence and fill grid step, at least 1ms. Units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`. | | `method` |string | `linear` | Fill method: `linear`, `previous` (LOCF), `next`, `nearest`, `cubic`, `pchip`, or `constant`. | | `fill_value` | float | — | Constant used when `method=constant`; required for it, invalid otherwise. | | `gap_threshold` |string |1.5× interval | Spacing between consecutive points above which a gap is detected, at least one interval. | | `max_fill_gap` |string |*(unlimited)* | Gaps longer than this are not filled, only reported. At least one `gap_threshold`. | | `fields` |string | all numeric |Space-separated numeric fields to fill with the configured method; numeric fields not listed are dropped. Non-numeric fields are carried into fill points by last known value unless excluded.| | `excluded_fields` |string | none | Space-separated fields of any type to exclude from the output. | | `mark_filled` |boolean| `false` | When `true`, points that received filled values carry a boolean marker field (`filled=true` by default); such a point may also hold real values of other fields. | |`filled_field_name` |string | `filled` | Name of the marker field written when `mark_filled` is true; must not collide with an existing source column. | |`report_measurement`|string | *(off)* | Optional measurement receiving one row per detected gap; written to `target_database` when set. | | `target_database` |string |*(trigger db)*| Database for writing output. Requires `target_measurement` (chain mode). | | `max_retries` |integer| `5` | Maximum number of write attempts. | | `config_file_path` |string | — | Path to a TOML file supplying all parameters — replaces the trigger arguments or HTTP body entirely. Relative paths resolve against `PLUGIN_DIR`. | ### Scheduled-only parameters |Parameter| Type |Default| Description | |---------|------|-------|-----------------------------------------------------------| |`window` |string|`10min`|How much history each run processes, at least one interval.| |`offset` |string| `0s` | Processing delay for late-arriving data. | ### HTTP-only parameters | Parameter | Type | Default | Description | |----------------|------|---------------|----------------------------------------------------------------------------------------------------------| |`backfill_start`|string|*(oldest data)*|ISO 8601 start of the repair range (for example, `2026-07-18T14:00:00Z`). Naive values are treated as UTC.| | `backfill_end` |string| *(now)* | ISO 8601 end of the repair range. | | `batch_size` |string| `30d` | Time span processed per batch, at least one `interval`. | ### TOML configuration Set the `PLUGIN_DIR` environment variable and reference the file with the`config_file_path` trigger argument (relative paths resolve against`PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`). The TOML file then supplies **all** parameters — it is mutually exclusive with inline trigger arguments. See[`gapfill_config_scheduler.toml`](gapfill_config_scheduler.toml) for an annotated template. ## Output modes * **Chain mode** (`target_measurement` set): the full series — existing points plus fills — is written to the target measurement each run, so downstream consumers read one continuous measurement. Raw data is never modified. * **In-place repair** (`target_measurement` omitted): fill values are appended into the source measurement. Filling is per field, so a fill may land on a timestamp where other fields already hold real values — the missing field’s value (and the marker, when enabled) is added to that point; existing values are never modified. Every value is written with the source column’s stored type: integer fills are rounded and UInt64 stays `uint`, so output never conflicts with the schema. Blending methods (`linear`, `cubic`, `pchip`, `constant`) produce float64, so in chain mode those columns — fills and copied values alike — become float in the target; in in-place mode such fills are cast back to the column’s type. ## Gap report With `report_measurement` set, each detected gap produces one row: * **Tags**: the series tags plus `gap_field` (the field the gap was detected in). * **Fields**: `gap_start_ns`, `gap_end_ns` (boundary timestamps), `duration_s`(float seconds), `points` (fill points actually written for the gap; `0`when skipped), `status` (`filled`, or `skipped` when the gap is longer than`max_fill_gap` or no point could be written). * **Timestamp**: the gap end, so re-reporting the same gap overwrites in place. Report rows are written next to the output: with `target_database` set they land in that database, not in the trigger’s. Alerting on this measurement (for example, with the[`notifier`](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/notifier)plugin) turns gapfill into a sensor-health monitor. ## Data requirements * Existing timestamps are never changed: fills are added on the epoch-anchored grid, existing points keep their original time. The output is continuous but not uniform — run[`resampler`](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/resampler)first when downstream consumers need an exact grid. * On jittery raw data, raise `gap_threshold` above the expected jitter (the default is 1.5× `interval`), otherwise normal spacing is detected as gaps and imputed points appear between healthy readings. * Detecting a gap requires both its boundary points inside one run’s query window (quantified below), so a longer outage is neither filled nor reported. Repair those with the HTTP trigger over an explicit range, and alert on them with a deadman check (e.g.[`threshold_deadman_checks`](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/threshold_deadman_checks)). * A gap is filled only after it closes (a point arrives on its right side); an ongoing outage is not extrapolated. * A series/field needs at least 2 points to be processed; `cubic` needs at least 4 (falls back to `linear` with a warning). * Visibility costs reads: each run queries three times the range it processes (the range plus lookback padding on both sides). A scheduled run processes`window + max(window, max_fill_gap)`, so `window=10min` without`max_fill_gap` reads about 60 minutes of history per run. HTTP backfill batches are padded the same way. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled (`--plugin-dir` configured). * **Python packages**: `scipy` and `influxdata-plugin-utils` (numpy is installed with scipy). ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install the Python dependencies into the plugin environment: ``` influxdb3 install package "influxdata-plugin-utils>=0.3.0" influxdb3 install package scipy ``` ## Trigger setup ### Scheduled trigger (continuous pipeline) ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/gapfill/gapfill.py \ --trigger-spec "every:1m" \ --trigger-arguments 'source_measurement=signal_resampled,target_measurement=signal_filled,interval=1s,method=linear,max_fill_gap=1min' \ gapfill_signal influxdb3 enable trigger --database mydb gapfill_signal ``` ### HTTP trigger (one-off historical repair) ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/gapfill/gapfill.py \ --trigger-spec "request:gapfill" \ gapfill_backfill influxdb3 enable trigger --database mydb gapfill_backfill curl -X POST "http://localhost:8181/api/v3/engine/gapfill" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "source_measurement": "sensors", "interval": "15s", "method": "pchip", "backfill_start": "2026-07-18T00:00:00Z", "backfill_end": "2026-07-19T00:00:00Z" }' ``` **Expected response** (fills appended in-place, since `target_measurement` is omitted): ```json { "status": "ok", "task_id": "…", "batches": 1, "rows_scanned": 5730, "gaps_filled": 12, "gaps_skipped": 0, "fills_written": 96, "rows_copied": 0 } ``` ## Example usage ### Example 1: Fill outage gaps after resampling (pipeline mode) With `signal_resampled` on a 1s grid containing 10–30s outage holes: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/gapfill/gapfill.py \ --trigger-spec "every:1m" \ --trigger-arguments 'source_measurement=signal_resampled,target_measurement=signal_filled,interval=1s,max_fill_gap=1min,mark_filled=true' \ gapfill_signal influxdb3 enable trigger --database mydb gapfill_signal influxdb3 query --database mydb \ "SELECT time, value, filled FROM signal_filled ORDER BY time DESC LIMIT 10" ``` **Expected output**: a continuous 1s series in `signal_filled`; timestamps that were missing in `signal_resampled` carry interpolated `value`s and`filled=true`, copied real points have `filled` null. ### Example 2: Constant fill for an event counter A gap in an event-rate series means “no events”, not “interpolate”: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/gapfill/gapfill.py \ --trigger-spec "every:5m" \ --trigger-arguments 'source_measurement=events_per_min,interval=1min,method=constant,fill_value=0' \ gapfill_events influxdb3 enable trigger --database mydb gapfill_events ``` **Expected output**: zero-valued points appended in-place at missing minutes. ### Example 3: Gap report for sensor-health monitoring ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/gapfill/gapfill.py \ --trigger-spec "every:1m" \ --trigger-arguments 'source_measurement=sensors,interval=15s,max_fill_gap=2min,report_measurement=gapfill_report' \ gapfill_sensors influxdb3 query --database mydb \ "SELECT * FROM gapfill_report WHERE status = 'skipped' ORDER BY time DESC" ``` **Expected output**: one row per gap with boundaries, duration, and status;`skipped` rows mark outages too long to impute — alert on those. ## Code overview ### Files * `gapfill.py`: The main plugin code containing the handlers for scheduled filling and HTTP backfill * `gapfill_config_scheduler.toml`: Example TOML configuration file ### Logging Logs are stored in the `_internal` database in the`system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'gapfill_signal' ORDER BY event_time DESC LIMIT 20" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Entry point for the scheduled trigger. Loads and validates the configuration, then fills a sliding window of history ending at `call_time - offset`. #### `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)` Entry point for the HTTP trigger. Repairs `[backfill_start, backfill_end)` in`batch_size` chunks, resolving the schema once, and returns the run totals. #### `run_gapfill(...)` One pass over a time range: resolves the schema, queries the padded window, groups rows by tag set, fills each series (and copies existing points in chain mode), and writes the result with retries. #### `fill_series(...)` Detects gaps of one series per numeric field, computes fill values on the epoch grid, attaches carried fields by last known value, and builds the gap report rows. ## Troubleshooting ### Common issues #### Issue: “required packages are not installed” **Cause**: The plugin environment lacks `scipy` or `influxdata-plugin-utils`. **Solution**: Run `influxdb3 install package "influxdata-plugin-utils>=0.3.0"`and `influxdb3 install package scipy`, then re-enable the trigger. #### Issue: Field type conflict on write **Cause**: The chain-mode target measurement already contains a field whose stored type conflicts with the plugin’s output (blending methods write float64; another producer may have created the column as integer). **Solution**: Use a fresh target measurement, or a selection method (`previous`, `next`, `nearest`), which preserves source types. In-place mode preserves integer column types by rounding fills. #### Issue: No fills appear **Cause**: The gap has not closed yet (no point on its right side), the gap is longer than `max_fill_gap` or the lookback, or the series/field has fewer than 2 points. **Solution**: Check the info logs for skip counts; increase `max_fill_gap` or repair the range explicitly via the HTTP trigger. #### Issue: “source\_measurement already has a column named ‘filled’” **Cause**: `mark_filled` is true and the marker name (`filled_field_name`, default `filled`) is taken by a source tag or by a non-boolean field; the plugin refuses to overwrite it. In in-place mode, a boolean column of that name is treated as the plugin’s own marker from earlier runs and reused. **Solution**: Set `filled_field_name` to a name not used by the source, or disable `mark_filled`. With `mark_filled` off, a source field named `filled`is processed normally. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Gapfill plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/gapfill) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_measurement | string | required | Source measurement to scan for gaps. | | target_measurement | string | (in-place) | Target measurement: the full series (existing points plus fills) is written there. Omit to append only the fill points into the source measurement. Must differ from the source. | | interval | string | 1s | Expected series cadence and fill grid step, at least 1ms. Units: us , ms , s , min , h , d , w . | | method | string | linear | Fill method: linear , previous (LOCF), next , nearest , cubic , pchip , or constant . | | fill_value | float | — | Constant used when method=constant ; required for it, invalid otherwise. | | gap_threshold | string | 1.5× interval | Spacing between consecutive points above which a gap is detected, at least one interval. | | max_fill_gap | string | (unlimited) | Gaps longer than this are not filled, only reported. At least one gap_threshold . | | fields | string | all numeric | Space-separated numeric fields to fill with the configured method; numeric fields not listed are dropped. Non-numeric fields are carried into fill points by last known value unless excluded. | | excluded_fields | string | none | Space-separated fields of any type to exclude from the output. | | mark_filled | boolean | false | When true , points that received filled values carry a boolean marker field ( filled=true by default); such a point may also hold real values of other fields. | | filled_field_name | string | filled | Name of the marker field written when mark_filled is true; must not collide with an existing source column. | | report_measurement | string | (off) | Optional measurement receiving one row per detected gap; written to target_database when set. | | target_database | string | (trigger db) | Database for writing output. Requires target_measurement (chain mode). | | max_retries | integer | 5 | Maximum number of write attempts. | | config_file_path | string | — | Path to a TOML file supplying all parameters — replaces the trigger arguments or HTTP body entirely. Relative paths resolve against PLUGIN_DIR . | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | window | string | 10min | How much history each run processes, at least one interval. | | offset | string | 0s | Processing delay for late-arriving data. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | backfill_start | string | (oldest data) | ISO 8601 start of the repair range (for example, 2026-07-18T14:00:00Z ). Naive values are treated as UTC. | | backfill_end | string | (now) | ISO 8601 end of the repair range. | | batch_size | string | 30d | Time span processed per batch, at least one interval . | --- # Geo enrichment plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/geo-enrichment/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later (uses the synchronous write API). > > The Geo Enrichment Plugin turns coordinates carried by incoming points into location attributes you can group by — a country and city, a zone or site you defined yourself, or a cell of a global grid. By default they merge back into the **same rows the coordinates came from**, so no join is needed to use them; naming a target table or database collects them separately instead. * **Four resolution strategies**: offline place lookup, point-in-polygon against your own zones, nearest site from your own list, and grid cells (H3, geohash or S2). Zones and sites come from GeoJSON or CSV, whichever you have * **Reads coordinates in many shapes**: separate `lat`/`lon` columns as numbers or strings, integer-scaled tracker output, a single combined column (`"55.75,37.61"`, WKT, GeoJSON), or an existing geohash/H3 index * **In-place or into a target table**: attributes written as fields merge into the existing row; written as tags they need a separate table * **HTTP backfill**: apply enrichment to history, retry only the rows that failed, or re-resolve everything after correcting a reference file * **Caches what it resolves**: a coordinate is resolved once and reused for every later point that rounds to the same key ## How enrichment lands on the row A row’s identity in InfluxDB 3 Enterprise is its **full tag set plus its timestamp**. That single fact decides where the plugin can write. Attributes written as **fields** leave the tag set untouched, so the write has the same identity as the original point and the columns merge into it: ``` gps,device=A lat=55.7558,lon=37.6173,speed=60 T ← your client gps,device=A geo_country="RU",geo_city="Moscow" T ← the plugin SELECT * FROM gps → device=A lat=55.7558 lon=37.6173 speed=60 geo_country=RU geo_city=Moscow ``` Attributes written as **tags** change the identity, producing a *second* row, so`output_mode=tag` requires `target_measurement` and the plugin rejects a configuration that would write tags back into the source table. |`output_mode`|destination | result | |-------------|------------|------------------------------------| | `field` |source table| merges into the existing row | | `field` |other table |full row copied, geo added as fields| | `tag` |other table | full row copied, geo added as tags | | `tag` |source table|rejected — would duplicate every row| Merging holds regardless of write order, survives compaction, and is not undone if your client re-sends the raw point later: each column keeps its last non-null value. ### The echo batch In-place mode writes into the very table the trigger watches, so the plugin’s own write comes back as a second invocation. Rows that already carry every configured output column are skipped, which ends the loop after that one extra pass. Expect roughly **twice the trigger invocations and twice the write volume**on an enriched table; where ingest headroom is tight, use `target_measurement`instead. ## Configuration ### Plugin metadata The plugin includes a JSON metadata schema in its docstring for[InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/)integration, defining `onwrite_args_config`, `http_args_config` and`http_body_config`. ### Core parameters | Parameter | Type | Default | Description | |---------------------|------|----------|----------------------------------------------------------------------------------------| |`source_measurements`|string|*required*| Space-separated tables to enrich. Other tables in the batch are ignored. | | `output_columns` |string|*required*|Space-separated `attribute:column` pairs, e.g. `country_code:geo_country city:geo_city`.| | `output_mode` |string| `field` | `field` or `tag`. `tag` requires `target_measurement`. | |`target_measurement` |string|*(empty)* | Destination table. Empty enriches the source table in place. | | `target_database` |string|*(empty)* | Destination database. Defaults to the trigger’s database. | | `unknown_value` |string|`UNKNOWN` | Written when a coordinate cannot be resolved. | In [TOML](#toml-configuration), `source_measurements` also takes a native array and `output_columns` a native table. Every configured column is always written; unresolved attributes get`unknown_value`. #### Column types Attributes are written as **strings**, including numbers such as `population`and any numeric GeoJSON property. Booleans are rendered as JSON, `true` /`false`. The one exception is `distance_m`, at every rank a float, which reports`-1` when nothing resolved. A numeric attribute therefore cannot be compared or aggregated as a number:`geo_pop > 1000000` compares text, and `'9' > '10381222'`. Cast it in the query, excluding the unresolved rows: ```sql SELECT * FROM gps WHERE CAST(geo_pop AS BIGINT) > 1000000 AND geo_pop != 'UNKNOWN' ``` ### Coordinate input Configure exactly one input mode. Columns may be tags or fields. | Parameter | Type | Default | Description | |---------------|------|---------|------------------------------------------------------------------| | `lat_field` |string| `lat` | Latitude column, number or string. | | `lon_field` |string| `lon` | Longitude column, number or string. | | `coord_scale` |number| `1` | Positive divisor turning scaled integers back into degrees. | | `point_field` |string|*(empty)*|Single column holding both coordinates, instead of the pair above.| |`point_format` |string|`lat_lon`| `lat_lon`, `lon_lat`, `wkt` (`POINT(lon lat)`) or `geojson`. | |`geohash_field`|string|*(empty)*| Geohash column, decoded to the cell center. | | `h3_field` |string|*(empty)*| H3 index column, decoded to the cell center. | Rows without usable coordinates are skipped. Coordinates outside [-90, 90] / [-180, 180] are counted and written as `unknown_value`. #### Scaled integer coordinates Many trackers report degrees multiplied by a fixed power of ten to avoid sending decimals, so `55.7558` arrives as `557558000`. `coord_scale` is the number both coordinates are divided by: | `lat` | `lon` |`coord_scale`| Resolved as | |-----------|-----------|-------------|----------------| |`557558000`|`376184000`| `1e7` |55.7558, 37.6184| |`55755800` |`37618400` | `1e6` |55.7558, 37.6184| | `55.7558` | `37.6184` |`1` (default)|55.7558, 37.6184| Pick the divisor that turns your raw number back into degrees: count the digits the device shifted. Write it plainly (`10000000`) or in scientific notation (`1e7`) — both are read as the same number. Any positive value is accepted, including fractions; zero and negatives are rejected when the trigger loads. The division happens after the coordinates are read, so it applies to every input mode, including `point_field`, `geohash_field` and `h3_field`. Those already decode to degrees, so scaling them is almost always a mistake — leave`coord_scale` at `1` unless the column truly holds scaled integers. ### Caching | Parameter |Type|Default | Description | |-------------------|----|--------|---------------------------------------------------------------------------| |`quantize_decimals`|int | `4` |Decimal places a coordinate is rounded to before it becomes a key. `0`–`9`.| | `cache_size` |int |`100000`| Distinct rounded coordinates kept before LRU eviction. `1` or more. | Resolving a coordinate costs far more than a dictionary lookup, and most coordinates repeat: a fixed sensor reports the same position thousands of times a day. Raw floats never repeat exactly — GPS noise moves the last digits — so the coordinate is rounded first and everything that rounds alike shares one result. |`quantize_decimals`| grid | effect | |-------------------|--------|---------------------------------------------------------| | 3 |≈ 111 m | very high hit rate, unusable near zone boundaries | | **4** | ≈ 11 m |default — GPS-noise sized, a stationary asset always hits| | 5 |≈ 1.1 m | near-exact, few hits from a moving asset | | 6 |≈ 0.11 m| effectively no rounding | The cost is a boundary error: a point within roughly that distance of a polygon edge may take a neighbor’s answer. At the default the window is smaller than consumer GPS error. Raise it where exact edge behavior matters. ### HTTP body parameters Every parameter above may be given in the request body under the same name, plus the backfill-only fields here. A request carries three layers of its own — the body, then the headers, then the query string, each overriding the one before — and all three override what the trigger holds: the[environment](#environment-variables), the trigger arguments, then the[TOML file](#toml-configuration) they name. A trigger created without arguments is configured by the request alone; one created with them holds the defaults each request overrides only where it names them. A body overrides a setting but cannot unset one: an empty string or `null`counts as absent, so the trigger’s value stands, and a body naming only `start`pairs it with the trigger’s `end`. A run that needs a setting the trigger holds to be empty, such as no `target_measurement` for an in-place run, needs a trigger that leaves it unset. `source_measurements` keeps its name but backfills one table per call: give several and the first is used, the rest are ignored with a warning. A field set to `null` counts as absent. A field the plugin does not know is refused with a 400, as is `config_file_path` whatever its value, so a misspelled name cannot pass unnoticed. | Parameter | Type | Default | Description | |---------------|------|---------|---------------------------------------------------------------------------------------------------| |`start` / `end`|string|*(empty)*|RFC 3339 bounds, given together. `start` inclusive, `end` exclusive. Omit both for the whole table.| | `batch_size` | int | `1000` | Rows read per page. Values below `1` are raised to `1`. | |`retry_unknown`| bool | `false` | Re-resolve rows whose geo column equals `unknown_value`. | | `force` | bool | `false` | Re-resolve every row regardless of its current values. | `start` and `end` keep nanosecond precision. In a TOML file write them as quoted strings: a bare TOML datetime keeps microseconds at most, so one is refused rather than truncated. `retry_unknown` and `force` take a JSON boolean or any of `true`/`false`, `yes`/`no`, `on`/`off`, `1`/`0` as a string; anything else is a 400. All five may also be set on the trigger or in its [TOML file](#toml-configuration), where they act as defaults the body overrides. Use `retry_unknown` after widening `max_radius_m`, and `force` after redrawing a zone — those rows already hold a resolved value, so `retry_unknown` would pass over them. The reference file is re-read on every HTTP call. #### Headers and query parameters The same names reach the plugin as headers, spelled`X-Influxdb3-Geo-Enrichment-` with underscores written as hyphens (`X-Influxdb3-Geo-Enrichment-Max-Radius-M` sets `max_radius_m`), and as query-string parameters, spelled exactly like the parameter (`?retry_unknown=true`). Header names are matched regardless of casing, which RFC 9110 makes meaningless. ```bash curl -X POST "http://localhost:8181/api/v3/engine/geo_backfill?retry_unknown=true" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -H "X-Influxdb3-Geo-Enrichment-Max-Radius-M: 5000" \ -d '{"start": "2026-08-01T00:00:00Z", "end": "2026-08-29T00:00:00Z"}' ``` A header the plugin does not ask for is ignored, since a client sends headers of its own on every request; an unknown query parameter is a 400, like an unknown body field, so a misspelled name cannot pass unnoticed. Neither layer can name`config_file_path`, for the reason the body cannot. ### Environment variables Every parameter can also come from an environment variable named`INFLUXDB3_GEO_ENRICHMENT_` in upper case — for example,`INFLUXDB3_GEO_ENRICHMENT_REFERENCE_FILE` sets `reference_file`. The environment is the lowest layer: a trigger argument overrides it, the TOML file overrides both, and on the HTTP trigger the request — its body, its headers and its query string — overrides them all.`INFLUXDB3_GEO_ENRICHMENT_CONFIG_FILE_PATH` names the TOML file when the trigger doesn’t carry a `config_file_path` argument. ### TOML configuration | Parameter | Type | Default | Description | |------------------|------|---------|---------------------------------------------------| |`config_file_path`|string|*(empty)*|`.toml` file, relative to `PLUGIN_DIR` or absolute.| A trigger argument on either trigger, or the`INFLUXDB3_GEO_ENRICHMENT_CONFIG_FILE_PATH` variable when the trigger leaves it unset; the file’s values override the other trigger arguments, and on the HTTP trigger the request overrides the file. The path is never read from a request — not from its body, its headers or its query string. It names a layer rather than setting a value: a request that could choose which file the trigger reads would take the trigger’s configuration out of the operator’s hands, so the body and the query string refuse it and a header spelling it is dropped. On the HTTP trigger the file is how a long setup is named once instead of repeated in every backfill request; each call then carries only what differs — usually the window: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "request:geo_backfill" \ --trigger-arguments 'config_file_path=geo_enrichment_config_data_writes.toml' \ geo_backfill ``` ```json { "start": "2026-08-01T00:00:00Z", "end": "2026-08-29T00:00:00Z", "force": true } ``` The five backfill fields may be set in the file as well, as the defaults the body overrides. ## Resolution strategies |Parameter | Type | Default | Description | |----------|------|---------|------------------------------------------| |`strategy`|string|`builtin`|`builtin`, `polygon`, `nearest` or `grid`.| One strategy is active per trigger. Each answers a different question and offers its own attributes for `output_columns`; asking for an attribute the active strategy cannot produce is a configuration error, reported at load time with the list of what is available. |Strategy | Answers | Reference data | Geometry it needs | |---------|-----------------------------|--------------------|---------------------| |`builtin`|what settlement is this near?|bundled, none to set| — | |`polygon`|which of my zones is this in?| `reference_file` |Polygon, MultiPolygon| |`nearest`|which of my sites is this at?| `reference_file` | Point | | `grid` | which grid cell is this in? | none | — | Each strategy’s own parameters are listed with it below. Parameters belonging to an inactive strategy are ignored, and no package is imported for a strategy or a file format you do not use. ### Reference file `polygon` and `nearest` read their zones or sites from one file. The **format**decides how it is read, the **strategy** decides how a point is matched against it, and the two are independent: either strategy takes either format. | Parameter | Type | Default | Description | |---------------------------|------|------------|--------------------------------------------------------------------| | `reference_file` |string| *required* |`.geojson`, `.json` or `.csv`, relative to `PLUGIN_DIR` or absolute.| | `reference_encoding` |string|`utf-8-sig` | Python codec name for a CSV file. GeoJSON is always UTF-8. | | `reference_lat_column` |string|*(detected)*| Latitude column of a CSV file. | | `reference_lon_column` |string|*(detected)*| Longitude column of a CSV file. | |`reference_geometry_column`|string|*(detected)*| WKT column of a CSV file. | #### Where the geometry comes from **GeoJSON** — a `FeatureCollection`, or a single `Feature`. Each feature’s`geometry` is the shape and its `properties` are the attributes. **CSV** — either a column of WKT, or a pair of coordinate columns: ```csv zone,geometry plant-A,"POLYGON((37.5 55.7, 37.8 55.7, 37.8 55.9, 37.5 55.9, 37.5 55.7))" ``` ```csv code,lat,lon,region KONA-01,19.64,-155.99,HI-WEST ``` Detection runs in this order, and an explicit setting always wins: 1. `reference_geometry_column`, read as WKT; 2. `reference_lat_column` / `reference_lon_column`; 3. a column named `geometry` or `wkt`, read as WKT; 4. a latitude column (`lat`, `latitude`) and a longitude column (`lon`, `lng`,`long`, `longitude`). Names are matched ignoring case. Setting a geometry column *and* a coordinate column is a configuration error — they are alternatives. When nothing matches, the error lists the columns actually parsed, along with the delimiter used to parse them. Every column that did not become geometry is an attribute. #### Reading a CSV The delimiter is detected from the header — comma, semicolon, tab and pipe are recognized, and a comma inside a quoted value does not confuse it. When the delimiter is *not* a comma, a comma inside a number is unambiguous and is read as a decimal mark, so European exports work as they are: ```csv code;lat;lon center;"55,7512";"37,6184" ``` The default encoding accepts UTF-8 with or without a byte-order mark, which covers files saved by Excel. Set `reference_encoding` for anything else, such as`cp1251`. GeoJSON is always UTF-8 by RFC 7946, so the setting does not apply to it. #### How attributes are named |Format |An attribute in `output_columns` is| |-------|-----------------------------------| |GeoJSON| a **JSONPath** into `properties` | | CSV | a **column name** | GeoJSON properties are arbitrary JSON, so a path reaches nested values; a CSV row is flat, so the name is the column. ``` GeoJSON: output_columns='zone:geo_zone owner.name:geo_owner owner.contact.email:geo_email' CSV: output_columns='zone:geo_zone owner:geo_owner' ``` In a path, `codes[0]` indexes an array, and a property whose own name contains a dot is reached by quoting it: `"odd.name"`. A path missing from *some* features is normal — those get `unknown_value`: ``` zones: {"zone": "plant-A", "owner": {"name": "ACME"}} {"zone": "plant-B"} output_columns='zone:geo_zone owner.name:geo_owner' → point in A: geo_zone=plant-A geo_owner=ACME → point in B: geo_zone=plant-B geo_owner=UNKNOWN ``` A path missing from *every* feature is a configuration error instead, because it is almost always a typo, and writing `UNKNOWN` forever would hide it behind a column that merely looks empty: ``` output_columns='zone:geo_zone owner.phone:geo_phone' → Configuration error: output_columns attribute 'owner.phone' matches no feature. ``` Paths are resolved once when the file is indexed, so they cost nothing per row and every mistake surfaces there rather than in your data: ``` 'owner' → resolves to a dict; only single values can be written. Point the path at a leaf, e.g. 'owner.'. 'codes[*]' → matches 2 values in one feature; a column holds a single value. 'owner[' → is not a valid JSONPath: Parse error near the end of string! ``` #### Geometry the strategy cannot use `polygon` needs areal geometry, `nearest` needs points. An entry of the wrong kind is skipped with a warning, so a stray label point among fifty zones does not stop the trigger; a file with **nothing** usable fails at startup rather than writing `UNKNOWN` into every row. A point is never reduced to a polygon’s centroid: the centroid of a region is not where anything is. Coordinates are read as WGS84 degrees, longitude first in WKT and GeoJSON. Data in a projected system is rejected as out of range rather than misread. ### `builtin` — nearest populated place, offline Answers “what settlement is this point in?” from the GeoNames snapshot bundled with `reverse_geocode`. No reference file to prepare — this is the zero-setup path. Attributes: `country_code`, `country`, `state`, `city`, `population`,`distance_m`. | Parameter | Type | Default | Description | |----------------|------|----------|-----------------------------------------------------------------------| |`min_population`| int | `0` | Consider only places at least this populous. `0` or more. | | `max_radius_m` |number|*no limit*|Meters. Points farther than this from the place are unknown. Above `0`.| `min_population` is a zoom control — the dataset goes down to hamlets, so a point on a city’s edge resolves to a suburb: ``` (55.5800, 37.5000) outskirts of Moscow min_population=0 → Kommunarka 4,684 min_population=10000 → Yasenevo 180,000 min_population=1000000 → Moscow 10,381,222 ``` The match is the nearest **city center**, not a boundary the point falls inside, so distances of a few kilometers are normal and mean nothing is wrong. There is no radius by default — a 1 km limit would discard roughly 8 of every 10 points in a city and virtually everything outside one. Set `max_radius_m` when a plausible-looking wrong answer is worse than a blank: a point in Antarctica resolves to South Africa, 5,000 km away, and a mid-ocean point to French Polynesia. Map `distance_m` to a column first and query it — that shows the real spread of your own data before you pick a number. > > > Raising `min_population` moves the answer farther away, so pair the two:`(41.6, -93.9)` in rural Iowa resolves to Waukee at `min_population=0`, but to**Chicago, some 500 km away**, at `min_population=1000000`. > > ### `polygon` — point inside a zone you drew Answers “which of my zones is this point in?” Attributes: whatever the[reference file](#reference-file) supplies. | Parameter | Type | Default | Description | |--------------------|------|----------|------------------------------------------------------------------------------------------| | `overlap_policy` |string|`smallest`|`smallest`, `largest`, `first` or `priority`. Winner when a point is inside several zones.| |`priority_attribute`|string|*(empty)* | Attribute ranked when `overlap_policy=priority`. Read only by that policy. | Zones usually overlap because they nest — a building inside a plant inside a region — so a point matches all three: * `smallest` — smallest area, the most specific zone. The default, because nesting is the common case. * `largest` — largest area, the most general zone. The region rather than the building, for rolling a detailed file up to a coarse column. * `first` — file order. Cheapest and fully predictable when zones never overlap. * `priority` — highest value of `priority_attribute`. One reference file can therefore feed two triggers at different levels of detail: ```bash --trigger-arguments 'output_columns=zone:geo_building,overlap_policy=smallest' --trigger-arguments 'output_columns=zone:geo_region,overlap_policy=largest' ``` ```json { "type": "FeatureCollection", "features": [ { "type": "Feature", "properties": { "facility": "KONA-01", "region": "HI-WEST" }, "geometry": { "type": "Polygon", "coordinates": [[[-156.0,19.6],[-155.9,19.6],[-155.9,19.7],[-156.0,19.7],[-156.0,19.6]]] } } ] } ``` A zone crossing the antimeridian must be **split at it**, as RFC 7946 requires. An unsplit ring runs the wrong way around the globe: it stops matching its own interior and starts matching the opposite side of the planet. The plugin warns at startup about any geometry spanning more than 180° of longitude. ### `nearest` — closest site from a list Answers “which of my sites is this point at?” for anyone who has site coordinates but no drawn boundaries. Attributes: whatever the[reference file](#reference-file) supplies, plus `distance_m`. | Parameter | Type |Default| Description | |---------------|------|-------|------------------------------------------------------------------------| |`nearest_count`| int | `1` | How many closest sites to describe. `1` or more. | |`max_radius_m` |number|`1000` |Meters. Points farther than this from every site are unknown. Above `0`.| Matching runs on the unit sphere, so the nearest site is the true great-circle nearest, not an artifact of longitude compression. `max_radius_m` is what keeps “nearest” meaningful — without it every point on Earth belongs to some site. Map `distance_m` to see how good each match was: a truck 40 m from the depot is *at* the depot, one 900 m away merely happens to be closest. Because it is a float column it cannot hold `unknown_value`; unresolved rows get **`-1`**. Raise it well past the default for assets that spend most of their time in transit. A fleet on the highway sits tens of kilometers from every depot, so at`1000` every row reads `UNKNOWN` and `-1` until the vehicle pulls into a yard. There is no “no limit” keyword: pass a value larger than half the Earth’s circumference, such as `max_radius_m=20000000`. #### More than one site per point `nearest_count` describes the closest *N* sites instead of just the closest one. Ranks after the first repeat every output column with a `_2`, `_3` suffix: ```bash --trigger-arguments 'output_columns=code:geo_site distance_m:geo_dist,nearest_count=3' ``` |geo\_site|geo\_dist|geo\_site\_2|geo\_dist\_2|geo\_site\_3|geo\_dist\_3| |---------|---------|------------|------------|------------|------------| |`KONA-01`| `189.7` | `WAIM-03` | `53731.9` | `HILO-02` | `95515.9` | A suffix group describes one site completely, so map the site’s name alongside its distance — `geo_dist_2` on its own does not say what it measured. Ranks are filtered by `max_radius_m` individually: a second site outside the radius leaves that group unresolved while the first stays populated. Asking for more sites than the file holds is not an error; the surplus ranks are always unresolved and the plugin warns once at startup. Ranks come from distance alone, so mid-route the neighbors are simply whatever the vehicle is driving past — read them as “what is nearby”, not as the route. With `output_mode=tag` every rank becomes a tag, and the plugin warns about it. The series key then covers the whole combination of ranks, and two nearly equidistant sites trade places as the point moves, opening a new series on each swap. Distances are floats and stay fields either way. ### `grid` — cell of a global grid Answers “which cell of a fixed worldwide grid is this point in?” No reference data. All points in a cell collapse to one value you can `GROUP BY`, which is what makes heatmaps possible when the query engine has no geo functions of its own. Attribute: `cell`. | Parameter | Type | Default | Description | |----------------|------|---------------|---------------------------------------------------| | `grid_type` |string| `h3` | `h3`, `geohash` or `s2`. | |`grid_precision`| int |`7` / `6` / `9`|Cell size. Range depends on `grid_type`, see below.| * **`h3`** — hexagons. Every neighbor is equidistant from the center, so neighborhood and distance analysis behaves well. Resolutions 0–15. * **`geohash`** — lat/lon rectangles. The identifier’s prefix is a coarser cell, so truncating it zooms out. Cells stretch away from the equator. Lengths 1–12. * **`s2`** — spherical quadrilaterals with even areas worldwide. Levels 0–30. |`grid_type`|`grid_precision`| cell size | |-----------|----------------|-----------------------| | `h3` | 6 |36.1 km², edge 3 724 m | | `h3` | **7** |5.16 km², edge 1 406 m | | `h3` | 8 | 0.74 km², edge 531 m | | `h3` | 9 | 0.11 km², edge 201 m | | `geohash` | 5 | 4 892 × 4 892 m | | `geohash` | **6** | 611 × 1 223 m | | `geohash` | 7 | 153 × 153 m | | `s2` | **9** | 324 km², side ≈ 18 km | | `s2` | 11 |20.3 km², side ≈ 4.5 km| | `s2` | 13 |1.27 km², side ≈ 1.1 km| > > > **Precision is a cardinality control.** Each finer step multiplies the distinct > values the column can take — the very thing this plugin otherwise exists to > contain. Stay at or above the defaults unless you have measured the effect. > > ## Software Requirements * **InfluxDB 3 Enterprise**: 3.8.2 or later, with the Processing Engine enabled * **Python packages**: `influxdata-plugin-utils>=0.4.0`, plus the packages for the strategies you use ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled: ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install the packages: ``` influxdb3 install package influxdata-plugin-utils influxdb3 install package reverse_geocode # strategy=builtin influxdb3 install package shapely # strategy=polygon influxdb3 install package scipy # strategy=nearest influxdb3 install package jsonpath-ng # a GeoJSON reference_file influxdb3 install package h3 # grid_type=h3, h3_field influxdb3 install package pygeohash # grid_type=geohash, geohash_field influxdb3 install package s2sphere # grid_type=s2 ``` Packages are imported lazily, so only install what your configuration uses. ## Trigger setup ### Write trigger (live enrichment) ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "table:gps" \ --trigger-arguments 'source_measurements=gps,output_columns=country_code:geo_country city:geo_city,strategy=builtin' \ geo_enrich_gps influxdb3 enable trigger --database mydb geo_enrich_gps ``` ### HTTP trigger (backfill) Without trigger arguments the request body carries the whole configuration; with them, they are the defaults the body overrides. See[TOML configuration](#toml-configuration) for naming a file on the trigger. ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "request:geo_backfill" \ geo_backfill influxdb3 enable trigger --database mydb geo_backfill curl -X POST "http://localhost:8181/api/v3/engine/geo_backfill" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "source_measurements": "gps", "output_columns": "country_code:geo_country city:geo_city", "strategy": "builtin", "start": "2026-08-01T00:00:00Z", "end": "2026-08-29T00:00:00Z", "retry_unknown": true }' ``` **Expected response:** ```json { "status": "ok", "measurement": "gps", "stats": { "rows": 5730, "resolved": 5719, "unresolved": 11, "no_coordinates": 0, "invalid_coordinates": 0, "skipped_enriched": 0, "cache_hits": 5602, "cache_misses": 128, "errors": 0, "written": 5730 } } ``` ## Example usage ### Example 1: Country and city on fleet positions (in place) ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "table:gps" \ --trigger-arguments 'source_measurements=gps,output_columns=country_code:geo_country city:geo_city,strategy=builtin,min_population=10000' \ geo_country ``` ```bash influxdb3 write --database mydb 'gps,device=truck7 lat=55.7558,lon=37.6173,speed=54' influxdb3 query --database mydb "SELECT device, speed, geo_country, geo_city FROM gps" ``` ``` device speed geo_country geo_city truck7 54 RU Moscow ``` ### Example 2: Your own plant zones, as real tags ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "table:gps" \ --trigger-arguments 'source_measurements=gps,target_measurement=gps_zoned,output_mode=tag,output_columns=facility:facility region:region,strategy=polygon,reference_file=/plugins/data/zones.geojson,overlap_policy=smallest' \ geo_zones ``` ```bash influxdb3 query --database mydb \ "SELECT facility, avg(speed) FROM gps_zoned GROUP BY facility" ``` ### Example 3: Nearest depot with match quality ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "table:gps" \ --trigger-arguments 'source_measurements=gps,output_columns=code:geo_site distance_m:geo_distance,strategy=nearest,reference_file=/plugins/data/sites.csv,max_radius_m=500' \ geo_sites ``` ```bash influxdb3 query --database mydb \ "SELECT geo_site, count(*) FROM gps WHERE geo_distance >= 0 GROUP BY geo_site" ``` ### Example 4: H3 cells for a heatmap ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/geo_enrichment/geo_enrichment.py \ --trigger-spec "table:gps" \ --trigger-arguments 'source_measurements=gps,output_columns=cell:geo_cell,strategy=grid,grid_type=h3,grid_precision=8' \ geo_cells ``` ```bash influxdb3 query --database mydb \ "SELECT geo_cell, count(*) AS hits FROM gps GROUP BY geo_cell ORDER BY hits DESC" ``` ### Example 5: Integer-encoded tracker with a combined column ```bash --trigger-arguments 'source_measurements=tracker,output_columns=country_code:geo_country,point_field=pos,point_format=lon_lat,coord_scale=1e7' ``` ## Code overview ### Files * `geo_enrichment.py` — the plugin * `geo_enrichment_config_data_writes.toml` — annotated configuration template * `manifest.toml` — plugin manifest * `requirements.txt` — runtime dependencies * `test_geo_enrichment.py` — pytest suite, runs without a live InfluxDB 3 Enterprise server * `README.md` — this documentation ### Logging Logs go to the `system.processing_engine_logs` table. Each run logs a summary: ``` rows=1200 resolved=1180 unresolved=20 no_coordinates=0 invalid=0 already_enriched=600 cache_hits=1150 cache_misses=50 errors=0 written=1200 ``` `already_enriched` counts echo-batch rows, so on a healthy in-place trigger it is roughly equal to the rows written on the previous pass. ### Main functions #### `process_writes(influxdb3_local, table_batches, args)` Enriches each WAL flush. Skips rows that already carry the output columns, then extracts, validates, resolves and writes. Writes do not retry — a WAL-flush trigger runs inline with ingestion, so a backoff sleep would throttle it. #### `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)` Pages through a time range with the same pipeline and retries failed writes. Re-reads the reference data on every call. #### Configuration Each source — the trigger arguments, the TOML file they name, the request body — is read by its own `influxdata-plugin-utils` parser, and each entry point hands the layers to `load_config()` lowest precedence first. `SETTING_VALIDATORS`gives every setting its default, cast and checks; `BACKFILL_VALIDATORS` does the same for the five per-request fields, which only the HTTP trigger applies, so a file shared with it cannot stop the write trigger. `prepare_config()` then checks the settings against each other, with messages that name the setting making the demand, and adds what the pipeline derives from them. #### `read_reference(cfg, requested_attributes)` Reads the reference file by extension into geometry/attribute records, which both`polygon` and `nearest` build their index from. #### `resolve_attributes(cfg, resolver, memo, lat, lon)` Rounds the coordinate, reuses a memoized result, otherwise calls the resolver. #### `build_enrichment_line(row, table, values, cfg, schema)` Builds the output line: in place only the source tags are reproduced, so the write merges; to a target table the whole row is copied. ## Troubleshooting ### Issue: “output\_mode=‘tag’ needs ’target\_measurement’” A tag is part of a row’s identity, so writing one into the source table creates a second row and doubles every aggregate. Either set `target_measurement`, or use`output_mode=field` to enrich in place. ### Issue: “Request body may not set ‘config\_file\_path’” `config_file_path` is a trigger argument, never a body field. ### Issue: rows have `UNKNOWN` everywhere * `strategy=nearest`: every site is farther than `max_radius_m`. Map`distance_m` to a column and query it to see the real distances. When only the`_2` and later groups are unresolved, either the radius excludes them or`nearest_count` exceeds the number of sites in the file. * `strategy=polygon`: the point falls outside every zone. Check that the file uses `[longitude, latitude]` order, which both GeoJSON and WKT require and which is the reverse of how coordinates are usually spoken. ### Issue: no geo columns appear at all * The trigger table must be listed in `source_measurements`. * Rows without usable coordinates are skipped silently; check `no_coordinates`in the summary log. * If coordinates arrive as scaled integers, set `coord_scale`. ### Issue: “‘attribute’ cannot be produced by strategy” `output_columns` names an attribute the strategy does not have. The error lists what is available; for `polygon` and `nearest` that comes from your own file, so a typo in a CSV header shows up here. A typo in a GeoJSON path is reported separately, as a path matching no feature. ### Issue: field type conflict on write Geo attributes are written as strings and `distance_m` as a float. If a column of that name already exists with another type, the whole write batch is rejected. Choose different column names, or drop the old column. ### Issue: the trigger fires twice per write Expected in place — see [The echo batch](#the-echo-batch). The second pass writes nothing. ### Issue: a package is missing The error names the package and what needs it — a strategy, a coordinate input, or the reference file’s format: ``` 'strategy=polygon' needs the 'shapely' package. Install it with 'influxdb3 install package shapely'. ``` ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Geo enrichment plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/geo_enrichment) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | output_mode | destination | result | | --- | --- | --- | | output_mode | destination | result | | field | source table | merges into the existing row | | field | other table | full row copied, geo added as fields | | tag | other table | full row copied, geo added as tags | | tag | source table | rejected — would duplicate every row | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_measurements | string | required | Space-separated tables to enrich. Other tables in the batch are ignored. | | output_columns | string | required | Space-separated attribute:column pairs, e.g. country_code:geo_country city:geo_city . | | output_mode | string | field | field or tag . tag requires target_measurement . | | target_measurement | string | (empty) | Destination table. Empty enriches the source table in place. | | target_database | string | (empty) | Destination database. Defaults to the trigger’s database. | | unknown_value | string | UNKNOWN | Written when a coordinate cannot be resolved. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | lat_field | string | lat | Latitude column, number or string. | | lon_field | string | lon | Longitude column, number or string. | | coord_scale | number | 1 | Positive divisor turning scaled integers back into degrees. | | point_field | string | (empty) | Single column holding both coordinates, instead of the pair above. | | point_format | string | lat_lon | lat_lon , lon_lat , wkt ( POINT(lon lat) ) or geojson . | | geohash_field | string | (empty) | Geohash column, decoded to the cell center. | | h3_field | string | (empty) | H3 index column, decoded to the cell center. | | lat | lon | coord_scale | Resolved as | | --- | --- | --- | --- | | lat | lon | coord_scale | Resolved as | | 557558000 | 376184000 | 1e7 | 55.7558, 37.6184 | | 55755800 | 37618400 | 1e6 | 55.7558, 37.6184 | | 55.7558 | 37.6184 | 1 (default) | 55.7558, 37.6184 | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | quantize_decimals | int | 4 | Decimal places a coordinate is rounded to before it becomes a key. 0 – 9 . | | cache_size | int | 100000 | Distinct rounded coordinates kept before LRU eviction. 1 or more. | | quantize_decimals | grid | effect | | --- | --- | --- | | quantize_decimals | grid | effect | | 3 | ≈ 111 m | very high hit rate, unusable near zone boundaries | | 4 | ≈ 11 m | default — GPS-noise sized, a stationary asset always hits | | 5 | ≈ 1.1 m | near-exact, few hits from a moving asset | | 6 | ≈ 0.11 m | effectively no rounding | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | start / end | string | (empty) | RFC 3339 bounds, given together. start inclusive, end exclusive. Omit both for the whole table. | | batch_size | int | 1000 | Rows read per page. Values below 1 are raised to 1 . | | retry_unknown | bool | false | Re-resolve rows whose geo column equals unknown_value . | | force | bool | false | Re-resolve every row regardless of its current values. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | (empty) | .toml file, relative to PLUGIN_DIR or absolute. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | strategy | string | builtin | builtin , polygon , nearest or grid . | | Strategy | Answers | Reference data | Geometry it needs | | --- | --- | --- | --- | | Strategy | Answers | Reference data | Geometry it needs | | builtin | what settlement is this near? | bundled, none to set | — | | polygon | which of my zones is this in? | reference_file | Polygon, MultiPolygon | | nearest | which of my sites is this at? | reference_file | Point | | grid | which grid cell is this in? | none | — | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | reference_file | string | required | .geojson , .json or .csv , relative to PLUGIN_DIR or absolute. | | reference_encoding | string | utf-8-sig | Python codec name for a CSV file. GeoJSON is always UTF-8. | | reference_lat_column | string | (detected) | Latitude column of a CSV file. | | reference_lon_column | string | (detected) | Longitude column of a CSV file. | | reference_geometry_column | string | (detected) | WKT column of a CSV file. | | Format | An attribute in output_columns is | | --- | --- | | Format | An attribute in output_columns is | | GeoJSON | a JSONPath into properties | | CSV | a column name | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | min_population | int | 0 | Consider only places at least this populous. 0 or more. | | max_radius_m | number | no limit | Meters. Points farther than this from the place are unknown. Above 0 . | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | overlap_policy | string | smallest | smallest , largest , first or priority . Winner when a point is inside several zones. | | priority_attribute | string | (empty) | Attribute ranked when overlap_policy=priority . Read only by that policy. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | nearest_count | int | 1 | How many closest sites to describe. 1 or more. | | max_radius_m | number | 1000 | Meters. Points farther than this from every site are unknown. Above 0 . | | geo_site | geo_dist | geo_site_2 | geo_dist_2 | geo_site_3 | geo_dist_3 | | --- | --- | --- | --- | --- | --- | | geo_site | geo_dist | geo_site_2 | geo_dist_2 | geo_site_3 | geo_dist_3 | | KONA-01 | 189.7 | WAIM-03 | 53731.9 | HILO-02 | 95515.9 | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | grid_type | string | h3 | h3 , geohash or s2 . | | grid_precision | int | 7 / 6 / 9 | Cell size. Range depends on grid_type , see below. | | grid_type | grid_precision | cell size | | --- | --- | --- | | grid_type | grid_precision | cell size | | h3 | 6 | 36.1 km², edge 3 724 m | | h3 | 7 | 5.16 km², edge 1 406 m | | h3 | 8 | 0.74 km², edge 531 m | | h3 | 9 | 0.11 km², edge 201 m | | geohash | 5 | 4 892 × 4 892 m | | geohash | 6 | 611 × 1 223 m | | geohash | 7 | 153 × 153 m | | s2 | 9 | 324 km², side ≈ 18 km | | s2 | 11 | 20.3 km², side ≈ 4.5 km | | s2 | 13 | 1.27 km², side ≈ 1.1 km | --- # Import plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/import/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The InfluxDB Import Plugin enables seamless data import from InfluxDB v1, v2, or v3 instances to InfluxDB 3 Enterprise Core/Enterprise. It provides comprehensive import capabilities with pause/resume functionality, progress tracking, conflict detection, and robust error handling. The plugin operates via HTTP endpoints, allowing you to start, pause, resume, cancel, and monitor imports through simple HTTP requests. Key features: * Import data from InfluxDB v1, v2, or v3 to InfluxDB 3 Enterprise * Automatic data sampling for optimal batch sizing * Memory bounded by `target_batch_size`, however much data a time window holds * Resume interrupted imports from the last checkpoint * Pause and cancel running imports * Crash recovery with stale import detection * Progress tracking and statistics * Tag/field conflict detection and resolution * Data type mismatch handling * Configurable time ranges and table filtering * Dry run mode for import planning (estimates, schema conflicts, configuration preview) * Support for both token and username/password authentication ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI), in the`trigger_arguments`field (API) when creating a trigger or via body of HTTP request. This plugin supports TOML configuration files, which can be specified using the `config_file_path` parameter. A key that is not a plugin parameter fails the request and is named in the error — in the trigger arguments, in the TOML file and in the request body alike. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |------------------|-------|--------|------------------------------------------------------------------------------| | `source_url` |string |required|Source InfluxDB URL (with optional port, for example, `http://localhost:8086`)| |`influxdb_version`|integer|required| Source InfluxDB version: 1, 2, or 3 | |`source_database` |string |required| Source database name to import from | ### Authentication Credentials are passed via HTTP headers on each request. | Header | Purpose | |-----------------|---------------------------------------------------| | `Source-Token` |Bearer/API token authentication (InfluxDB v1/v2/v3)| |`Source-Username`| Basic auth username (InfluxDB v1) | |`Source-Password`| Basic auth password (InfluxDB v1) | **Method 1: Token-based authentication** Use the `Source-Token` header for token-based authentication: ```bash curl -X POST http://localhost:8181/api/v3/engine/import?action=start \ -H "Content-Type: application/json" \ -H "Source-Token: my-secret-token" \ -d '{ "source_url": "http://localhost:8086", "influxdb_version": 2, "source_database": "telegraf" }' ``` **Method 2: Username/Password authentication** Use the `Source-Username` and `Source-Password` headers for basic authentication: ```bash curl -X POST http://localhost:8181/api/v3/engine/import?action=start \ -H "Content-Type: application/json" \ -H "Source-Username: admin" \ -H "Source-Password: my-password" \ -d '{ "source_url": "http://localhost:8086", "influxdb_version": 1, "source_database": "telegraf" }' ``` > > > **Note**: Authentication errors from the source InfluxDB are returned directly. The plugin does not validate > credentials upfront. > > ### Optional parameters | Parameter | Type | Default | Description | |-------------------|--------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `dest_database` | string | none | Destination database name in InfluxDB 3 Enterprise (if not specified, uses database where trigger was created) | | `start_timestamp` | string | none | Import start time — RFC3339, Unix seconds or nanoseconds, or `YYYY-MM-DD`; a value naming no zone is read as UTC. If not specified, starts from oldest data | | `end_timestamp` | string | none | Import end time, same formats as `start_timestamp`. If not specified, imports to newest data | |`query_interval_ms`| integer | 100 | Delay between queries in milliseconds to avoid overloading source database (0 or greater) | |`import_direction` | string |“oldest\_first”| Import direction: “oldest\_first” or “newest\_first” | |`target_batch_size`| integer | 2000 | Rows per chunk read from the source and per write, and the target rows per query window (1 or greater) | | `table_filter` |string | array| none |Tables to import: a JSON array (request body or TOML), e.g. `["cpu", "mem", "disk"]`, which keeps a name containing a dot intact; or a dot-separated string elsewhere, e.g. `cpu.mem.disk`. A space in a name is written as an underscore (noted under schema issues); if two names sanitize to the same one, only the first imports and the rest are reported `failed`. If not specified, imports all tables| | `dry_run` | boolean | false | If true, generates import plan without processing data (shows estimates, schema conflicts, and configuration) | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|-----------------------------------------------------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path, absolute or relative to the plugin directory. Trigger arguments or `INFLUXDB3_IMPORT_CONFIG_FILE_PATH`| *A relative `config_file_path` is resolved against the plugin directory, taken from `PLUGIN_DIR`, then`INFLUXDB3_PLUGIN_DIR`, then the parent of the Processing Engine virtual environment. The last of these is always set, so a file placed next to the plugin is found without any environment variable.* The path comes from the trigger arguments or from `INFLUXDB3_IMPORT_CONFIG_FILE_PATH`, the argument winning; a request body that carries`config_file_path` is refused. #### Example TOML configuration [import\_config.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/import/import_config.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Source InfluxDB instance**: InfluxDB v1.x or v2.x instance accessible via HTTP/HTTPS. * **Python packages**: * `influxdata-plugin-utils>=0.5.0` (shared configuration, validation and write helpers) * `requests` (for HTTP communication with source InfluxDB) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine: ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.5.0" influxdb3 install package requests ``` ## Trigger setup ### HTTP trigger setup Create an HTTP trigger to handle import requests: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/import/import.py \ --trigger-spec "request:import" \ --run-asynchronous \ import_trigger ``` Enable the trigger: ```bash influxdb3 enable trigger --database mydb import_trigger ``` The endpoint is registered at `/api/v3/engine/import`. ### Why `--run-asynchronous` is required An import runs inside the HTTP request that starts it, and without this flag the Processing Engine serves one invocation of a trigger at a time. A `pause`,`status` or `cancel` request would then wait in line until the import it is meant to control has already finished. With the flag, the engine runs invocations of the trigger concurrently, and the control actions are answered while the import is in progress. Without the flag the plugin still imports data correctly — only the control actions become unusable. ## HTTP Endpoint The import plugin provides the following type of requests: ### Start Import Start a new import from source InfluxDB to InfluxDB 3 Enterprise. **Request**: `POST /api/v3/engine/import?action=start` **Headers**: * `Source-Token: my-token` (or `Source-Username` + `Source-Password`) * `Content-Type: application/json` **Request body** (JSON): ```json { "source_url": "http://localhost:8086", "influxdb_version": 1, "source_database": "telegraf", "dest_database": "imported_data", "start_timestamp": "2024-01-01T00:00:00Z", "end_timestamp": "2024-12-31T23:59:59Z", "table_filter": ["cpu", "mem", "disk"] } ``` Any of these keys may arrive as a query parameter or as an`X-Influxdb3-Import-` header instead, both of which override the body. See “Configuration Priority and Loading” below. ### Get Import Status Check the status and progress of a import. **Request**: `GET /api/v3/engine/import?action=status&import_id=` The answer holds `summary` with the table counts and `total_rows_imported`,`table_details` with one entry per table, and `config` with the settings the import was started with. A table that failed to write some of its windows reports them under `table_details[].errors`, and `summary.tables_with_errors`counts such tables. A space in a measurement name is written as an underscore and noted under schema issues; a table that wrote no rows while windows failed, or whose sanitized name collides with another table’s (only the first of the two imports), ends with status `failed`;`summary.failed_tables` counts these, and `overall_status` stays `completed`once every table is either completed or failed. ### Pause Import Pause a running import to resume later. **Request**: `POST /api/v3/engine/import?action=pause&import_id=` > > > **Note**: Returns error if import is not found, already paused, already cancelled, or already completed. > > ### Resume Import Resume a paused or interrupted import. **Request**: `POST /api/v3/engine/import?action=resume&import_id=` **Headers**: * `Source-Token: my-token` (or `Source-Username` + `Source-Password`) > > > **Note**: Credentials must be provided via headers when resuming. Authentication credentials are not stored for > security reasons and must be provided when resuming. Returns error if import is not found, already cancelled, already > completed, or actively running. > > The response field `errors` is the number of windows that failed across every run of the import, not just this one. The failures themselves stay in the `errors` column of `import_state`. #### Crash recovery If the plugin or the server crashes during an import, the import state may be left as “running” even though nothing is actually running. The resume action handles this with **stale import detection**: * When the import state is “running”, the plugin checks the timestamp of the last `import_state` record. * If the last update is older than **5 minutes**, the import is considered stale (crashed) and resume is allowed. * If no `import_state` records exist at all (the import crashed before processing any tables), the import is restarted from the beginning. Each table then continues from its own checkpoint. A table that had written rows resumes after the last of them, even part way through a window; a table that had written none, including one that never started, is imported from the beginning. If the plugin itself crashes (for example, source database becomes unavailable), it writes a paused state before exiting, so the import can be resumed with a regular resume call after fixing the issue. ### Cancel Import Cancel a running import. Cancelled imports cannot be resumed. **Request**: `POST /api/v3/engine/import?action=cancel&import_id=` > > > **Note**: Returns error if import is not found, already cancelled, or already completed. > > ### Test Connection Test connectivity to a URL and identify if it’s an InfluxDB instance. Uses a 5-second timeout for fast feedback. **Request**: `POST /api/v3/engine/import?action=test_connection` **Request body** (JSON): ```json { "source_url": "http://localhost:8086" } ``` > > > **Note**: If port is omitted, it is inferred from the scheme (`http` → 80, `https` → 443). > > **Success response** (InfluxDB v1/v2 detected): ```json { "success": true, "version": "2.7.0", "build": "OSS" } ``` **Success response** (InfluxDB v3 detected via `cluster-uuid` header): ```json { "success": true, "version": "3.x.x", "build": "" } ``` > > > **Note**: InfluxDB v3 does not expose version headers without authentication. Detection uses the `cluster-uuid` header > instead. > > **Failure response** (not InfluxDB or unreachable): ```json { "success": false, "message": "Not an InfluxDB instance" } ``` **Failure response** (InfluxDB requires authentication, version unknown): ```json { "success": false, "message": "Unable to determine InfluxDB version" } ``` > > > **Note**: When InfluxDB returns 401/403 without version headers, the connection test cannot determine the version. > This typically means authentication is required. The instance is likely InfluxDB, but version detection requires valid > credentials. > > ### List Databases Get list of databases from source InfluxDB instance. **Request**: `POST /api/v3/engine/import?action=databases` **Headers**: * `Source-Token: my-token` (or `Source-Username` + `Source-Password`) * `Content-Type: application/json` **Request body** (JSON): ```json { "source_url": "http://localhost:8086", "influxdb_version": 1 } ``` ### List Tables Get list of tables/measurements from a source database. **Request**: `POST /api/v3/engine/import?action=tables` **Headers**: * `Source-Token: my-token` (or `Source-Username` + `Source-Password`) * `Content-Type: application/json` **Request body** (JSON): ```json { "source_url": "http://localhost:8086", "influxdb_version": 1, "source_database": "telegraf" } ``` ## Example usage ### Example 1: Basic import with token authentication Import all data from an InfluxDB v1 instance: ```bash # Create and enable HTTP trigger influxdb3 create trigger \ --database mydb \ --plugin-filename import.py \ --trigger-spec "request:import" \ --run-asynchronous \ import_trigger influxdb3 enable trigger --database mydb import_trigger # Start import via HTTP curl -X POST http://localhost:8181/api/v3/engine/import?action=start \ -H "Source-Token: my-super-secret-token" \ -H "Content-Type: application/json" \ -d '{ "source_url": "http://localhost:8086", "influxdb_version": 1, "source_database": "telegraf", "dest_database": "imported_data" }' ``` ### Expected results * Plugin connects to source InfluxDB at `http://localhost:8086` (port from URL) * Discovers all measurements in the `telegraf` database * Estimates import time based on data sampling * Imports all data to InfluxDB 3 Enterprise in the `imported_data` database * Logs import\_id for tracking statistics ### Example 2: Time-range import with table filtering Import specific tables within a date range: ```bash # Start import with time range and table filter curl -X POST http://localhost:8181/api/v3/engine/import?action=start \ -H "Source-Username: admin" \ -H "Source-Password: my-password" \ -H "Content-Type: application/json" \ -d '{ "source_url": "http://influxdb-source.example.com:8086", "influxdb_version": 1, "source_database": "telegraf", "dest_database": "production_metrics", "start_timestamp": "2024-01-01T00:00:00Z", "end_timestamp": "2024-12-31T23:59:59Z", "table_filter": ["cpu", "mem", "disk", "network"], "import_direction": "newest_first", "target_batch_size": 5000 }' ``` ### Expected results * Imports only `cpu`, `mem`, `disk`, and `network` measurements * Processes data from January 1, 2024 to December 31, 2024 * Imports newest data first * Uses larger batch size (5000 rows) for better performance ### Example 3: Pause, check status, and resume import Monitor and control a long-running import: ```bash # Start import (logs import_id, does not return it immediately) curl -X POST http://localhost:8181/api/v3/engine/import?action=start \ -H "Source-Token: my-token" \ -H "Content-Type: application/json" \ -d '{ "source_url": "http://localhost:8086", "influxdb_version": 2, "source_database": "large_database", "dest_database": "imported" }' # Find import_id from logs: influxdb3 query --database _internal "SELECT log_text FROM system.processing_engine_logs WHERE trigger_name = 'import_trigger' AND log_text LIKE '%Starting import%' ORDER BY event_time DESC LIMIT 1" # Set the import_id from logs IMPORT_ID="" # Pause import (for example, during high-traffic hours) curl -X POST "http://localhost:8181/api/v3/engine/import?action=pause&import_id=$IMPORT_ID" # Check status after import completion (paused, cancelled, or completed) curl "http://localhost:8181/api/v3/engine/import?action=status&import_id=$IMPORT_ID" # Resume later curl -X POST "http://localhost:8181/api/v3/engine/import?action=resume&import_id=$IMPORT_ID" \ -H "Source-Token: my-token" ``` ### Expected results * Import starts and logs a unique import\_id (check logs to obtain it) * Import continues running in the background, logging progress * Pause command stops import gracefully at current position * Status endpoint returns comprehensive statistics **only after import completion** (paused, cancelled, or finished) * Resume command continues from the exact point where it was paused and returns final results upon completion ### Example 4: Dry run for import plan ```bash curl -X POST http://localhost:8181/api/v3/engine/import?action=start \ -H "Source-Token: my-token" \ -H "Content-Type: application/json" \ -d '{ "source_url": "http://localhost:8086", "influxdb_version": 1, "source_database": "telegraf", "dry_run": true }' ``` ### Expected results With `dry_run: true`, the plugin generates a comprehensive import plan **without processing any data**. It only performs: * Schema inspection (tags and fields) * Data sampling for time estimation * Conflict detection The response returns immediately with a detailed import plan: ```json { "import_id": "abc123...", "status": "dry_run_plan", "source": { "url": "http://localhost:8086", "database": "telegraf", "influxdb_version": 1 }, "destination": { "database": "imported_data" }, "time_range": { "start": "all data", "end": "all data" }, "import_settings": { "direction": "oldest_first", "target_batch_size": 2000, "query_interval_ms": 100 }, "tables": { "total": 5, "list": [ "cpu", "mem", "disk", "network", "processes" ], "filtered": "all tables" }, "estimated_import": { "total_rows": 5000000, "estimated_duration": "1 hour 15 minutes", "estimated_duration_seconds": 4500, "per_table_estimates": [ { "measurement": "cpu", "estimated_rows": 1000000, "estimated_seconds": 900, "rows_per_second": 0.38 }, { "measurement": "mem", "estimated_rows": 800000, "estimated_seconds": 720, "rows_per_second": 0.31 } ] }, "schema_conflicts": { "total": 2, "details": [ { "measurement": "cpu", "type": "tag_field_conflict", "conflicts": [ "host", "region" ], "resolution": "Tags will be renamed with '_tag' suffix: host -> host_tag, region -> region_tag" } ] } } ``` **Note**: Dry run mode is lightweight: it reads no rows, and its row counts are estimates from a few sample counts of five days or less per table (see[Automatic batch sizing](#automatic-batch-sizing)). Use it to: * Preview import scope and estimates * Identify schema conflicts before import * Validate configuration and connectivity * Plan import time windows ## Using TOML Configuration Files This plugin supports using TOML configuration files to specify all plugin arguments. ### Setting Up TOML Configuration 1. **Start InfluxDB 3 Enterprise with the Processing Engine**: ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. **Copy the example TOML configuration file to your plugin directory**: ``` cp import_config.toml ~/.plugins/ ``` 3. **Edit the TOML file** to match your requirements: ``` # Required parameters source_url = "http://localhost:8086" influxdb_version = 1 source_database = "telegraf" # Optional parameters dest_database = "imported_data" start_timestamp = "2024-01-01T00:00:00Z" end_timestamp = "2024-12-31T23:59:59Z" table_filter = ["cpu", "mem", "disk"] ``` > > > **Note**: Credentials are NOT stored in TOML files. Provide them via HTTP headers on each request. > > 1. **Create a trigger using the `config_file_path` argument**: ``` influxdb3 create trigger \ --database mydb \ --plugin-filename import.py \ --trigger-spec "request:import" \ --trigger-arguments config_file_path=import_config.toml \ --run-asynchronous \ import_trigger ``` 2. **Start import via HTTP** (config from TOML file will be used as defaults, can be overridden in request body): ``` curl -X POST http://localhost:8181/api/v3/engine/import?action=start ``` ## Configuration Priority and Loading The import plugin loads configuration from multiple sources with the following priority order (highest to lowest): 1. **Query Parameters** (highest priority) - Parameters in the query string of the `action=start` request 2. **Request Headers** - Parameters spelled `X-Influxdb3-Import-` 3. **HTTP Request Body** - JSON parameters in POST request body 4. **TOML Configuration File** - Parameters from file specified in `config_file_path` 5. **Trigger Arguments** - Parameters from `--trigger-arguments` when creating trigger 6. **Environment Variables** (lowest priority) - System environment variables Each source overrides only the keys it sets, so a TOML file may hold the stable settings while the request supplies the time range for one import. The`config_file_path` comes from the trigger arguments or from`INFLUXDB3_IMPORT_CONFIG_FILE_PATH`, the argument winning; the request may not point at a configuration file, and neither may the file itself. Only `action=start` assembles its configuration from these layers. The other actions read less: * `action=resume` reads no settings from the request at all. It continues with the configuration its import was started with, kept in the `import_config`table. * `action=test_connection`, `action=databases` and `action=tables` take their source parameters from the JSON request body only — never from the environment, the trigger arguments or the TOML file. Each of these actions is documented above with the keys it needs. `databases` and `tables` also read the credential headers; `test_connection` only probes the URL and sends none. ### Configuration Loading Process When a import starts, the plugin loads configuration in this order: ```python # 1. Start with environment variables (lowest priority) IMPORT_SOURCE_URL and its four companions, then INFLUXDB3_IMPORT_ * on top # 2. Override with trigger arguments (--trigger-arguments) config_file_path=import_config.toml, source_url=http://localhost:8086, etc. # 3. Override with TOML file contents (if config_file_path specified) [from import_config.toml file] # 4. Override with the HTTP request body { "source_url": "http://localhost:8086", ... } # 5. Override with request headers X-Influxdb3-Import-Source-Url: http://localhost:8086 # 6. Override with query parameters (highest priority) ?action=start&source_url=http://localhost:8086 ``` ### Request Headers Supported Every parameter except `config_file_path` can come from a header named`X-Influxdb3-Import-` with underscores written as hyphens — for example, `X-Influxdb3-Import-Source-Url` sets `source_url` and`X-Influxdb3-Import-Target-Batch-Size` sets `target_batch_size`. Casing does not matter, as RFC 9110 asks. A header the plugin does not ask for is ignored, so the request still carries whatever else the client sends. Credentials are **not** among these. They keep their own unprefixed headers, and`X-Influxdb3-Import-Source-Token` is ignored rather than read as a token: ```bash -H "Source-Token: my-super-secret-token" -H "Source-Username: admin" -H "Source-Password: my-password" ``` A header value must be ASCII. InfluxDB 3 Enterprise closes the connection without a reply when one is not, and the plugin is never invoked, so a `table_filter` naming a measurement outside ASCII belongs in the request body or the query string. ### Query Parameters Supported Every parameter except `config_file_path` can come from the query string of an`action=start` request, spelled exactly as the parameter is. Unlike a header, a query parameter is matched case-sensitively, so `Source_Url` is refused: ```bash curl -X POST "http://localhost:8181/api/v3/engine/import?action=start&source_url=http://localhost:8086&source_database=telegraf&influxdb_version=1" ``` Each action accepts only the parameters it reads, and an unknown one is refused and named along with what that action accepts — `?action=status&import_ids=abc`answers `Query parameters may not set 'import_ids'; accepted keys: ['action', 'import_id']`. Three cautions specific to the query string: * A `+` in a timestamp is decoded as a space, so write the offset as `Z` or percent-encode it: `start_timestamp=2024-01-01T00%3A00%3A00%2B03%3A00`. * An empty value is ignored rather than treated as a value, so `?table_filter=`does not clear a filter set by the TOML file. * A value outside ASCII is carried correctly when percent-encoded as UTF-8, which makes the query string the way to name such a measurement. ### Environment Variables Supported Every parameter can come from an environment variable named`INFLUXDB3_IMPORT_` in upper case — for example,`INFLUXDB3_IMPORT_SOURCE_URL` sets `source_url` and`INFLUXDB3_IMPORT_TARGET_BATCH_SIZE` sets `target_batch_size`.`INFLUXDB3_IMPORT_CONFIG_FILE_PATH` names the TOML file when the trigger doesn’t carry a `config_file_path` argument. Five of the settings are read from a second name as well: * `IMPORT_SOURCE_URL` → `source_url` * `IMPORT_SOURCE_DATABASE` → `source_database` * `IMPORT_DEST_DATABASE` → `dest_database` * `IMPORT_START_TIMESTAMP` → `start_timestamp` * `IMPORT_END_TIMESTAMP` → `end_timestamp` When a setting is given under both spellings, `INFLUXDB3_IMPORT_*` wins. ## Data Type Mismatch Handling The plugin automatically handles data type mismatches that can occur in older InfluxDB versions where different nodes might have different field types for the same field name. ### How It Works 1. **Schema Detection**: At import start, plugin queries source database for field types using `SHOW FIELD KEYS` 2. **Runtime Type Checking**: For each data point, plugin checks if the actual value type matches the expected field type 3. **Automatic Field Creation**: If type mismatch is detected, plugin creates a new field with a type suffix ### Supported Type Suffixes When type mismatches occur, the plugin appends these suffixes: * `_string` - for string values * `_integer` - for integer values * `_float` - for float values * `_boolean` - for boolean values ## Code overview ### Files * `import.py`: The main plugin code containing HTTP request handler and import logic * `import_config.toml`: Example TOML configuration file ### Logging Logs are stored in the `_internal` database in the `system.processing_engine_logs` table: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'import_trigger'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ### Import state tracking The plugin creates several measurements to track import state: #### `import_config` Stores import configuration (credentials excluded for security). ```bash influxdb3 query --database mydb "SELECT * FROM import_config WHERE import_id = 'your-import-id'" ``` #### `import_state` Tracks per-table import progress. One row per chunk written, holding the table’s status, its running row count and the timestamp of the last row written, to the nanosecond. That checkpoint is what a resume continues from, so an import continues from its last written chunk whether it was paused or interrupted. The row count, the checkpoint and the failures all carry across a resume, so the latest row of a table always holds its totals. Every row also carries an `errors` column, holding the chunks the table failed to write, as JSON (the key is still named `failed_windows`): ```json { "failed_windows": 240, "errors": [ { "time_range": "2026-01-01 00:00:00+00:00 to 2026-01-01 02:00:00+00:00", "error": "invalid column type for column 'room', expected iox::column_type::field::integer, got iox::column_type::tag" } ] } ``` A table that failed nothing records `{"failed_windows": 0, "errors": []}`.`failed_windows` counts every chunk that failed, while `errors` holds only the first 50 of them — a table that fails on every chunk repeats one reason, and the whole list would outgrow a single point. Rows written while the import is still running sample 3 of them instead of 50, to keep a row per chunk small;`failed_windows` is exact on every row. Rows that cannot be converted to line protocol count their chunk as failed too, so they are recorded rather than dropped silently. A space in a measurement name is written as an underscore (and noted under schema issues); a table ends with status `failed` when it wrote no rows while windows failed, or when its sanitized name collides with another table’s (only the first of the two imports). `action=status` returns the same object for each table under`table_details[].errors`, and counts the tables that failed anything as`summary.tables_with_errors`. This is where to look when an import reports`completed` with fewer rows than expected: the status answers long after the entries in `system.processing_engine_logs` have been trimmed away. ```bash influxdb3 query --database mydb "SELECT * FROM import_state WHERE import_id = 'your-import-id' ORDER BY time DESC" ``` #### `import_pause_state` Stores pause/cancel/completed state for controlling running imports. ```bash influxdb3 query --database mydb "SELECT * FROM import_pause_state WHERE import_id = 'your-import-id' ORDER BY time DESC LIMIT 1" ``` ### Main functions #### `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)` HTTP request handler that routes to appropriate import actions based on the `action` query parameter. Each action accepts only the query parameters it reads, so an unknown one is refused and named alongside the accepted keys. Reads the credentials from the `Source-Token`, `Source-Username` and `Source-Password` headers and passes them to the action handlers; a header that was not sent is absent from that dict. #### `load_import_settings(influxdb3_local, task_id, args, request_body, request_headers, query_settings)` Assembles the import configuration from environment variables, trigger arguments, the TOML file, the request body, the`X-Influxdb3-Import-*` headers and the query parameters, in that order of precedence, then validates it: required parameters must be present, numeric and boolean parameters are coerced, and `influxdb_version` and `import_direction`must be one of their supported values. The TOML file path comes from the trigger arguments or from`INFLUXDB3_IMPORT_CONFIG_FILE_PATH`. Raises on the first problem, naming the parameter. #### `start_import(influxdb3_local, config, credentials, task_id)` Starts a new import process: 1. Performs pre-flight checks (connectivity, measurements discovery) 2. Estimates import time based on data sampling 3. Creates import configuration and state records 4. Initiates table-by-table import 5. On any unhandled error, writes paused state so the import can be resumed after fixing the issue #### `import_table(influxdb3_local, config, credentials, import_id, measurement, start_time, end_time, task_id, ...)` Imports a single table: 1. Finds actual data boundaries within specified range 2. Samples data to determine optimal batch window size 3. Detects and resolves tag/field conflicts 4. Queries each window as a chunked answer and reads it a chunk at a time 5. Converts each chunk to line protocol and writes it before reading the next 6. Tracks progress per chunk, splits windows the source cannot answer, and checks for pause/cancel signals #### `resume_import(influxdb3_local, import_id, credentials, task_id)` Resumes an interrupted import: 1. Detects stale imports — if the import state is “running” but the last update is older than 5 minutes, treats it as crashed and allows resume 2. If no `import_state` records exist (crashed before processing any tables), restarts from the beginning 3. Loads saved import configuration 4. Identifies incomplete tables and their last checkpoint 5. Continues import from checkpoint positions 6. Stops on the first table the user pauses or cancels, leaving the import resumable 7. On any unhandled error, writes paused state so the import can be resumed again #### `get_import_stats(influxdb3_local, import_id, task_id)` Returns comprehensive statistics for a import including overall status, per-table progress, timing information, and configuration. Each entry of `table_details` carries the table’s `errors` object, and `summary.tables_with_errors`counts the tables that failed to write anything. #### `check_source_connection(body_data, session)` Tests connectivity to a URL and identifies if it’s an InfluxDB instance (5-second timeout): 1. Validates that `source_url` is provided 2. Infers port from scheme if not specified (http→80, https→443) 3. Sends request to `/ping` endpoint 4. Returns success with version/build from `X-Influxdb-*` headers (v1/v2) 5. Falls back to `cluster-uuid` header detection for v3 (returns `version: "3.x.x"`) 6. Returns failure with message if not InfluxDB or unreachable #### `get_source_databases_list(body_data, credentials, session)` Lists databases from source InfluxDB instance: 1. Validates required parameters 2. For v1: Executes `SHOW DATABASES` through `/query`, filters out `_internal` 3. For v2: the same query, filtering out the system buckets, which InfluxDB 2 marks with a leading underscore 4. For v3: Queries `/api/v3/configure/database` 5. Returns sorted list of database names #### `get_source_tables_list(body_data, credentials, session)` Lists tables/measurements from a source database: 1. Validates required parameters including `source_database` 2. For v1 and v2: Executes `SHOW MEASUREMENTS` through `/query` 3. For v3: Executes `SHOW TABLES` through `/api/v3/query_sql` 4. Returns sorted list of table names ### Key algorithms #### Automatic batch sizing The plugin samples data at different time intervals to determine optimal window size: ```python # Test intervals: 1 hour, 10 hours, 1 day, 5 days # Calculate rows per second from samples # Determine window size to achieve target_batch_size optimal_window = target_batch_size / avg_rows_per_second ``` The same rate gives the row counts of the import estimate (`rows_per_second`times the table’s time range), and the import reuses it rather than sampling again. The plugin never counts a whole measurement: on a large measurement with many fields, a `COUNT(*)` over its full range makes the source read every point of every field, which can exhaust the source’s memory. A range shorter than one hour is counted whole, since that is no larger than a sample. #### Streaming windows in chunks The sampled window size is an estimate from a few counts, and data is rarely even: a window sized for a quiet month can hold millions of rows from a busy week. So the plugin never reads a window whole: * Each window is requested with `chunked=true&chunk_size=`, which InfluxDB v1, v2 and v3 all support on `/query`, and the answer is read as a stream. * Each chunk is converted and written before the next one is read, and the checkpoint moves with every chunk. Memory follows `target_batch_size`, not the size of the window. * Rows at the instant of a chunk’s last row can continue in the next chunk, so they are held back and written with it. A checkpoint therefore never names an instant with rows still unwritten. * An answer that breaks off, or ends while still marked partial, is never taken for a finished window. A window that fails is retried as follows: * **The answer broke off after some chunks were written**: the plugin continues from the last written row. * **No answer within 30 seconds, an answer that broke off before any row, a statement error, or one answer document over 64 MiB** (what a source that ignores chunking sends): the plugin retries the window at half its size, down to one second. The size grows back by doubling once windows succeed again. * **The source cannot be reached**: the plugin retries with backoff (1s → 2s → 4s → 8s). When a one-second window still fails, the table is paused at its checkpoint, ready for `action=resume`. #### Tag/field conflict resolution When a column name exists as both tag and field in source data: ```python # Original data has conflict: # tag: room # field: room # Plugin renames conflicting tag: # tag: room_tag # field: room (unchanged) ``` #### Resume checkpoint tracking During import, the plugin saves checkpoints: ```python # Save paused_at_time after every chunk written # (data timestamp of the last row written, not record timestamp) # On resume: # 1. Load last paused_at_time # 2. Add 1 nanosecond to avoid duplicates # 3. Continue import from (paused_at_time + 1ns) ``` ## Troubleshooting ### Common issues #### Issue: “Source query failed” and the import stops **Cause**: The source accepted the request and answered with a failed statement — a point limit, a series limit, or a query it could not plan. InfluxDB reports these with HTTP 200 and the reason inside the body, so the plugin raises on them rather than reading an empty answer as “no data”. **Solution**: the plugin first retries the window at half its size, down to one second, since a source may refuse a window for its size. If even that fails, the reason is in the message and in `system.processing_engine_logs`. The table is left `paused` with its checkpoint, so `action=resume` continues it once the source is able to answer. #### Issue: “Failed to connect to source database” error **Solution**: 1. Verify source InfluxDB is running and accessible: ``` curl http:///ping ``` 2. Check network connectivity and firewall rules 3. Verify credentials are correct 4. For InfluxDB v2/v3, ensure you’re using token authentication #### Issue: Authentication errors from source InfluxDB **Solution**: Pass credentials via HTTP headers: 1. For token auth: Add header `-H "Source-Token: your-token"` 2. For username/password: Add headers `-H "Source-Username: user" -H "Source-Password: pass"` 3. Verify credentials work directly against source InfluxDB 4. Check that headers are not being stripped by proxies #### Issue: `pause`, `status` or `cancel` does not answer while an import runs **Solution**: The trigger was created without `--run-asynchronous`, so the Processing Engine serves one invocation at a time and the control request waits for the import to finish. Recreate the trigger with the flag: ```bash influxdb3 delete trigger --database mydb --force import_trigger influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/import/import.py \ --trigger-spec "request:import" \ --run-asynchronous \ import_trigger ``` #### Issue: “Import is already running” after server crash **Solution**: 1. Wait at least 5 minutes after the crash — the plugin uses a 5-minute stale import threshold 2. Call the resume endpoint with credentials: ``` curl -X POST "http://localhost:8181/api/v3/engine/import?action=resume&import_id=$IMPORT_ID" \ -H "Source-Token: my-token" ``` 3. The plugin detects the stale state and resumes from the last checkpoint (or restarts from the beginning if no checkpoint exists) #### Issue: “Import already completed” when trying to resume **Solution**: 1. Check import status: ``` curl "http://localhost:8181/api/v3/import?action=status&import_id=" ``` 2. If truly incomplete, check for status discrepancies in `import_state` table 3. Start a new import if needed #### Issue: Tag/field conflicts causing warnings **Solution**: This is informational only. The plugin automatically renames conflicting tags with a `_tag` suffix: * Original tag `temperature` → `temperature_tag` * Field `temperature` remains unchanged #### Issue: InfluxDB 3 Enterprise runs out of memory during an import **Cause**: Versions before 0.5.0 read each query window whole, and built every row of it in memory inside the InfluxDB 3 Enterprise server before writing. On uneven data a single window could hold millions of rows, which took the server down. **Solution**: update the plugin. It now reads each window as a stream and holds no more than one chunk of `target_batch_size` rows at a time. If memory is still tight, lower `target_batch_size`. #### Issue: Slow import performance **Solution**: 1. Increase `target_batch_size` (for example, from 2000 to 5000) 2. Decrease `query_interval_ms` if source can handle higher load 3. Use table filtering to import tables in parallel using multiple triggers 4. Check network latency between source and destination ### Performance considerations * **Network bandwidth**: Main bottleneck for large imports. Use local network when possible. * **Source database load**: The plugin includes rate limiting (`query_interval_ms`) to avoid overwhelming source. * **Batch size optimization**: Plugin automatically samples data to determine optimal batch size, but you can override with `target_batch_size`. * **Connection pooling**: Plugin uses HTTP session with connection pooling for better performance. * **Retry logic**: Built-in exponential backoff (1s → 2s → 4s → 8s → 16s) for transient errors. * **Memory**: Bounded by `target_batch_size` rows per chunk, whatever the window size. A larger value means fewer, larger writes and more memory per chunk. ## Import best practices 1. **Use table filtering**: Import critical tables first, then others in batches 2. **Plan for pauses**: Pause during high-traffic hours if sharing infrastructure 3. **Verify data**: Compare row counts and sample data after import 4. **Handle conflicts**: Review log warnings about tag/field conflicts ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Import plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/import) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | source_url | string | required | Source InfluxDB URL (with optional port, for example, http://localhost:8086 ) | | influxdb_version | integer | required | Source InfluxDB version: 1, 2, or 3 | | source_database | string | required | Source database name to import from | | Header | Purpose | | --- | --- | | Header | Purpose | | Source-Token | Bearer/API token authentication (InfluxDB v1/v2/v3) | | Source-Username | Basic auth username (InfluxDB v1) | | Source-Password | Basic auth password (InfluxDB v1) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | dest_database | string | none | Destination database name in InfluxDB 3 Enterprise (if not specified, uses database where trigger was created) | | start_timestamp | string | none | Import start time — RFC3339, Unix seconds or nanoseconds, or YYYY-MM-DD ; a value naming no zone is read as UTC. If not specified, starts from oldest data | | end_timestamp | string | none | Import end time, same formats as start_timestamp . If not specified, imports to newest data | | query_interval_ms | integer | 100 | Delay between queries in milliseconds to avoid overloading source database (0 or greater) | | import_direction | string | “oldest_first” | Import direction: “oldest_first” or “newest_first” | | target_batch_size | integer | 2000 | Rows per chunk read from the source and per write, and the target rows per query window (1 or greater) | | table_filter | string | array | none | Tables to import: a JSON array (request body or TOML), e.g. ["cpu", "mem", "disk"] , which keeps a name containing a dot intact; or a dot-separated string elsewhere, e.g. cpu.mem.disk . A space in a name is written as an underscore (noted under schema issues); if two names sanitize to the same one, only the first imports and the rest are reported failed . If not specified, imports all tables | | dry_run | boolean | false | If true, generates import plan without processing data (shows estimates, schema conflicts, and configuration) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path, absolute or relative to the plugin directory. Trigger arguments or INFLUXDB3_IMPORT_CONFIG_FILE_PATH | --- # InfluxDB to Iceberg plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/influxdb-to-iceberg/ The InfluxDB to Iceberg Plugin enables data transfer from InfluxDB 3 Enterprise to Apache Iceberg tables. Transfer time series data to Iceberg for long-term storage, analytics, or integration with data lake architectures. The plugin supports both scheduled batch transfers of historical data and on-demand transfers via HTTP API. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Scheduler trigger parameters #### Required parameters | Parameter | Type |Default | Description | |-----------------|------|--------|----------------------------------------------------------------------------------| | `measurement` |string|required| Source measurement containing data to transfer | | `window` |string|required|Time window for data transfer. Format: `` (for example, “1h”, “30d”)| |`catalog_configs`|string|required| Base64-encoded JSON string containing Iceberg catalog configuration | #### Optional parameters | Parameter | Type | Default | Description | |--------------------|------|----------------|----------------------------------------------------------------------------------| | `included_fields` |string|all fields/tags |Dot-separated list of fields and tags to include (for example, “usage\_user.host”)| | `excluded_fields` |string| none | Dot-separated list of fields and tags to exclude | | `namespace` |string| “default” | Iceberg namespace for the target table | | `table_name` |string|measurement name| Iceberg table name | |`auto_update_schema`|string| false |Automatically update Iceberg table schema when data doesn’t match existing schema | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. #### Example TOML configuration [influxdb\_to\_iceberg\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/influxdb_to_iceberg/influxdb_to_iceberg_config_scheduler.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ### HTTP trigger parameters #### Request body structure | Parameter | Type |Required| Description | |--------------------|-------|--------|-----------------------------------------------------------------------------------------------------------------------------| | `measurement` |string | Yes | Source measurement containing data to transfer | | `catalog_configs` |object | Yes |Iceberg catalog configuration dictionary. See [PyIceberg catalog documentation](https://py.iceberg.apache.org/configuration/)| | `included_fields` | array | No | List of field and tag names to include in replication | | `excluded_fields` | array | No | List of field and tag names to exclude from replication | | `namespace` |string | No | Target Iceberg namespace (default: “default”) | | `table_name` |string | No | Target Iceberg table name (default: measurement name) | | `batch_size` |string | No | Batch size duration for processing (default: “1d”). Format: `` | | `backfill_start` |string | No | ISO 8601 datetime with timezone for backfill start | | `backfill_end` |string | No | ISO 8601 datetime with timezone for backfill end | |`auto_update_schema`|boolean| No | Automatically update Iceberg table schema when data doesn’t match existing schema (default: false) | ## Schema management * Automatically creates Iceberg table schema from the first batch of data * Maps Pandas data types to Iceberg types: * `int64` → `IntegerType` * `float64` → `FloatType` * `datetime64[us]` → `TimestampType` * `object` → `StringType` * Fields with no null values are marked as `required` * The `time` column is converted to `datetime64[us]` for Iceberg compatibility * Tables are created in format: `.` ### Automatic schema updates When `auto_update_schema=true`: * **New fields**: Automatically added to Iceberg table schema as optional (nullable) columns * **Missing fields**: Added to DataFrame with null values based on existing schema types * **Schema evolution**: Ensures data compatibility between InfluxDB and Iceberg without manual intervention * **Backward compatibility**: Existing data remains valid as new columns are always optional ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: * `pandas` (for data manipulation) * `pyarrow` (for Parquet support) * `pyiceberg[catalog-options]` (for Iceberg integration) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package pandas influxdb3 install package pyarrow influxdb3 install package "pyiceberg[s3fs,hive,sql-sqlite]" ``` **Note:** Include the appropriate PyIceberg extras based on your catalog type: * `[s3fs]` for S3 storage * `[hive]` for Hive metastore * `[sql-sqlite]` for SQL catalog with SQLite * See [PyIceberg documentation](https://py.iceberg.apache.org/#installation) for all options ## Schema requirement The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. ## Trigger setup ### Scheduled data transfer Periodically transfer data from InfluxDB 3 Enterprise to Iceberg: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/influxdb_to_iceberg/influxdb_to_iceberg.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'measurement=cpu,window=1h,catalog_configs="eyJ1cmkiOiAiaHR0cDovL25lc3NpZTo5MDAwIn0=",namespace=monitoring,table_name=cpu_metrics' \ hourly_iceberg_transfer ``` ### HTTP API endpoint Create an on-demand transfer endpoint: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/influxdb_to_iceberg/influxdb_to_iceberg.py" \ --trigger-spec "request:replicate" \ iceberg_http_transfer ``` Enable the trigger: ```bash influxdb3 enable trigger --database mydb iceberg_http_transfer ``` The endpoint is registered at `/api/v3/engine/replicate`. ## Example usage ### Example 1: Basic scheduled transfer Transfer CPU metrics to Iceberg every hour: ```bash # Create trigger with base64-encoded catalog config # Original JSON: {"uri": "http://nessie:9000"} # Base64: eyJ1cmkiOiAiaHR0cDovL25lc3NpZTo5MDAwIn0= influxdb3 create trigger \ --database metrics \ --path "gh:influxdata/influxdb_to_iceberg/influxdb_to_iceberg.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'measurement=cpu,window=24h,catalog_configs="eyJ1cmkiOiAiaHR0cDovL25lc3NpZTo5MDAwIn0="' \ cpu_to_iceberg # Write test data influxdb3 write \ --database metrics \ "cpu,host=server1 usage_user=45.2,usage_system=12.1" # After trigger runs, data is available in Iceberg table "default.cpu" ``` **Expected output** * Creates Iceberg table `default.cpu` with schema matching the measurement * Transfers all CPU data from the last 24 hours * Appends new data on each hourly run ### Example 2: HTTP backfill with field filtering Backfill specific fields from historical data: ```bash # Create and enable HTTP trigger influxdb3 create trigger \ --database metrics \ --path "gh:influxdata/influxdb_to_iceberg/influxdb_to_iceberg.py" \ --trigger-spec "request:replicate" \ iceberg_backfill influxdb3 enable trigger --database metrics iceberg_backfill # Request backfill via HTTP curl -X POST http://localhost:8181/api/v3/engine/replicate \ -H "Authorization: Bearer YOUR_TOKEN" \ -d '{ "measurement": "temperature", "catalog_configs": { "type": "sql", "uri": "sqlite:///path/to/catalog.db" }, "included_fields": ["temp_celsius", "humidity", "sensor_id"], "namespace": "weather", "table_name": "temperature_history", "batch_size": "12h", "backfill_start": "2024-01-01T00:00:00+00:00", "backfill_end": "2024-01-07T00:00:00+00:00" }' ``` **Expected output** * Creates Iceberg table `weather.temperature_history` * Transfers only `temp_celsius` and `humidity` fields * Processes data in 12-hour batches for the specified week * Returns status of the backfill operation ### Example 3: S3-backed Iceberg catalog Transfer data to Iceberg tables stored in S3: ```bash # Create catalog config JSON cat > catalog_config.json << EOF { "type": "sql", "uri": "sqlite:///iceberg/catalog.db", "warehouse": "s3://my-bucket/iceberg-warehouse/", "s3.endpoint": "http://minio:9000", "s3.access-key-id": "minioadmin", "s3.secret-access-key": "minioadmin", "s3.path-style-access": true } EOF # Encode to base64 CATALOG_CONFIG=$(base64 < catalog_config.json) # Create trigger influxdb3 create trigger \ --database metrics \ --path "gh:influxdata/influxdb_to_iceberg/influxdb_to_iceberg.py" \ --trigger-spec "every:30m" \ --trigger-arguments "measurement=sensor_data,window=1h,catalog_configs=\"$CATALOG_CONFIG\",namespace=iot,table_name=sensors" \ s3_iceberg_transfer ``` ## Code overview ### Files * `influxdb_to_iceberg.py`: The main plugin code containing handlers for scheduled and HTTP triggers * `influxdb_to_iceberg_config_scheduler.toml`: Example TOML configuration file for scheduled triggers ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled data transfers. Queries data within the specified window and appends to Iceberg tables. Key operations: 1. Parses configuration and decodes catalog settings 2. Queries source measurement with optional field filtering 3. Creates Iceberg table if needed 4. Appends data to Iceberg table #### `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)` Handles on-demand data transfers via HTTP. Supports backfill operations with configurable batch sizes. Key operations: 1. Validates request body parameters 2. Determines backfill time range 3. Processes data in batches 4. Returns transfer status ## Troubleshooting ### Common issues #### Issue: “Failed to decode catalog\_configs” error **Solution**: Ensure the catalog configuration is properly base64-encoded: ```bash # Create JSON file echo '{"uri": "http://nessie:9000"}' > config.json # Encode to base64 base64 config.json ``` #### Issue: “Failed to create Iceberg table” error **Solution**: 1. Verify catalog configuration is correct 2. Check warehouse path permissions 3. Ensure required PyIceberg extras are installed:`bash influxdb3 install package "pyiceberg[s3fs]"` #### Issue: No data in Iceberg table after transfer **Solution**: 1. Check if source measurement contains data:`bash influxdb3 query --database mydb "SELECT COUNT(*) FROM measurement"` 2. Verify time window covers data:`bash influxdb3 query --database mydb "SELECT MIN(time), MAX(time) FROM measurement"` 3. Check logs for errors:`bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_level = 'ERROR'"` #### Issue: “Incompatible change: cannot add required column” error **Solution**: This occurs when trying to add a required (non-nullable) column to an existing table. With `auto_update_schema=true`, new columns are automatically added as optional. If you encounter this error: 1. Ensure `auto_update_schema=true` in your configuration 2. Check that you’re using the latest version of the plugin ### Debugging tips 1. **Test catalog connectivity**: ```python from pyiceberg.catalog import load_catalog catalog = load_catalog("my_catalog", **catalog_configs) print(catalog.list_namespaces()) ``` 1. **Verify field names**: ``` influxdb3 query --database mydb "SHOW FIELD KEYS FROM measurement" ``` 2. **Use smaller windows** for initial testing: ``` --trigger-arguments 'window=5m,...' ``` ### Performance considerations * **File sizing**: Each scheduled run creates new Parquet files. Use appropriate window sizes to balance file count and size * **Batch processing**: For HTTP transfers, adjust `batch_size` based on available memory * **Field and tag filtering**: Use `included_fields` to reduce data volume when only specific fields and tags are needed * **Catalog choice**: SQL catalogs (SQLite) are simpler but REST catalogs scale better ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [InfluxDB to Iceberg plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/influxdb_to_iceberg) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[iceberg](/influxdb3/enterprise/tags/iceberg/)[export](/influxdb3/enterprise/tags/export/)[data-lake](/influxdb3/enterprise/tags/data-lake/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Source measurement containing data to transfer | | window | string | required | Time window for data transfer. Format: (for example, “1h”, “30d”) | | catalog_configs | string | required | Base64-encoded JSON string containing Iceberg catalog configuration | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | included_fields | string | all fields/tags | Dot-separated list of fields and tags to include (for example, “usage_user.host”) | | excluded_fields | string | none | Dot-separated list of fields and tags to exclude | | namespace | string | “default” | Iceberg namespace for the target table | | table_name | string | measurement name | Iceberg table name | | auto_update_schema | string | false | Automatically update Iceberg table schema when data doesn’t match existing schema | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | | Parameter | Type | Required | Description | | --- | --- | --- | --- | | Parameter | Type | Required | Description | | measurement | string | Yes | Source measurement containing data to transfer | | catalog_configs | object | Yes | Iceberg catalog configuration dictionary. See PyIceberg catalog documentation | | included_fields | array | No | List of field and tag names to include in replication | | excluded_fields | array | No | List of field and tag names to exclude from replication | | namespace | string | No | Target Iceberg namespace (default: “default”) | | table_name | string | No | Target Iceberg table name (default: measurement name) | | batch_size | string | No | Batch size duration for processing (default: “1d”). Format: | | backfill_start | string | No | ISO 8601 datetime with timezone for backfill start | | backfill_end | string | No | ISO 8601 datetime with timezone for backfill end | | auto_update_schema | boolean | No | Automatically update Iceberg table schema when data doesn’t match existing schema (default: false) | --- # Kafka subscriber plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/kafka-subscriber/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The Kafka Subscriber Plugin enables real-time ingestion of Kafka messages into InfluxDB 3 Enterprise. Subscribe to Kafka topics and automatically transform messages into time-series data with support for JSON, Line Protocol, and custom text formats. The plugin uses consumer groups for reliable message delivery and provides flexible offset commit policies for different use cases. The consumer is reused across scheduled invocations to avoid a consumer-group rebalance every cycle, and it optionally supports a dead-letter queue topic for failed messages and id-based deduplication for messages without their own timestamp. **Binary formats:** `avro`, `jsonschema`, and `protobuf` (Confluent wire format) are decoded via Confluent Schema Registry. `avro` can also be decoded as raw schemaless Avro with a local `.avsc` schema, and `protobuf` with a local `.proto` schema. See [Binary formats](#binary-formats). ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. This plugin supports TOML configuration files for complex mapping scenarios, which can be specified using the `config_file_path` parameter. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters In TOML configuration, `bootstrap_servers`, `topics`, and `group_id` are placed under the `[kafka]` section; `table_name` and `table_name_field` are placed under `[mapping.json]` or `[mapping.text]` section. | Parameter | Type | Default | Description | |-------------------|------|-----------|-----------------------------------------------------------------------------------------------------------------------------| |`bootstrap_servers`|string| required | Space-separated list of Kafka broker addresses (for example, “kafka1:9092 kafka2:9092”) | | `topics` |string| required | Space-separated list of topics (for example, “sensor\_data metrics”) | | `group_id` |string| required | Kafka consumer group ID (must be unique per consumer group) | | `table_name` |string|conditional|InfluxDB measurement name for storing data. Required for all formats except `lineprotocol`, unless `table_name_field` is set.| |`table_name_field` |string| none | JSON field name or regex pattern to extract table name dynamically from each message. Alternative to static `table_name`. | ### Connection parameters In TOML configuration, these parameters are placed under the `[kafka]` section. | Parameter | Type | Default | Description | |-------------------|------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`auto_offset_reset`|string|“earliest”| Where to start consuming on first connect: “earliest” or “latest” | |`max_poll_records` | int | 500 | Maximum messages per scheduled call. Set to 0 for unlimited. | |`max_poll_interval`| int | 300 |Maximum seconds between scheduled poll cycles before the broker considers the consumer dead (maps to `max.poll.interval.ms`). The consumer is reused across invocations, so it must stay in the group between polls — increase this when the trigger interval is longer than \~5 minutes.| ### Offset Commit Policy In TOML configuration, this parameter is placed under the `[kafka]` section. | Parameter | Type | Default | Description | |----------------------|------|-------------|-------------------------------------------------| |`offset_commit_policy`|string|“on\_success”|When to commit offsets: “on\_success” or “always”| **Policy behavior:** | Policy | Behavior | |------------|------------------------------------------------------------------------------------------| |`on_success`|Commit offsets only after ALL messages in the batch are successfully processed and written| | `always` | Commit offsets immediately after receiving messages, regardless of processing success | **When to use each policy:** * **on\_success (recommended)**: Use when data integrity is important. Failed messages will be reprocessed on the next trigger execution. * **always**: Use in high-throughput scenarios where occasional data loss is acceptable, or when you have external error handling (failed messages are logged to `kafka_exceptions` table). **Interaction with `dlq_topic`:** Because the consumer is reused across invocations, its fetch position advances as messages are polled. Under `on_success`, when a message still needs reprocessing — a transient InfluxDB write failure, or a parse failure (poison message) that could **not** be offloaded — the plugin **rewinds the consumer** to the lowest such offset on each affected partition, so those messages are re-read on the next cycle. Only the affected partitions are rewound; partitions whose messages all succeeded are not. A parse failure that is **successfully** published to `dlq_topic` is treated as handled: the consumer is not rewound for it, and it is preserved in the DLQ instead of being re-read. Without `dlq_topic`, a poison message under `on_success` is rewound and reprocessed on every cycle until it is resolved (its original payload is not stored in `kafka_exceptions`, so configure `dlq_topic` if you need to skip past and preserve poison messages). With `always`, the offset is committed immediately regardless and the consumer is never rewound, so a failed message is **not** reprocessed; `dlq_topic` simply preserves the failed messages that would otherwise be dropped. > > > **Note:** Rewinding re-reads from the failed offset, so messages on the same partition that succeeded *after* the failed one are re-read and re-written. Enable [deduplication](#deduplication-parameters) (or use messages carrying their own timestamp) to avoid duplicate points from this at-least-once retry. > > ### Security parameters In TOML configuration, this parameter is placed under the `[kafka]` section. | Parameter | Type | Default | Description | |-------------------|------|-----------|---------------------------------------------------------------------| |`security_protocol`|string|“PLAINTEXT”|Security protocol: “PLAINTEXT”, “SSL”, “SASL\_PLAINTEXT”, “SASL\_SSL”| ### SASL Authentication parameters In TOML configuration, these parameters are placed under the `[kafka.sasl]` section with shortened names. |Parameter (CLI) | Parameter (TOML) | Type |Default| Description | |----------------|--------------------------|------|-------|---------------------------------------------------------| |`sasl_mechanism`|`[kafka.sasl]` `mechanism`|string| none |SASL mechanism: “PLAIN”, “SCRAM-SHA-256”, “SCRAM-SHA-512”| |`sasl_username` |`[kafka.sasl]` `username` |string| none | SASL username (required with sasl\_mechanism) | |`sasl_password` |`[kafka.sasl]` `password` |string| none | SASL password (required with sasl\_mechanism) | **Note:** All three SASL parameters must be provided together when using SASL authentication. ### SSL/TLS parameters In TOML configuration, these parameters are placed under the `[kafka.ssl]` section with shortened names. | Parameter (CLI) | Parameter (TOML) | Type |Default| Description | |------------------|----------------------------|------|-------|-----------------------------------------| | `ssl_ca_cert` | `[kafka.ssl]` `ca_cert` |string| none | Path to CA certificate file | | `ssl_cert` |`[kafka.ssl]` `client_cert` |string| none |Path to client certificate for mutual TLS| | `ssl_key` | `[kafka.ssl]` `client_key` |string| none |Path to client private key for mutual TLS| |`ssl_key_password`|`[kafka.ssl]` `key_password`|string| none |Password for encrypted client private key| **Note:** For mutual TLS, both `ssl_cert` and `ssl_key` must be provided together. ### Logging parameters In TOML configuration, `enable_full_logging` is placed directly under the `[kafka]` section. | Parameter | Type |Default| Description | |---------------------|-------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`enable_full_logging`|boolean| false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths) into log output. Enable temporarily for debugging.| ### Dead-letter queue parameters In TOML configuration, `dlq_topic` is placed under the `[kafka]` section. | Parameter | Type |Default| Description | |-----------|------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`dlq_topic`|string| none |Kafka topic to publish messages that fail to parse. The original payload and key are produced unchanged, with `source_topic`, `error_type` and `error_message` set as Kafka headers. When unset, no DLQ topic is used.| **Behavior:** * Only **parse failures** (poison messages) are routed to the DLQ topic — including messages missing a required `dedup_id_field`. Transient write failures to InfluxDB are **not** sent to the DLQ; they keep their offsets uncommitted (with `on_success`) so they are retried on the next cycle. * Failures continue to be recorded in the `kafka_exceptions` table regardless of `dlq_topic`. * The DLQ producer reuses the same `bootstrap_servers` and security settings as the consumer. * **The DLQ topic must already exist on the broker.** The plugin does not create it. If your broker has `auto.create.topics.enable=false` (typical for production and managed services such as Confluent Cloud or MSK), create the topic beforehand. When the topic is missing, delivery fails and — under `on_success` — the offset is not committed, so the batch is reprocessed until the topic exists. The real broker reason (for example `Unknown topic or partition`) is logged so you can fix it quickly. ### Deduplication parameters In TOML configuration, `dedup_window` is placed under the `[kafka]` section; `dedup_id_field` and `dedup_id_name` are placed under `[mapping.json]` or `[mapping.text]` section. | Parameter | Type | Default | Description | |----------------|------|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`dedup_id_field`|string| none |Extractor for a unique message id (enables deduplication). Holds **only** the extractor: JSON — a field name in CLI args (extracted via `$.`) or a JSONPath in TOML (e.g. `$.event_id`); Text — a regex pattern with a capturing group.| |`dedup_id_name` |string|“dedup\_id”| Field name the extracted id is written under and tracked by. Optional; only used with `dedup_id_field`. | | `dedup_window` |string| “24h” | Lookback window for the duplicate check (e.g. `30m`, `24h`, `7d`). Only used with `dedup_id_field`. | **Behavior:** * Deduplication is **only active when no `timestamp_field` is configured**. When messages carry their own timestamp, InfluxDB already overwrites duplicates by `(measurement, tags, time)`, so no extra check is needed. * When active, the plugin extracts the id, writes it under the `dedup_id_name` field (default `dedup_id`), and — before writing — checks whether that id already exists within `dedup_window`. The check is a **single query per table per cycle**, constrained by both the time window and the batch’s ids (`... WHERE time >= AND "" IN (...)`), so it returns few rows. Duplicates within the same batch are also dropped. * If the dedup query fails, the plugin **fails open** (writes the records) rather than dropping data. * A message that is missing the configured id is treated as a parse failure (routed to the DLQ / `kafka_exceptions`). For JSON arrays, individual elements missing the id are skipped with a warning. * **Performance:** for very high-volume topics prefer a shorter `dedup_window` to keep the query light. * Pick a `dedup_id_name` that does not collide with a field you already map — it would be written twice. ### Message format parameters In TOML configuration, `format` is placed under the `[kafka]` section; `timestamp_field` is placed under `[mapping.json]` or `[mapping.text]` section. | Parameter | Type |Default| Description | |-----------------|------|-------|--------------------------------------------------------------------| | `format` |string|“json” |Message format: json, lineprotocol, text, avro, jsonschema, protobuf| |`timestamp_field`|string| none | Field containing timestamp (format depends on message format) | **Supported formats:** | Format | Description | |--------------|----------------------------------------------------------------------| | `json` | JSON with JSONPath field mapping | |`lineprotocol`| InfluxDB Line Protocol passthrough | | `text` | Plain text with regex-based parsing | | `avro` | Avro via Confluent Schema Registry (JSONPath mapping) | | `jsonschema` | JSON Schema via Confluent Schema Registry (JSONPath) | | `protobuf` |Protobuf (Confluent wire format) via Schema Registry or a local .proto| **Format-specific timestamp\_field syntax:** |Format| Syntax | Split Method | Example (CLI) | Example (TOML) | |------|-------------------|------------------|----------------|------------------| | JSON |`field_name:format`|Split by first `:`|`"timestamp:ms"`|`"$.timestamp:ms"`| | Text | `regex:format` |Split by last `:` |`"ts:(\\d+):ms"`| `"ts:(\\d+):ms"` | **Supported timestamp formats:** * `ns` - nanoseconds (Unix timestamp) * `ms` - milliseconds (Unix timestamp) * `s` - seconds (Unix timestamp) * `datetime` - ISO 8601 string (for example, “2021-12-01T12:00:00Z”) For `avro` and `jsonschema`, a field decoded as a native `datetime`/`date`(Avro/JSON Schema logical type) is converted to a timestamp automatically, regardless of the configured format specifier. ### JSON format parameters In TOML configuration, `tags` are placed under `[mapping.json.tags]` section and `fields` under `[mapping.json.fields]` section. |Parameter| Type |Default | Description | |---------|------|--------|-------------------------------------------------------------------------| | `tags` |string| none | Space-separated tag names. Example: “room sensor location” | |`fields` |string|required|Space-separated field mappings. Format: “name:type=jsonpath” without `$.`| **Field specification format:** `"temp:float=temperature hum:int=humidity status:bool=online"` **Supported field types:** `int`, `uint`, `float`, `string`, `bool` ### Text format parameters In TOML configuration, `tags` are placed under `[mapping.text.tags]` section and `fields` under `[mapping.text.fields]` section. |Parameter| Type |Default | Description | |---------|------|--------|------------------------------------------------------------------| | `tags` |string| none | Space-separated tag patterns. Format: “name=regex\_pattern” | |`fields` |string|required|Space-separated field patterns. Format: “name:type=regex\_pattern”| ### Binary formats The decoded record is a dict (or a top-level list, expanded into one record per element) for all binary formats, so field/tag/timestamp mapping uses the same JSONPath syntax as the JSON format, under `[mapping.avro]`,`[mapping.jsonschema]`, or `[mapping.protobuf]`. Per-message decode failures (schema id not found, malformed payload) are handled like parse failures: the message is logged to `kafka_exceptions`, sent to`dlq_topic` if configured, or replayed otherwise. Binary payloads are forwarded to the DLQ as the original raw bytes. An unreachable or misconfigured Schema Registry is caught when the decoder is first built: the cycle is aborted before any message is polled or committed, so nothing is lost, and the run retries on the next schedule. #### Avro / JSON Schema (Schema Registry) Set `format` to `avro` or `jsonschema` to consume binary messages in the Confluent wire format (magic byte + 4-byte schema id + payload). The schema is fetched from the Schema Registry by the id embedded in each message — no local schema files are needed. The parameter names are identical in CLI/API arguments and TOML. In TOML they are placed under the `[kafka.schema_registry]` section. | Parameter | Type |Default | Description | |-------------------------------------|------|--------|-----------------------------------------------------------------------------------------------------------| | `schema_registry_url` |string|required|Schema Registry URL. Required for `avro` / `jsonschema`, and for `protobuf` without `protobuf_schema_file`.| | `schema_registry_username` |string| none | Basic-auth username (or API key for Confluent Cloud). | | `schema_registry_password` |string| none | Basic-auth password (or API secret). Used with username. | | `schema_registry_ca_cert` |string| none | CA certificate for TLS to the registry. | | `schema_registry_client_cert` |string| none | Client certificate for mutual TLS to the registry. | | `schema_registry_client_key` |string| none | Client private key for mutual TLS. Requires client cert. | |`schema_registry_client_key_password`|string| none | Password for an encrypted client private key. | > > > **Note:** `avro` and `jsonschema` require the extra packages `httpx`,`fastavro`, and `jsonschema`. > > #### Avro (local .avsc schema, schemaless) Set `format` to `avro` and `avro_schema_file` to a local `.avsc` schema to consume Avro messages produced **without** a Schema Registry — raw schemaless encoding with no Confluent wire header. The whole message value is the Avro body and is decoded with the local schema; the Schema Registry is **not** contacted. The parameter name is identical in CLI/API arguments and TOML. In TOML it is placed under the `[kafka.avro]` section. | Parameter | Type |Default | Description | |------------------|------|--------|-----------------------------------------------------------------------------| |`avro_schema_file`|string|required|Path to a local `.avsc`/`.json` schema (absolute or relative to PLUGIN\_DIR).| > > > **Note:** requires the `fastavro` package. The schema is parsed once and > cached. Writer and reader schema are assumed identical (no schema evolution). > > #### Protobuf (local .proto or Schema Registry) Set `format` to `protobuf` to consume Protobuf messages in the Confluent wire format. The schema source is chosen implicitly, like `avro`: * **Local `.proto`** — set `protobuf_schema_file`. The wire framing is parsed locally and the schema is read from the file; the Schema Registry is not contacted. * **Schema Registry** — omit `protobuf_schema_file` and set `schema_registry_url`. The `.proto` (and any referenced schemas) is fetched by the wire-frame schema id, compiled with `protoc`, and cached per schema id. The `[kafka.schema_registry]`section (url, auth, TLS) is shared with `avro`/`jsonschema`. The message index selects which message type to decode, or set`protobuf_message_type` to choose it explicitly. In registry mode the schema matches the producer’s, so the message index is always correct and`protobuf_message_type` is usually unnecessary. The parameter names are identical in CLI/API arguments and TOML. In TOML they are placed under the `[kafka.protobuf]` section. | Parameter | Type | Default | Description | |-----------------------|------|------------------|-----------------------------------------------------------------------------------------| |`protobuf_schema_file` |string| conditional |Path to a local `.proto` (absolute or relative to PLUGIN\_DIR). Omit to use the registry.| |`protobuf_include_dir` |string|schema file’s dir | Include directory for resolving `import` statements (local mode only). | |`protobuf_message_type`|string|wire message index| Fully-qualified message name to decode (e.g. `sensors.SensorReading`). | > > > **Note:** `protobuf` requires `grpcio-tools` (bundles `protoc` to compile the`.proto`); it is included in the plugin’s dependencies and installed > automatically. The `.proto` is > compiled once and cached. `MessageToDict` semantics apply: proto3 scalar fields > equal to their default (`0`/`""`/`false`) are omitted from the record;`int64`/`uint64` become strings; enums become their names; `bytes` become > base64 strings. > > See examples 11–14 in [kafka\_config\_example.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/kafka_subscriber/kafka_config_example.toml). ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|-----------------------------------------------| |`config_file_path`|string| none |Path to TOML config file (absolute or relative)| ### File path resolution All file paths in the plugin (configuration file, TLS certificates) follow the same resolution logic: * **Absolute paths** (for example, `/etc/kafka/config.toml`) are used as-is * **Relative paths** (for example, `config.toml`, `certs/ca.crt`) are resolved against the plugin directory, taken from `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV` If a relative path is specified and none of these can be resolved, the plugin will return an error. #### Example TOML configuration [kafka\_config\_example.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/kafka_subscriber/kafka_config_example.toml) - comprehensive configuration example with all formats and security options ## Data requirements The plugin automatically creates the target measurement table on first write. Field mappings are required for JSON and Text formats to specify which fields to extract and their data types. ### Message encoding requirements * **Text formats** (`json`, `text`, `lineprotocol`): payloads must be UTF-8 encoded. * **Binary formats** (`avro`, `jsonschema`, `protobuf`): payloads are decoded from binary (see [Binary formats](#binary-formats)). * **Non-empty payloads**: Empty or whitespace-only messages are automatically skipped. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages** (all bundled in `manifest.toml` and installed automatically with the plugin): * `confluent-kafka>=2.15.0` (Kafka client based on librdkafka) — all formats * `jsonpath-ng` (JSONPath field mapping) — all formats * `fastavro` — `avro` format (both Schema Registry and local `.avsc`) * `httpx`, `authlib`, `cachetools` — any Schema Registry format (`avro` / `jsonschema` / registry-mode `protobuf`) * `jsonschema` (library) — `jsonschema` message format only * `protobuf`, `grpcio-tools` — `protobuf` format only ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package confluent-kafka influxdb3 install package jsonpath-ng ``` ## Trigger setup ### Scheduled ingestion with TOML configuration Recommended for production use with complex mappings: ```bash # 1. Set PLUGIN_DIR environment variable export PLUGIN_DIR=~/.plugins # 2. Copy and edit configuration file cp kafka_config_example.toml $PLUGIN_DIR/my_kafka_config.toml # Edit my_kafka_config.toml with your Kafka cluster and mapping settings # 3. Create the trigger influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/kafka_subscriber/kafka_subscriber.py \ --trigger-spec "every:10s" \ --trigger-arguments config_file_path=my_kafka_config.toml \ kafka_ingestion ``` ### Scheduled ingestion with command-line arguments For simple JSON message ingestion: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/kafka_subscriber/kafka_subscriber.py \ --trigger-spec "every:5s" \ --trigger-arguments 'bootstrap_servers=kafka1:9092 kafka2:9092,topics=sensors.temperature sensors.humidity,group_id=influxdb3_consumer,format=json,table_name=sensor_data,fields=temp:float=temperature hum:int=humidity,tags=location sensor_id' \ kafka_sensors ``` ### Secure Kafka connection with SASL\_SSL ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/kafka_subscriber/kafka_subscriber.py \ --trigger-spec "every:10s" \ --trigger-arguments 'bootstrap_servers=kafka1:9093 kafka2:9093,topics=secure.data,group_id=influxdb3_secure,format=json,table_name=secure_data,security_protocol=SASL_SSL,sasl_mechanism=SCRAM-SHA-512,sasl_username=myuser,sasl_password=mypass,ssl_ca_cert=certs/ca.crt,fields=value:float=value' \ secure_kafka ``` ## Message Formats ### JSON Format The primary use case for structured data. Supports nested fields using JSONPath expressions. #### TOML Configuration ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["sensors.temperature"] group_id = "influxdb3_json" format = "json" [mapping.json] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.json.tags] location = "$.location" sensor_id = "$.sensor.id" [mapping.json.fields] temperature = ["$.temp", "float"] humidity = ["$.humidity", "int"] status = ["$.online", "bool"] ``` #### Example Message ```json { "timestamp": 1638360000000, "location": "warehouse_a", "sensor": { "id": "sensor_001" }, "temp": 22.5, "humidity": 65, "online": true } ``` #### Resulting Data ``` sensor_data,location=warehouse_a,sensor_id=sensor_001 temperature=22.5,humidity=65i,status=true 1638360000000000000 ``` #### JSON Array Support Process batch messages containing arrays of JSON objects: ```json [ {"timestamp": 1638360000000, "sensor_id": "001", "temperature": 22.5}, {"timestamp": 1638360001000, "sensor_id": "002", "temperature": 23.1}, {"timestamp": 1638360002000, "sensor_id": "003", "temperature": 21.8} ] ``` **Array processing behavior:** * Each array element is processed independently as a separate data point * If one element fails to parse, the others continue processing (partial success) * Parse errors for individual elements are logged as warnings and the element is skipped (not written to `kafka_exceptions`) * Statistics count 1 Kafka message = 1 unit (regardless of array size) ### Line Protocol Format For messages already in InfluxDB line protocol format, use passthrough mode. No mapping configuration needed. #### TOML Configuration ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["influxdb.metrics"] group_id = "influxdb3_lineprotocol" format = "lineprotocol" ``` #### Example Message ``` sensor_data,location=warehouse_a,sensor_id=001 temperature=22.5,humidity=65i 1638360000000000000 ``` ### Text Format Parse plain text messages using regular expressions: #### TOML Configuration ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["legacy.logs"] group_id = "influxdb3_text" format = "text" [mapping.text] table_name = "sensor_logs" timestamp_field = "ts:(\\d+):ms" [mapping.text.tags] location = "location=([^,\\s]+)" [mapping.text.fields] temperature = ["temp:([\\d.]+)", "float"] humidity = ["hum:(\\d+)", "int"] ``` #### Example Message ``` location=warehouse_a,temp:22.5,hum:65,ts:1638360000000 ``` ### Avro Format (Schema Registry) Decode Avro messages in the Confluent wire format. The schema is resolved from the Schema Registry by the id embedded in each message; mapping uses JSONPath on the decoded record. #### TOML Configuration ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["sensors.avro"] group_id = "influxdb3_avro" format = "avro" [kafka.schema_registry] schema_registry_url = "http://schema-registry:8081" # schema_registry_username = "sr_api_key" # optional basic auth # schema_registry_password = "sr_api_secret" [mapping.avro] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.avro.tags] device_id = "$.device_id" [mapping.avro.fields] temperature = ["$.temperature", "float"] humidity = ["$.humidity", "float"] ``` ### Avro Format (local .avsc schema, schemaless) Decode raw schemaless Avro (no Schema Registry, no wire header) with a local`.avsc` schema. Set `avro_schema_file` to switch `format = "avro"` to this mode. ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["sensors.avro"] group_id = "influxdb3_avro_file" format = "avro" [kafka.avro] avro_schema_file = "schemas/sensor.avsc" [mapping.avro] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.avro.tags] device_id = "$.device_id" [mapping.avro.fields] temperature = ["$.temperature", "float"] humidity = ["$.humidity", "float"] ``` ### JSON Schema Format (Schema Registry) Decode JSON-Schema messages in the Confluent wire format. Configuration mirrors the Avro example, using `format = "jsonschema"` and `[mapping.jsonschema]`. ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["events.jsonschema"] group_id = "influxdb3_jsonschema" format = "jsonschema" [kafka.schema_registry] schema_registry_url = "http://schema-registry:8081" [mapping.jsonschema] table_name = "events" timestamp_field = "$.ts:ms" [mapping.jsonschema.fields] value = ["$.value", "float"] ``` ### Protobuf Format (local .proto schema) Decode Protobuf messages in the Confluent wire format using a local `.proto`file. The Schema Registry is not contacted; the message type is selected by the wire message index, or set `protobuf_message_type` to choose it explicitly. ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["sensors.protobuf"] group_id = "influxdb3_protobuf" format = "protobuf" [kafka.protobuf] protobuf_schema_file = "schemas/sensor.proto" # protobuf_include_dir = "schemas" # optional, for .proto imports # protobuf_message_type = "sensors.SensorReading" # optional, defaults to wire message index [mapping.protobuf] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.protobuf.tags] device_id = "$.device_id" [mapping.protobuf.fields] temperature = ["$.temperature", "float"] humidity = ["$.humidity", "float"] ``` ### Protobuf Format (Schema Registry) Omit `protobuf_schema_file` and set `schema_registry_url` to fetch and compile the `.proto` (and any referenced schemas) by the wire-frame schema id. The`[kafka.protobuf]` section can be omitted entirely. ```toml [kafka] bootstrap_servers = ["kafka:9092"] topics = ["sensors.protobuf"] group_id = "influxdb3_protobuf_sr" format = "protobuf" [kafka.schema_registry] schema_registry_url = "http://schema-registry:8081" [mapping.protobuf] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.protobuf.tags] device_id = "$.device_id" [mapping.protobuf.fields] temperature = ["$.temperature", "float"] humidity = ["$.humidity", "float"] ``` ## Statistics and Monitoring The plugin tracks comprehensive statistics and writes them to the `kafka_stats` table on every plugin invocation. ### kafka\_stats Table | Field |Type | Description | |-------------------------|-----|----------------------------------------| | `topic` (tag) | tag | Kafka topic name | | `partition` (tag) | tag | Partition number | | `consumer_group` (tag) | tag | Consumer group ID | |`bootstrap_servers` (tag)| tag | Kafka cluster address | | `messages_received` | int | Total messages received | | `messages_processed` | int | Successfully processed messages | | `messages_failed` | int | Failed messages | | `last_offset` | int |Last processed offset for this partition| | `success_rate` |float| Percentage of successful messages | ### Querying Statistics ```bash # Get latest statistics influxdb3 query --database mydb \ "SELECT * FROM kafka_stats ORDER BY time DESC LIMIT 10" # Success rate by topic influxdb3 query --database mydb \ "SELECT topic, partition, success_rate, messages_processed, messages_failed FROM kafka_stats WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC" # Check consumer lag indicators influxdb3 query --database mydb \ "SELECT topic, partition, last_offset FROM kafka_stats WHERE consumer_group = 'influxdb3_consumer' ORDER BY time DESC LIMIT 10" ``` ## Error Handling Parse errors and message processing failures are logged to the `kafka_exceptions` table: ### kafka\_exceptions Table | Field | Type | Description | |------------------|------|--------------------------------------------| | `topic` (tag) | tag | Kafka topic where error occurred | |`partition` (tag) | tag | Partition number | |`error_type` (tag)| tag |Type of error (for example, JSONDecodeError)| | `offset` | int | Message offset | | `error_message` |string| Detailed error message | ### Checking for Errors ```bash influxdb3 query --database mydb \ "SELECT * FROM kafka_exceptions ORDER BY time DESC LIMIT 10" ``` ### Dead-letter queue When `dlq_topic` is configured, messages that fail to parse are additionally republished to that Kafka topic with the original payload and key, so they can be inspected or replayed without re-reading the source topic. Error context is attached as Kafka headers: | Header | Description | |---------------|----------------------------------------------| |`source_topic` |Topic the message was originally consumed from| | `error_type` | Exception type (e.g. `JSONDecodeError`) | |`error_message`| Error detail (truncated to 1KB) | Transient InfluxDB write failures are not sent to the DLQ — they are retried on the next cycle (with `offset_commit_policy=on_success`). ## Troubleshooting ### Check Plugin Logs ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'kafka_ingestion' ORDER BY time DESC LIMIT 20" ``` ### Common Issues #### “confluent-kafka library not installed” or “No module named ‘confluent\_kafka’” ```bash influxdb3 install package confluent-kafka ``` If you encounter `librdkafka` related errors: * Ubuntu/Debian: `sudo apt-get install librdkafka-dev` * macOS: `brew install librdkafka` #### “Configuration file not found” * For relative paths, ensure `PLUGIN_DIR` environment variable is set * For absolute paths, verify the file exists at the specified location ```bash # For relative paths export PLUGIN_DIR=~/.plugins ls $PLUGIN_DIR/my_kafka_config.toml # Or use absolute path ls /etc/kafka/my_kafka_config.toml ``` #### “Failed to connect to Kafka cluster” * Verify bootstrap\_servers addresses and ports * Check network connectivity to Kafka brokers * For SSL connections, verify certificate paths * For SASL authentication, verify credentials #### “SASL mechanism required when security\_protocol includes SASL” When using `SASL_PLAINTEXT` or `SASL_SSL`, you must provide: * `sasl_mechanism` * `sasl_username` * `sasl_password` #### “No fields were mapped from JSON data” * Verify JSONPath expressions in field mappings (use `$.` prefix) * Check that JSON structure matches your paths * Review `kafka_exceptions` table for detailed errors #### Messages not being processed * Check trigger status: `influxdb3 show summary --database mydb` * Verify Kafka connection in plugin logs * Check consumer group lag using Kafka tools * Increase trigger frequency (for example, from `every:10s` to `every:5s`) * If many messages are queued, increase `max_poll_records` or set to `0` (unlimited) #### Offset commit issues with `on_success` policy If using `offset_commit_policy=on_success` and seeing repeated message processing: * Check `kafka_exceptions` table for processing errors * Fix the root cause of failures * Consider using `offset_commit_policy=always` if some data loss is acceptable ## Architecture ### How It Works 1. **Scheduled Trigger**: Plugin runs on schedule (for example, `every:10s`) 2. **Configuration Caching**: Plugin configuration is parsed once and cached between executions 3. **Persistent Consumer**: The Kafka consumer is created once and reused across invocations (cached), so it stays in its consumer group and no rebalance happens every cycle 4. **Consumer Poll**: Consumer polls for available messages (drains all available messages) 5. **Offset Commit**: Based on `offset_commit_policy`: * `always`: Commits immediately after poll * `on_success`: Commits only after successful processing of all messages 6. **Deduplication** (optional): When `dedup_id_field` is set and no message timestamp is configured, duplicate ids within `dedup_window` are skipped 7. **Parse & Write**: Messages parsed according to format and written to InfluxDB 8. **Error Tracking**: Parse errors logged to `kafka_exceptions` table (and `dlq_topic` if configured) 9. **Statistics**: Written to `kafka_stats` table on every plugin invocation ### Performance Optimization The plugin includes several optimizations for high-throughput scenarios: * **Persistent Consumer**: The consumer is reused across trigger executions instead of reconnecting each cycle, eliminating the consumer-group rebalance (and its latency/broker overhead) that a connect/disconnect-per-cycle would cause. It stays in the group while the schedule interval stays below `max_poll_interval`. * **Configuration Caching**: Plugin configuration is parsed once and reused across all trigger executions * **Pre-compiled Patterns**: JSONPath expressions and regex patterns are compiled once during parser initialization * **Batch Polling**: Consumer drains available messages in a single trigger execution (limited by `max_poll_records`, default 500) * **Manual Offset Commit**: Provides control over exactly-once vs at-least-once semantics ### Consumer Group Behavior * A single consumer connection is created and reused across trigger executions; it is only rebuilt after a fatal connection/poll error or a configuration change * Set `max_poll_interval` above your trigger interval so the broker does not evict the reused consumer between polls (this would force a rebalance) * Consumer group ID ensures partitions are assigned consistently * Offsets are committed to Kafka for durability * Multiple plugin instances with the same `group_id` will share partitions (load balancing) ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Kafka subscriber plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/kafka_subscriber) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | bootstrap_servers | string | required | Space-separated list of Kafka broker addresses (for example, “kafka1:9092 kafka2:9092”) | | topics | string | required | Space-separated list of topics (for example, “sensor_data metrics”) | | group_id | string | required | Kafka consumer group ID (must be unique per consumer group) | | table_name | string | conditional | InfluxDB measurement name for storing data. Required for all formats except lineprotocol , unless table_name_field is set. | | table_name_field | string | none | JSON field name or regex pattern to extract table name dynamically from each message. Alternative to static table_name . | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | auto_offset_reset | string | “earliest” | Where to start consuming on first connect: “earliest” or “latest” | | max_poll_records | int | 500 | Maximum messages per scheduled call. Set to 0 for unlimited. | | max_poll_interval | int | 300 | Maximum seconds between scheduled poll cycles before the broker considers the consumer dead (maps to max.poll.interval.ms ). The consumer is reused across invocations, so it must stay in the group between polls — increase this when the trigger interval is longer than ~5 minutes. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | offset_commit_policy | string | “on_success” | When to commit offsets: “on_success” or “always” | | Policy | Behavior | | --- | --- | | Policy | Behavior | | on_success | Commit offsets only after ALL messages in the batch are successfully processed and written | | always | Commit offsets immediately after receiving messages, regardless of processing success | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | security_protocol | string | “PLAINTEXT” | Security protocol: “PLAINTEXT”, “SSL”, “SASL_PLAINTEXT”, “SASL_SSL” | | Parameter (CLI) | Parameter (TOML) | Type | Default | Description | | --- | --- | --- | --- | --- | | Parameter (CLI) | Parameter (TOML) | Type | Default | Description | | sasl_mechanism | [kafka.sasl] mechanism | string | none | SASL mechanism: “PLAIN”, “SCRAM-SHA-256”, “SCRAM-SHA-512” | | sasl_username | [kafka.sasl] username | string | none | SASL username (required with sasl_mechanism) | | sasl_password | [kafka.sasl] password | string | none | SASL password (required with sasl_mechanism) | | Parameter (CLI) | Parameter (TOML) | Type | Default | Description | | --- | --- | --- | --- | --- | | Parameter (CLI) | Parameter (TOML) | Type | Default | Description | | ssl_ca_cert | [kafka.ssl] ca_cert | string | none | Path to CA certificate file | | ssl_cert | [kafka.ssl] client_cert | string | none | Path to client certificate for mutual TLS | | ssl_key | [kafka.ssl] client_key | string | none | Path to client private key for mutual TLS | | ssl_key_password | [kafka.ssl] key_password | string | none | Password for encrypted client private key | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths) into log output. Enable temporarily for debugging. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | dlq_topic | string | none | Kafka topic to publish messages that fail to parse. The original payload and key are produced unchanged, with source_topic , error_type and error_message set as Kafka headers. When unset, no DLQ topic is used. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | dedup_id_field | string | none | Extractor for a unique message id (enables deduplication). Holds only the extractor: JSON — a field name in CLI args (extracted via $. ) or a JSONPath in TOML (e.g. $.event_id ); Text — a regex pattern with a capturing group. | | dedup_id_name | string | “dedup_id” | Field name the extracted id is written under and tracked by. Optional; only used with dedup_id_field . | | dedup_window | string | “24h” | Lookback window for the duplicate check (e.g. 30m , 24h , 7d ). Only used with dedup_id_field . | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | format | string | “json” | Message format: json, lineprotocol, text, avro, jsonschema, protobuf | | timestamp_field | string | none | Field containing timestamp (format depends on message format) | | Format | Description | | --- | --- | | Format | Description | | json | JSON with JSONPath field mapping | | lineprotocol | InfluxDB Line Protocol passthrough | | text | Plain text with regex-based parsing | | avro | Avro via Confluent Schema Registry (JSONPath mapping) | | jsonschema | JSON Schema via Confluent Schema Registry (JSONPath) | | protobuf | Protobuf (Confluent wire format) via Schema Registry or a local .proto | | Format | Syntax | Split Method | Example (CLI) | Example (TOML) | | --- | --- | --- | --- | --- | | Format | Syntax | Split Method | Example (CLI) | Example (TOML) | | JSON | field_name:format | Split by first : | "timestamp:ms" | "$.timestamp:ms" | | Text | regex:format | Split by last : | "ts:(\\d+):ms" | "ts:(\\d+):ms" | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tags | string | none | Space-separated tag names. Example: “room sensor location” | | fields | string | required | Space-separated field mappings. Format: “name:type=jsonpath” without $. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tags | string | none | Space-separated tag patterns. Format: “name=regex_pattern” | | fields | string | required | Space-separated field patterns. Format: “name:type=regex_pattern” | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | schema_registry_url | string | required | Schema Registry URL. Required for avro / jsonschema , and for protobuf without protobuf_schema_file . | | schema_registry_username | string | none | Basic-auth username (or API key for Confluent Cloud). | | schema_registry_password | string | none | Basic-auth password (or API secret). Used with username. | | schema_registry_ca_cert | string | none | CA certificate for TLS to the registry. | | schema_registry_client_cert | string | none | Client certificate for mutual TLS to the registry. | | schema_registry_client_key | string | none | Client private key for mutual TLS. Requires client cert. | | schema_registry_client_key_password | string | none | Password for an encrypted client private key. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | avro_schema_file | string | required | Path to a local .avsc / .json schema (absolute or relative to PLUGIN_DIR). | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | protobuf_schema_file | string | conditional | Path to a local .proto (absolute or relative to PLUGIN_DIR). Omit to use the registry. | | protobuf_include_dir | string | schema file’s dir | Include directory for resolving import statements (local mode only). | | protobuf_message_type | string | wire message index | Fully-qualified message name to decode (e.g. sensors.SensorReading ). | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | Path to TOML config file (absolute or relative) | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | topic (tag) | tag | Kafka topic name | | partition (tag) | tag | Partition number | | consumer_group (tag) | tag | Consumer group ID | | bootstrap_servers (tag) | tag | Kafka cluster address | | messages_received | int | Total messages received | | messages_processed | int | Successfully processed messages | | messages_failed | int | Failed messages | | last_offset | int | Last processed offset for this partition | | success_rate | float | Percentage of successful messages | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | topic (tag) | tag | Kafka topic where error occurred | | partition (tag) | tag | Partition number | | error_type (tag) | tag | Type of error (for example, JSONDecodeError) | | offset | int | Message offset | | error_message | string | Detailed error message | | Header | Description | | --- | --- | | Header | Description | | source_topic | Topic the message was originally consumed from | | error_type | Exception type (e.g. JSONDecodeError ) | | error_message | Error detail (truncated to 1KB) | --- # MAD anomaly detection plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/mad-anomaly-detection/ The MAD-Based Anomaly Detection Plugin provides real-time anomaly detection for time series data in InfluxDB 3 Enterprise using Median Absolute Deviation (MAD). Detect outliers in your field values as data is written, with configurable thresholds for both count-based and duration-based alerts. The plugin maintains in-memory deques for efficient computation and integrates with the Notification Sender Plugin to deliver alerts via multiple channels. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |----------------|------|--------|--------------------------------------------------------------------------| | `measurement` |string|required| Source measurement to monitor for anomalies | |`mad_thresholds`|string|required| MAD threshold conditions. Format: `field:k:window_count:threshold` | | `senders` |string|required|Dot-separated list of notification channels (for example, “slack.discord”)| ### MAD threshold parameters | Component | Description | Example | |--------------|--------------------------------------------------------------|-------------| | `field_name` | The numeric field to monitor | `temp` | | `k` | MAD multiplier for the anomaly cutoff (float, ≥ 0) | `2.5` | |`window_count`|Number of recent points for MAD computation (integer, 2–10000)| `20` | | `threshold` | Consecutive outliers (integer, ≥ 1) or a duration |`5` or `2min`| Multiple thresholds are separated by `@`: `temp:2.5:20:5@load:3:10:2min` Durations use the format ``, where unit is `us` (microseconds), `ms` (milliseconds), `s` (seconds), `min` (minutes), `h` (hours), `d` (days), or `w` (weeks). Thresholds that share a field and `window_count` share one MAD window, so you can combine a count-based and a duration-based alert on the same detector: `temp:2.5:20:5@temp:2.5:20:2min`. Invalid thresholds are skipped with a warning; if none remain, the plugin logs an error and stops. Repeated identical thresholds are also skipped with a warning, because they would share one counter. ### Optional parameters | Parameter | Type | Default | Description | |-------------------------|------|------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `influxdb3_auth_token` |string| env var | API token for InfluxDB 3 Enterprise (or use INFLUXDB3\_AUTH\_TOKEN env var) | | `state_change_count` |string| “0” |Number of transitions between normal and outlier state, within the MAD window, at which notifications are suppressed. Use 2 or greater; `1` is treated as `0`. See *Flip Detection*| |`notification_count_text`|string|see *Default notification templates*| Template for count-based alerts with variables: $table, $field, $threshold\_count, $tags | |`notification_time_text` |string|see *Default notification templates*| Template for duration-based alerts with variables: $table, $field, $threshold\_time, $tags | | `notification_path` |string| “notify” | URL path for the notification sending plugin | | `port_override` |string| “8181” | Port number where InfluxDB accepts requests | #### Default notification templates * Count: `"MAD count alert: Field $field in $table outlier for $threshold_count consecutive points. Tags: $tags"` * Time: `"MAD duration alert: Field $field in $table outlier for $threshold_time. Tags: $tags"` ### Notification channel parameters #### Slack | Parameter | Type |Required| Description | |-------------------|------|--------|---------------------------| |`slack_webhook_url`|string| Yes | Webhook URL from Slack | | `slack_headers` |string| No |Base64-encoded HTTP headers| #### Discord | Parameter | Type |Required| Description | |---------------------|------|--------|---------------------------| |`discord_webhook_url`|string| Yes | Webhook URL from Discord | | `discord_headers` |string| No |Base64-encoded HTTP headers| #### HTTP | Parameter | Type |Required| Description | |------------------|------|--------|------------------------------------| |`http_webhook_url`|string| Yes |Custom webhook URL for POST requests| | `http_headers` |string| No | Base64-encoded HTTP headers | #### SMS/WhatsApp (via Twilio) | Parameter | Type |Required| Description | |--------------------|------|--------|------------------------------------------------| | `twilio_sid` |string| Yes |Twilio Account SID (or use TWILIO\_SID env var) | | `twilio_token` |string| Yes |Twilio Auth Token (or use TWILIO\_TOKEN env var)| |`twilio_from_number`|string| Yes | Sender phone number | | `twilio_to_number` |string| Yes | Recipient phone number | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Relative paths are resolved against the first directory that is set: `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. Only that directory is used — the file is not looked up in the remaining ones. When `config_file_path` is set, the TOML file provides the whole configuration and inline trigger arguments are ignored. `INFLUXDB3_AUTH_TOKEN` from the environment still applies when `influxdb3_auth_token` is not set in the file. In TOML, `senders` and `mad_thresholds` use native structures (a list and a list of entries) instead of the inline string formats, though the inline strings are also accepted. The plugin caches the loaded configuration for 10 minutes to keep the write path fast, so configuration changes take effect within that window. #### Example TOML configuration [mad\_anomaly\_config\_data\_writes.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/mad_check/mad_anomaly_config_data_writes.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Python packages**: * `influxdata-plugin-utils>=0.3.0` (configuration loading, parsing, and schema introspection) * `requests` (for notification delivery) * **Notification Sender Plugin** *(optional)*: Required if using the `senders` parameter. See the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/). ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package influxdata-plugin-utils influxdb3 install package requests ``` 3. *(Optional)* For notifications, install the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) and create an HTTP trigger for it. ## Schema requirement The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. ## Trigger setup ### Real-time anomaly detection Detect anomalies as data is written: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/mad_check/mad_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments 'measurement=cpu,mad_thresholds="temp:2.5:20:5@load:3:10:2min",senders=slack,slack_webhook_url="$SLACK_WEBHOOK_URL"' \ mad_anomaly_detector ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ## Example usage ### Example 1: Basic count-based anomaly detection Detect when temperature exceeds 2.5 MADs from the median for 5 consecutive points: ```bash # Create trigger for count-based detection influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/mad_check/mad_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments 'measurement=environment,mad_thresholds="temperature:2.5:20:5",senders=slack,slack_webhook_url="$SLACK_WEBHOOK_URL"' \ temp_anomaly_detector # Write test data with an anomaly influxdb3 write \ --database sensors \ "environment,room=office temperature=22.1" influxdb3 write \ --database sensors \ "environment,room=office temperature=22.3" influxdb3 write \ --database sensors \ "environment,room=office temperature=45.8" # Anomaly # Continue writing anomalous values... ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. **Expected output** * Plugin maintains a 20-point window of recent temperature values * Computes median and MAD from this window * When temperature exceeds median ± 2.5\*MAD for 5 consecutive points, sends Slack notification * Notification includes: “MAD count alert: Field temperature in environment outlier for 5 consecutive points. Tags: room=office” ### Example 2: Duration-based anomaly detection with multiple fields Monitor CPU load and memory usage with different thresholds: ```bash # Create trigger with multiple thresholds influxdb3 create trigger \ --database monitoring \ --path "gh:influxdata/mad_check/mad_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments 'measurement=system_metrics,mad_thresholds="cpu_load:3:30:2min@memory_used:2.5:30:5min",senders=slack.discord,slack_webhook_url="$SLACK_WEBHOOK_URL",discord_webhook_url="$DISCORD_WEBHOOK_URL"' \ system_anomaly_detector ``` Set `SLACK_WEBHOOK_URL` and `DISCORD_WEBHOOK_URL` to your webhook URLs. **Expected output** * Monitors two fields independently: * `cpu_load`: Alerts when exceeds 3 MADs for 2 minutes * `memory_used`: Alerts when exceeds 2.5 MADs for 5 minutes * Sends notifications to both Slack and Discord ### Example 3: Anomaly detection with flip suppression Prevent alert fatigue from rapidly fluctuating values: ```bash # Create trigger with flip suppression influxdb3 create trigger \ --database iot \ --path "gh:influxdata/mad_check/mad_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments 'measurement=sensor_data,mad_thresholds="vibration:2:50:10",state_change_count=3,senders=http,http_webhook_url="$HTTP_WEBHOOK_URL",notification_count_text="Vibration anomaly detected on $table. Field: $field, Tags: $tags"' \ vibration_monitor ``` Set `HTTP_WEBHOOK_URL` to your HTTP webhook endpoint. **Expected output** * Detects vibration anomalies exceeding 2 MADs for 10 consecutive points * Suppresses notifications once the value has switched between normal and outlier state 3 times within the 50-point window, so two switches are still tolerated * Sends custom formatted message to HTTP endpoint ## Using TOML Configuration Files This plugin supports using TOML configuration files to specify all plugin arguments. ### Important Requirements **To use TOML configuration files, you must set the `PLUGIN_DIR` environment variable in the InfluxDB 3 Enterprise host environment.** ### Setting Up TOML Configuration 1. **Start InfluxDB 3 Enterprise with the PLUGIN\_DIR environment variable set**: ```bash PLUGIN_DIR=~/.plugins influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 1. **Copy the example TOML configuration file to your plugin directory**: ```bash cp mad_anomaly_config_data_writes.toml ~/.plugins/ ``` 1. **Edit the TOML file** to match your requirements: ``` # Required parameters measurement = "cpu" mad_thresholds = "temp:2.5:20:5@load:3:10:2min" senders = "slack" # Notification settings slack_webhook_url = "$SLACK_WEBHOOK_URL" notification_count_text = "Custom alert: $field anomaly detected" ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. 2. **Create a trigger using the `config_file_path` argument**: ``` influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/mad_check/mad_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments config_file_path=mad_anomaly_config_data_writes.toml \ mad_toml_trigger ``` ## Code overview ### Files * `mad_check_plugin.py`: The main plugin code containing the handler for data write triggers * `mad_anomaly_config_data_writes.toml`: Example TOML configuration file * `test_mad_check.py`: Pytest suite, runs without a live InfluxDB 3 Enterprise server * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`, `requests`) * `requirements-dev.txt`: Development dependencies (`pytest`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ### Main functions #### `process_writes(influxdb3_local, table_batches, args)` Handles real-time anomaly detection on incoming data. Key operations: 1. Filters table batches for the specified measurement 2. Maintains in-memory deques of recent values per field 3. Computes MAD for each monitored field 4. Tracks consecutive outliers and duration 5. Sends notifications when thresholds are met ### Key algorithms #### MAD (Median Absolute Deviation) Calculation ```python median = statistics.median(values) mad = statistics.median([abs(x - median) for x in values]) threshold = k * mad is_anomaly = abs(value - median) > threshold ``` When more than half of the values in the window are identical, `mad` is `0` and the bounds collapse onto the median, so any different value counts as an outlier no matter how large `k` is. This affects flat signals: a stable sensor, a metric that is usually `0`, or a low-resolution integer field. The effect also works in reverse — once outliers fill more than half of the window they become the new median and stop being detected. #### Flip Detection The plugin keeps the recent outlier flags of each threshold in a deque the size of `window_count` and counts transitions between normal and outlier state. Once the number of transitions reaches `state_change_count`, the alert is computed as usual but not delivered, and a warning is logged instead. This prevents alert fatigue from values that switch in and out of the outlier state. An alert that follows normal data always records one normal-to-outlier transition, so `state_change_count` must be 2 or greater to leave sustained anomalies alone. A value of `1` would suppress every alert; the plugin logs a warning and treats it as `0`. A count threshold needs consecutive outliers, so when it fires the last `threshold` flags are all outliers and only `window_count - threshold` transitions can remain in the window. Suppression therefore requires `window_count >= threshold + state_change_count`; otherwise the plugin logs a `Flip suppression never triggers` warning naming the field. Duration thresholds have no such limit. ## Troubleshooting ### Common issues #### Issue: No notifications being sent **Solution**: 1. Verify the Notification Sender Plugin is installed and running 2. Check webhook URLs are correct:`bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%notification%'"` 3. Ensure notification channel parameters are provided for selected senders #### Issue: “No valid MAD thresholds provided” error **Solution**: Each invalid threshold is logged as a warning naming the part that failed. Check the format: * Count-based: `field:k:window_count:count` (for example, `temp:2.5:20:5`) * Duration-based: `field:k:window_count:duration` (for example, `temp:2.5:20:2min`) * Multiple thresholds separated by `@` * `k` must not be negative, `window_count` must be 2 or greater, the count must be 1 or greater #### Issue: Alerts are logged but never delivered **Solution**: Look for `Suppressed count alert` or `Suppressed duration alert` warnings. They mean flip suppression is active. Raise `state_change_count`, or remove it to disable suppression. #### Issue: Too many false positive alerts **Solution**: 1. Increase the k multiplier (for example, from 2.5 to 3.0) 2. Increase the threshold count or duration 3. Enable flip suppression with `state_change_count` 4. Increase the window size for more stable statistics If the log line reports `mad=0.000`, the window has no spread and `k` has no effect. Require the change to persist with a count or duration threshold instead. #### Issue: Missing anomalies (false negatives) **Solution**: 1. Decrease the k multiplier 2. Decrease the threshold count or duration 3. Check if data has seasonal patterns that affect the median ### Debugging tips 1. **Check whether windows are still filling up**: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%Waiting for%points for MAD%'" ``` 1. **Check MAD calculations** (logged for detected outliers only): ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%MAD calculation%'" ``` 1. **Test with known anomalies**: Write test data with obvious outliers to verify detection ### Performance considerations * **Memory usage**: Each field and series maintains a deque of `window_count` values * **Computation**: MAD is computed on every data write for monitored fields * **Caching**: Measurement and tag names are cached for 1 hour, the loaded configuration for 10 minutes * **Early exit**: Writes that contain no rows of the configured measurement return before thresholds, senders and tags are parsed; the configuration and the table list come from the cache * **Notification delivery**: Each alert is sent in a single attempt with a 5-second timeout; retries would hold up the write path * **Logging**: MAD calculations are logged only for points detected as outliers, so a calm table produces two log lines per write ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [MAD-based anomaly detection plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/mad_check) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[anomaly-detection](/influxdb3/enterprise/tags/anomaly-detection/)[statistics](/influxdb3/enterprise/tags/statistics/)[monitoring](/influxdb3/enterprise/tags/monitoring/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Source measurement to monitor for anomalies | | mad_thresholds | string | required | MAD threshold conditions. Format: field:k:window_count:threshold | | senders | string | required | Dot-separated list of notification channels (for example, “slack.discord”) | | Component | Description | Example | | --- | --- | --- | | Component | Description | Example | | field_name | The numeric field to monitor | temp | | k | MAD multiplier for the anomaly cutoff (float, ≥ 0) | 2.5 | | window_count | Number of recent points for MAD computation (integer, 2–10000) | 20 | | threshold | Consecutive outliers (integer, ≥ 1) or a duration | 5 or 2min | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | influxdb3_auth_token | string | env var | API token for InfluxDB 3 Enterprise (or use INFLUXDB3_AUTH_TOKEN env var) | | state_change_count | string | “0” | Number of transitions between normal and outlier state, within the MAD window, at which notifications are suppressed. Use 2 or greater; 1 is treated as 0 . See Flip Detection | | notification_count_text | string | see Default notification templates | Template for count-based alerts with variables: $table, $field, $threshold_count, $tags | | notification_time_text | string | see Default notification templates | Template for duration-based alerts with variables: $table, $field, $threshold_time, $tags | | notification_path | string | “notify” | URL path for the notification sending plugin | | port_override | string | “8181” | Port number where InfluxDB accepts requests | | Parameter | Type | Required | Description | | --- | --- | --- | --- | | Parameter | Type | Required | Description | | slack_webhook_url | string | Yes | Webhook URL from Slack | | slack_headers | string | No | Base64-encoded HTTP headers | | Parameter | Type | Required | Description | | --- | --- | --- | --- | | Parameter | Type | Required | Description | | discord_webhook_url | string | Yes | Webhook URL from Discord | | discord_headers | string | No | Base64-encoded HTTP headers | | Parameter | Type | Required | Description | | --- | --- | --- | --- | | Parameter | Type | Required | Description | | http_webhook_url | string | Yes | Custom webhook URL for POST requests | | http_headers | string | No | Base64-encoded HTTP headers | | Parameter | Type | Required | Description | | --- | --- | --- | --- | | Parameter | Type | Required | Description | | twilio_sid | string | Yes | Twilio Account SID (or use TWILIO_SID env var) | | twilio_token | string | Yes | Twilio Auth Token (or use TWILIO_TOKEN env var) | | twilio_from_number | string | Yes | Sender phone number | | twilio_to_number | string | Yes | Recipient phone number | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | --- # Mqtt publisher plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/mqtt-publisher/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later and `paho-mqtt>=2.1,<3`. > > The MQTT Publisher Plugin sends rows out of InfluxDB 3 Enterprise to an MQTT broker. It publishes **what is already in the database** — it computes nothing — which makes it the outbound half of the pair whose inbound half is `mqtt_subscriber`. * **Three ways to be driven**: a write trigger publishes rows as they land, a scheduled trigger publishes what a table has gained since the previous tick, and an HTTP trigger replays a time range on demand * **Payload shapes consumers expect**: JSON, InfluxDB line protocol, or a template you write; one message per row, one per field with the bare value in the body, or one carrying a whole batch * **Topics built from the data**: `plant/{machine}/{_table}` resolves per row, and a substituted value can never add a topic level * **Survives a broker outage**: nothing is handed to a disconnected client; a write trigger buffers under a byte budget and a drop policy, while a scheduled trigger stops reading and leaves the rows in the database * **Runs no threads**: the connection lives in the trigger’s cache and is serviced by the invocation itself ## Contents * [The three triggers](#the-three-triggers) — what they share, how they differ * [Common behaviour](#common-behaviour) — the connection and the life of a message * [Write trigger](#write-trigger) · [Scheduled trigger](#scheduled-trigger) · [HTTP trigger](#http-trigger)— each with its own behaviour, full parameter list and examples * [Topics](#topics) · [Payloads](#payloads) · [Row selection](#row-selection) * [Where settings come from](#where-settings-come-from) — layers and formats * [Installation](#installation) · [Observability](#observability) · [Troubleshooting](#troubleshooting) ## The three triggers One plugin file serves all three. They share the connection, the topic and payload machinery, the filters and the buffer; they differ only in **where the rows come from** and therefore in what can go wrong. | | **Write** (`table:…`, `all_tables`) | **Scheduled** (`every:…`, `cron:…`) | **HTTP** (`request:…`) | |------------------------|-------------------------------------------------------|------------------------------------------------|----------------------------------------------| | Rows come from | the batch being written | a query over a moving window | a query over a range you name | | Fires when | data lands | the schedule says so | a request arrives | | Latency | milliseconds | up to one interval | on demand | | Position is kept by | nothing — the batch is all there is | a **watermark** per table, in the cache | the request’s own range | |While the broker is down|rows are **buffered in memory** under `max_queue_bytes`| nothing is read; the rows stay in the database | the call stops and reports how far it got | | Risk to understand | memory during a long outage |`settle_delay` versus how late the source writes|a range with no bounds replaying a whole table| |Publishes `status_topic`| yes | yes | no | **Which one to use** | Your source | Trigger | Why | |-----------------------------------------------------------------------|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| | A table your clients write | **write** | Publishes within milliseconds, and nothing is read back from the database | |A table another plugin derives — a rollup, a forecast, an anomaly score| **write** |A derived row carries the timestamp of the period it describes and is written well after it, so a schedule would find it below the watermark and never publish it| | A table you want sampled rather than mirrored |**scheduled**| One pass per tick, coalesced, paced by the schedule | | History, on demand | **HTTP** | A bounded replay that leaves the watermark alone | The three can coexist on one database — a write trigger for the live stream and an HTTP trigger for backfills is the common pairing. Give them **different**`client_id` values, or none at all, because a broker allows one session per identifier and evicts the previous holder. ## Common behaviour Everything in this section is the same whichever trigger you use. ### The connection A connection is built once and kept in `influxdb3_local.cache`, so an invocation that already has one spends well under a millisecond publishing. There is **no background network loop**: the invocation pumps the socket itself, which is what collects acknowledgements and notices a connection that has gone. An invocation does this, in order: 1. **Pump briefly.** A pending `FIN` from the broker is seen before any payload is handed over. 2. **Decide whether the client can be trusted.** It is replaced — never repaired — if the connection dropped, if a connection setting changed, or if it has been silent longer than `max_silence`. 3. **Drain the buffer**, handing over as many messages as the in-flight window and the back-pressure threshold allow. 4. **Pump again**, collecting whatever acknowledgements arrive within the budget. Three consequences are worth knowing before configuring anything: | | What follows | |------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Nothing runs between invocations |A trigger invoked less often than 1.5 × `keep_alive` loses its connection between calls and reconnects on the next one — cheaply, but `status_topic` then reports the invocation rhythm rather than availability| |Acknowledgements are collected only while an invocation runs| At QoS ≥ 1 the confirmed count trails the published one by up to one invocation, and no more than `max_inflight` messages are outstanding at a time | | A dropped connection is replaced, never repaired | Its unconfirmed messages go back to the **head** of the buffer, which is what lets a message the broker keeps refusing be retired after `max_delivery_attempts` | **Why replace rather than reconnect.** `reconnect()` leaves paho holding the messages that were in flight and resending them itself, so the plugin could never retire one the broker keeps refusing: a single oversized message would close the connection on every retry and nothing behind it would ever move. **Why no keepalive ping.** Proving a quiet link needs a `PINGREQ`, which paho does not expose. A trigger quiet for longer than `max_silence` is close to being dropped by the broker anyway, so the plugin pre-emptively builds a new client instead — and on a busy trigger `max_silence` is never reached. **QoS 0 and silence.** With `qos = 0` there is no acknowledgement to prove the link is alive, and paho’s `PINGRESP` is not visible to the plugin, so a trigger invoked less often than `max_silence` rebuilds its client on each invocation. That is harmless — a fresh client still delivers — only slightly wasteful; raise`max_silence` or use `qos = 1` if the reconnects are unwanted. ### The life of one message ``` row ──filters──▶ topic + payload ──▶ buffer ──▶ publish() ──▶ broker │ │ │ │ skipped too large drop_policy unconfirmed ──ack──▶ confirmed → dropped max_message_age │ reason ≥ 0x80 → dropped ``` * **Filters** are applied first, so nothing is encoded for a row that will not be published. * **Encoding** produces the topic and the payload. A message larger than`max_payload_bytes` is dropped and logged; a `batch` is split instead. * **The buffer** holds what has not been handed over, bounded by`max_queue_bytes` — counted as payload + topic + 320 bytes of Python overhead, so the limit tracks real memory rather than payload bytes. * **Handing over** happens only while the client is connected and the window has room. At QoS ≥ 1 the message waits in `unconfirmed` until acknowledged. * **A reason code of `0x80` or above** (MQTT 5 only) names the message it refers to, so there is nothing to retry: it is dropped at once, counted and logged, without spending `max_delivery_attempts`. The exception is `0x97`*quota exceeded*, which is about the rate rather than the message. **There is no delivery guarantee in either direction.** A message can be lost: the buffer lives in memory and a restart discards it, QoS 0 packets go with a dropped connection, and `drop_policy` and `max_message_age` discard by design. A message can equally be duplicated: an invocation that fails after handing rows over re-reads them, and QoS 1 is at-least-once by definition. ## Write trigger Publishes rows as they are written. The trigger specification may be`table:` or `all_tables`; rows of a table not named in`source_measurements` are ignored either way, so `all_tables` is safe and is what you want when the buffer should drain on any write. ### How it behaves * **The batch is all there is.** These rows exist nowhere the plugin can read them back from, so the in-memory buffer is the only thing between a broker outage and lost data. That is the opposite of the scheduled trigger, and it is why `max_queue_bytes`, `drop_policy` and `max_message_age` matter here. * **Ingest is never stalled.** An invocation hands over what it can within`pump_budget` and returns; whatever is left waits for the next write. * **No schema is needed up front.** A trigger created before the first write to its table starts with a warning and checks its placeholders and filter types once the table appears. * **A column added later is picked up.** On a write trigger the row that brings a new column triggers one schema re-read, so a new tag is emitted as a tag immediately. A scheduled or backfill trigger selects only the columns it already knows, so it picks up a new column within an hour — the schema cache’s TTL — or at once if the trigger is recreated. What an outage looks like, with `max_queue_bytes = 64 MiB`,`drop_policy = drop_oldest`, `max_message_age = 5min`: ``` t+0s broker killed t+1s the pump notices; nothing more is handed over at any QoS t+0..∞ every invocation appends to the buffer and returns as fast as ever; once the budget is reached, drop_policy decides what goes t+7min broker returns; the next invocation reconnects, drains the buffer oldest-first discarding anything past max_message_age, then publishes the new rows ``` ### Setup ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_publisher/mqtt_publisher.py \ --trigger-spec "table:press_line" \ --trigger-arguments 'broker_url=mqtt://broker:1883,source_measurements=press_line,topic=plant/{machine}/{_table}' \ mqtt_publish_press influxdb3 enable trigger --database mydb mqtt_publish_press ``` ### Examples Source row used throughout: ``` press_line,machine=press7,line=l1 temp=21.5,pressure=4.2,state="run" 1791019630123456789 ``` **Mirror a table to one topic** ```bash --trigger-arguments "broker_url=mqtt://broker:1883,source_measurements=press_line,topic=test4" ``` ``` test4 {"time":"2026-10-03T09:27:10.123456789Z","machine":"press7","line":"l1", "temp":21.5,"pressure":4.2,"state":"run"} ``` **A topic per machine, line protocol** — the format to use when the far side is another InfluxDB, because `mqtt_subscriber` parses it: ```bash --trigger-arguments "broker_url=mqtt://broker:1883,source_measurements=press_line,topic=plant/{machine}/{_table},payload_format=lineprotocol" ``` ``` plant/press7/press_line press_line,machine=press7,line=l1 temp=21.5,pressure=4.2,state="run" 1791019630123456789 ``` **One topic per field, bare values — the Unified Namespace shape** ```bash --trigger-arguments "broker_url=mqtt://broker:1883,source_measurements=press_line,topic=plant/{machine}/{_field},payload_fields=temp pressure,layout=field,retain=true" ``` ``` plant/press7/temp 21.5 plant/press7/pressure 4.2 ``` `retain = true` means a dashboard subscribing an hour later sees the current value at once instead of waiting for the next change. **Publish only the rows above a threshold.** The template contains commas, which the trigger CLI would split, so this one goes in a TOML file: ```toml source_measurements = ["press_line"] filters = [["temp", ">=", 185]] topic = "plant/{machine}/cmd/setpoint" payload_format = "template" payload_template = '{"setpoint": 180, "reason": "temp {temp}"}' qos = 1 ``` ``` plant/press7/cmd/setpoint {"setpoint": 180, "reason": "temp 187.2"} ``` Remember that `filters` selects **rows, not transitions**: while the temperature stays above 185, every row that arrives is published. **Publishing a value another plugin derives.** The trigger is `onwrite`, not`every:…`, and that is not a matter of taste: a rollup row carries the timestamp of the *start* of the bin it summarises and is written once that bin has closed, so a scheduled trigger would find every such row below its watermark. ```bash influxdb3 create trigger --database plant \ --plugin-filename gh:influxdata/mqtt_publisher/mqtt_publisher.py \ --trigger-spec "table:press_line_1m" \ --trigger-arguments "broker_url=mqtt://broker:1883,source_measurements=press_line_1m,topic=plant/{machine}/rollup,payload_fields=temp_avg temp_max" \ publish_rollup ``` **Announcing availability** ```bash --trigger-arguments "broker_url=mqtt://broker:1883,source_measurements=press_line,topic=plant/{machine}/{_table},status_topic=plant/publisher/status" ``` ``` plant/publisher/status online (retained, on connect) plant/publisher/status offline (retained, the will, when the connection dies) ``` ### Parameters **Connection** | Parameter | Default | Example | What it does | |-----------------------|-----------------------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `broker_url` | **required** | `mqtt://broker.example.com:1883` | Broker address and transport in one value: mqtt://, mqtts://, ws://, wss://, unix:///path, or a bare host[:port]. The port defaults by scheme (1883, 8883, 80, 443) and TLS is chosen by the scheme. | | `client_id` |`influxdb3-pub-<8 hex>`| `plant-publisher` |MQTT client identifier, defaulting to influxdb3-pub- and kept across reconnects. The broker allows one session per identifier and evicts the previous holder, so set it only when ACLs require a known one, per trigger rather than in a shared file, and only on a trigger pinned to one node.| | `username` | *(empty)* | `publisher` | User name for the broker. | | `password` | *(empty)* | `` | Password for the broker. Prefer the TOML file or the environment: trigger arguments are readable from the system tables. | | `mqtt_version` | `3.1.1` | `5` | Protocol version: ‘3.1.1’ (default), ‘5’ or ‘3.1’. Version 5 adds reason codes on acknowledgements, message expiry and broker-declared limits; without it a refused message cannot be told from a delivered one. | | `keep_alive` | `300` | `60` | Seconds declared in CONNECT, 300 by default. Large on purpose because nothing pings between invocations, but it also caps a TLS handshake and a WebSocket upgrade, so lower it on an unreliable TLS path. | | `max_silence` | half of `keep_alive` | `150s` | How long the connection may be quiet before an invocation replaces it rather than writing into it blind. Defaults to half of keep\_alive, and follows an MQTT 5 broker’s Server Keep Alive when one is imposed. | | `connect_timeout` | `2s` | `5s` | Bounds the TCP connect and the wait for CONNACK. It does not bound name resolution or a TLS handshake. Defaults to 2s. | | `pump_budget` | `50ms` | `200ms` | How long one write invocation may spend running the network loop collecting acknowledgements. Raise it together with max\_inflight when a QoS 1 trigger buffers against a healthy broker. Defaults to 50ms. | |`reconnect_min_backoff`| `1s` | `2s` | Shortest gap between connection attempts, doubling on each failure. Defaults to 1s. | |`reconnect_max_backoff`| `120s` | `60s` | Ceiling for the backoff doubling, reset once a CONNACK succeeds. Defaults to 120s. | | `tls_ca_cert` | *(empty)* | `certs/ca.pem` | CA certificate for mqtts:// or wss://. A relative path resolves under PLUGIN\_DIR; omit to use the system trust store. | | `tls_cert` | *(empty)* | `certs/client.pem` | Client certificate for mutual TLS. | | `tls_key` | *(empty)* | `certs/client.key` | Client private key for mutual TLS. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | `tls_insecure` | `false` | `true` | Skip server certificate verification. Defaults to false and should stay there outside a laboratory. | | `ws_path` | from `broker_url` | `/mqtt` | Path for a WebSocket connection when it differs from the one in broker\_url. Ignored outside ws:// and wss://. | | `ws_headers` | *(empty)* |`Authorization:'Bearer abc' X-Tenant:'plant-1'`| Extra headers on the WebSocket upgrade, for brokers that authenticate there: space-separated name:‘value’ pairs, or a table of name to value in a TOML file or a request body. Ignored outside ws:// and wss://. | **Availability** | Parameter | Default | Example | What it does | |------------------------|---------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `status_topic` |*(empty)*|`plant/publisher/status`|Retained availability topic. The online payload is published on every connect and the offline one is registered as the MQTT will. Meaningful only when invocations are more frequent than keep\_alive; the HTTP trigger never touches it.| |`status_online_payload` |`online` | `up` | Payload published retained on status\_topic after every successful connect. Defaults to ‘online’. | |`status_offline_payload`|`offline`| `down` | Payload registered as the MQTT will, so the broker emits it when the connection dies unexpectedly. Defaults to ‘offline’. | **Source** | Parameter | Default | Example | What it does | |---------------------|------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurements`|**required**|`press_line weld_line`|Tables to publish, space-separated or as a list. On a write trigger, batches for any other table are ignored, so the trigger specification may safely be all\_tables. On a schedule each is read in turn; on HTTP the first is the default table a request may override.| **Row selection** | Parameter | Default | Example | What it does | |--------------|---------|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `filters` |*(empty)*| `temp:'>=185' state:'!=error'` | Comparisons applied before any message is built, with =, !=, \>, \<, \>= and \<=. As a string, space-separated column:‘operator value’ pairs with each value quoted; in a TOML file or a request body, also a list of [column, operator, value] triples. A row lacking the column fails a positive test and passes ‘!=’. | |`allow_values`|*(empty)*|`machine:'press7 press8' line:'l1'`|Membership tests: as a string, space-separated column:‘values’ pairs with each list quoted; in a TOML file or a request body, also a table of column to values. Only rows whose value is listed are published, and a column not named here is unconstrained. Values are matched as strings, so for a numeric or boolean column use filters instead.| |`deny_values` |*(empty)*| `machine:'press9' line:'l9'` | Same forms as allow\_values. Rows whose value is listed are skipped, a deny match wins over an allow match, and a row lacking the column is published. | **Message** | Parameter | Default | Example | What it does | |---------------------|------------|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `topic` |**required**| `plant/{machine}/{_table}` | Topic to publish to. A plain string is valid; {column} is replaced by that column’s value in the row, {\_table} by the table name and {\_field} by the field name under layout=field. A substituted value may not contain /, + or #. | |`topic_missing_value`| *(unset)* | `unknown` | Substitute for a placeholder whose column is empty on a row. Unset, such a row is skipped and counted. | | `payload_format` | `json` | `lineprotocol` | Encoder: ‘json’ (default), ’lineprotocol’ or ’template’. Ignored under layout=field, whose payload is the bare value. | | `payload_fields` | *(empty)* | `temp pressure` | Columns that reach the payload, space-separated or as a list; empty means every field and tag of the row. It narrows the payload only, not what a topic placeholder may refer to. | | `payload_template` | *(empty)* |`{"setpoint": 180, "reason": "temp {temp}"}`| Required when payload\_format=template. Uses the same placeholders as topic; a brace group that is not a placeholder stays literal, and {{ }} produce literal braces. | | `layout` | `row` | `field` | ‘row’ (default) publishes one message per row, ‘batch’ one message per topic per invocation, ‘field’ one message per field carrying the bare value. | | `time_format` | `rfc3339` | `ms` |How a timestamp is rendered in a json payload and in a {time} placeholder, which payload\_template accepts and topic refuses: ‘rfc3339’ (default), ‘ms’ or ’ns’. ’ns’ exceeds the largest integer JavaScript represents exactly. Ignored by lineprotocol, whose grammar requires integer nanoseconds.| | `qos` | `0` | `1` | Delivery guarantee asked of the broker: 0 (default), 1 or 2. QoS 2 is not recommended, being not exactly-once under a clean session and hiding a refusal at PUBREC. | | `retain` | `false` | `true` | Set the MQTT retain flag, so a subscriber arriving later sees the current value at once. Defaults to false. | **Limits and buffering** | Parameter | Default | Example | What it does | |-----------------------|-------------|-------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `max_messages` | `10000` | `2000` |Source rows one query reads at a time, which is what LIMIT receives: the page size of a scheduled tick and of a backfill call. The number of messages follows from layout. It does not apply to a write trigger, whose batch is encoded whole and bounded by max\_queue\_bytes instead. Defaults to 10000.| | `max_payload_bytes` | `122880` | `262144` | Largest message built, counting topic and properties. A batch is split to stay under it; a single row that still does not fit is dropped, counted and logged. Defaults to 122880, eight bytes short of AWS IoT Core’s cap. | |`max_delivery_attempts`| `3` | `5` | Hand-overs before a message is treated as undeliverable, then dropped, counted and logged. An attempt is charged only to the oldest unconfirmed message. Defaults to 3. | | `max_queue_bytes` | `67108864` | `16777216` | Bound on the in-memory buffer, counted as payload plus topic plus a per-message overhead so that it tracks real memory. Per trigger, and every trigger shares one engine process. Defaults to 67108864. | | `drop_policy` |`drop_oldest`|`drop_newest`| What happens at max\_queue\_bytes: ‘drop\_oldest’ (default) keeps the freshest data, ‘drop\_newest’ preserves the earliest events. | | `max_message_age` | `0` | `5min` | Discard a message whose row is older than this, measured from the row’s timestamp and applied on both the direct and the buffered path. On a schedule it also clamps the watermark and must exceed settle\_delay. A backfill is exempt. Defaults to 0, which never discards. | |`max_unconfirmed_bytes`| `16777216` | `4194304` | Back-pressure threshold, as the total size of messages handed over and not yet confirmed. Defaults to 16777216. | | `max_inflight` | `20` | `100` | How many QoS \>= 1 messages may await acknowledgement at once. Nothing past it is handed over, because paho would hold it outside every policy here. Defaults to 20. | **Common** | Parameter | Default | Example | What it does | |---------------------|------------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `config_file_path` | *(empty)* |`mqtt_publisher_config.toml`|TOML file under PLUGIN\_DIR holding any of these settings. Read after the environment and the trigger arguments and winning over both; the place for values containing commas and for credentials.| | `stats_interval` | `1min` | `5min` | How often counters are written to mqtt\_publisher\_stats and summarised in the log. Defaults to 1min; 0 writes no measurement. | | `instance_name` |hash of source and topic| `press-line` | Tag on mqtt\_publisher\_stats telling several triggers on one broker apart. Defaults to a stable hash of source\_measurements and topic. | |`enable_full_logging`| `false` | `true` | When false (the default), an exception’s own message is withheld so that broker URLs and paths cannot reach the log. Reason codes, errno and the setting an error names are logged either way. | ## Scheduled trigger Publishes what each source table has gained since the previous tick. Create it **without** `--run-asynchronous`: a watermark describes one reader, and two ticks overlapping on a cold cache publish the same first page twice. ### How it behaves **There is no lookback setting**, because a fixed window is not idempotent: five minutes of lookback on a one-minute schedule publishes every row five times, and duplicates that would collapse inside a table are five real MQTT messages. A **watermark per source table** defines the range instead: * bounded below by `time > watermark` and above by the invocation’s start time less `settle_delay`; * a **whole page** is read into the buffer, and the watermark follows it only once that page has drained — no further page is read while anything is pending; * on a cold cache the watermark starts at *now − `settle_delay`*, so the first tick publishes nothing and the second publishes what appeared since. **Why a page at a time.** Handing over is capped by the in-flight window, so a tick can hand over fewer rows than it read. If the watermark followed the last row handed over and those rows fell inside a group sharing one timestamp — 300 machines on aligned seconds produce exactly that — the watermark could not enter the group without abandoning the rest, nor stay below it without re-reading the same rows next tick. The group would republish for ever. **Why a page always ends on a timestamp boundary.** The cursor is a timestamp, so a page stopping inside a group would advance past rows that were never published. A page that is entirely one timestamp is re-read whole, up to`max_group_rows`. **`settle_delay` exists because the watermark moves along the data’s timestamps, not along the order rows were written.** Two lags push a row’s visibility past its timestamp: * **The server’s own write-to-visibility lag.** A freshly written row is not queryable until the next WAL flush (up to `--wal-flush-interval`, default 1s). A tick whose window reaches `now` reads nothing yet, advances the watermark past that time, and the row is lost once it flushes. This hits **every**writer, which is why `settle_delay` defaults to **2s** — above the default flush interval. Keep it at or above the server’s `--wal-flush-interval`. * **A source that batches.** It writes a row late relative to the time it carries, and once the watermark has passed that time the row is never published: ``` 12:00:05 tick publishes rows with time ≤ 12:00:05, watermark = 12:00:05 12:00:08 Telegraf flushes a batch timestamped 12:00:00 … 12:00:07 everything in it at or below 12:00:05 is already behind — lost ``` Telegraf’s default `flush_interval` is 10 s. A table **another plugin derives**is late by that plugin’s whole interval: a minute rollup carries the timestamp of the start of the bin and can only be written once the bin has closed, so no few-second `settle_delay` ever covers it. Use a write trigger on that table, which has no watermark at all. **The database is the buffer.** While the client cannot take anything, the tick reads nothing and leaves the watermark where it is, so an outage costs memory nothing and loses nothing: ``` t+0s broker killed t+1min the tick connects, fails, logs it and reads nothing; the watermark stays where it was t+1..7m the same each tick; the buffer stays empty and memory is flat t+7min broker returns; each tick publishes up to max_messages per page, oldest first, until it has caught up ``` **A tick works through as many pages as `tick_budget` allows**, not one. One page per tick is a ceiling, not a pace: a source writing 18 000 rows a minute against a `max_messages` of 10 000 never catches up on an `every:1min`schedule. A tick that stops with a full page still waiting logs a warning. **Filters are compiled into the query**, so `max_messages` counts rows that will actually be published rather than rows scanned. ### Setup ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_publisher/mqtt_publisher.py \ --trigger-spec "every:1min" \ --trigger-arguments 'broker_url=mqtt://broker:1883,source_measurements=press_line,topic=plant/{machine}/sampled,settle_delay=15s' \ mqtt_publish_sampled influxdb3 enable trigger --database mydb mqtt_publish_sampled ``` ### Example A minute’s worth of readings, coalesced into one message per machine, with fifteen seconds of tolerance for a batching source: ```toml source_measurements = ["press_line"] topic = "plant/{machine}/sampled" settle_delay = "15s" layout = "batch" payload_fields = ["temp", "pressure"] max_message_age = "10min" qos = 1 ``` ``` plant/press7/sampled [{"time":"…:10.123456789Z","temp":21.5,"pressure":4.2}, {"time":"…:11.004000000Z","temp":21.6,"pressure":4.2}] ``` `max_message_age` must exceed `settle_delay`, and should exceed`settle_delay` plus the schedule interval — otherwise each tick skips the part of its window the next tick’s clamp has already passed. The plugin refuses the first; the second it can only log, because the trigger specification never reaches it. ### Parameters **Connection** | Parameter | Default | Example | What it does | |-----------------------|-----------------------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `broker_url` | **required** | `mqtt://broker.example.com:1883` | Broker address and transport in one value: mqtt://, mqtts://, ws://, wss://, unix:///path, or a bare host[:port]. The port defaults by scheme (1883, 8883, 80, 443) and TLS is chosen by the scheme. | | `client_id` |`influxdb3-pub-<8 hex>`| `plant-publisher` |MQTT client identifier, defaulting to influxdb3-pub- and kept across reconnects. The broker allows one session per identifier and evicts the previous holder, so set it only when ACLs require a known one, per trigger rather than in a shared file, and only on a trigger pinned to one node.| | `username` | *(empty)* | `publisher` | User name for the broker. | | `password` | *(empty)* | `` | Password for the broker. Prefer the TOML file or the environment: trigger arguments are readable from the system tables. | | `mqtt_version` | `3.1.1` | `5` | Protocol version: ‘3.1.1’ (default), ‘5’ or ‘3.1’. Version 5 adds reason codes on acknowledgements, message expiry and broker-declared limits; without it a refused message cannot be told from a delivered one. | | `keep_alive` | `300` | `60` | Seconds declared in CONNECT, 300 by default. Large on purpose because nothing pings between invocations, but it also caps a TLS handshake and a WebSocket upgrade, so lower it on an unreliable TLS path. | | `max_silence` | half of `keep_alive` | `150s` | How long the connection may be quiet before an invocation replaces it rather than writing into it blind. Defaults to half of keep\_alive, and follows an MQTT 5 broker’s Server Keep Alive when one is imposed. | | `connect_timeout` | `2s` | `5s` | Bounds the TCP connect and the wait for CONNACK. It does not bound name resolution or a TLS handshake. Defaults to 2s. | |`reconnect_min_backoff`| `1s` | `2s` | Shortest gap between connection attempts, doubling on each failure. Defaults to 1s. | |`reconnect_max_backoff`| `120s` | `60s` | Ceiling for the backoff doubling, reset once a CONNACK succeeds. Defaults to 120s. | | `tls_ca_cert` | *(empty)* | `certs/ca.pem` | CA certificate for mqtts:// or wss://. A relative path resolves under PLUGIN\_DIR; omit to use the system trust store. | | `tls_cert` | *(empty)* | `certs/client.pem` | Client certificate for mutual TLS. | | `tls_key` | *(empty)* | `certs/client.key` | Client private key for mutual TLS. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | `tls_insecure` | `false` | `true` | Skip server certificate verification. Defaults to false and should stay there outside a laboratory. | | `ws_path` | from `broker_url` | `/mqtt` | Path for a WebSocket connection when it differs from the one in broker\_url. Ignored outside ws:// and wss://. | | `ws_headers` | *(empty)* |`Authorization:'Bearer abc' X-Tenant:'plant-1'`| Extra headers on the WebSocket upgrade, for brokers that authenticate there: space-separated name:‘value’ pairs, or a table of name to value in a TOML file or a request body. Ignored outside ws:// and wss://. | **Availability** | Parameter | Default | Example | What it does | |------------------------|---------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `status_topic` |*(empty)*|`plant/publisher/status`|Retained availability topic. The online payload is published on every connect and the offline one is registered as the MQTT will. Meaningful only when invocations are more frequent than keep\_alive; the HTTP trigger never touches it.| |`status_online_payload` |`online` | `up` | Payload published retained on status\_topic after every successful connect. Defaults to ‘online’. | |`status_offline_payload`|`offline`| `down` | Payload registered as the MQTT will, so the broker emits it when the connection dies unexpectedly. Defaults to ‘offline’. | **Source** | Parameter | Default | Example | What it does | |---------------------|------------|----------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurements`|**required**|`press_line weld_line`| Tables to publish, space-separated or as a list. On a write trigger, batches for any other table are ignored, so the trigger specification may safely be all\_tables. On a schedule each is read in turn; on HTTP the first is the default table a request may override. | | `settle_delay` | `2s` | `30s` |Hold back the upper bound of the range, so rows still below the server’s write-to-visibility lag (up to `--wal-flush-interval`, default 1s) are not skipped and rows a batching source writes late are still above the watermark when they appear. Keep it at or above `--wal-flush-interval`; a table another plugin derives needs that plugin’s whole interval plus its offset and run time. Defaults to 2s.| | `tick_budget` | `5s` | `20s` | How long one tick may keep working through pages before leaving the rest to the next one. One page per tick cannot catch up with a source writing more than max\_messages per interval. Defaults to 5s. | | `max_group_rows` | `100000` | `500000` | Most rows read for a single timestamp when a whole page carries one, since a page cannot stop inside a timestamp group without losing the rest of it. Beyond it the surplus is dropped, counted and logged, and the cursor advances anyway. Defaults to 100000. | **Row selection** | Parameter | Default | Example | What it does | |--------------|---------|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `filters` |*(empty)*| `temp:'>=185' state:'!=error'` | Comparisons applied before any message is built, with =, !=, \>, \<, \>= and \<=. As a string, space-separated column:‘operator value’ pairs with each value quoted; in a TOML file or a request body, also a list of [column, operator, value] triples. A row lacking the column fails a positive test and passes ‘!=’. | |`allow_values`|*(empty)*|`machine:'press7 press8' line:'l1'`|Membership tests: as a string, space-separated column:‘values’ pairs with each list quoted; in a TOML file or a request body, also a table of column to values. Only rows whose value is listed are published, and a column not named here is unconstrained. Values are matched as strings, so for a numeric or boolean column use filters instead.| |`deny_values` |*(empty)*| `machine:'press9' line:'l9'` | Same forms as allow\_values. Rows whose value is listed are skipped, a deny match wins over an allow match, and a row lacking the column is published. | **Message** | Parameter | Default | Example | What it does | |---------------------|------------|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `topic` |**required**| `plant/{machine}/{_table}` | Topic to publish to. A plain string is valid; {column} is replaced by that column’s value in the row, {\_table} by the table name and {\_field} by the field name under layout=field. A substituted value may not contain /, + or #. | |`topic_missing_value`| *(unset)* | `unknown` | Substitute for a placeholder whose column is empty on a row. Unset, such a row is skipped and counted. | | `payload_format` | `json` | `lineprotocol` | Encoder: ‘json’ (default), ’lineprotocol’ or ’template’. Ignored under layout=field, whose payload is the bare value. | | `payload_fields` | *(empty)* | `temp pressure` | Columns that reach the payload, space-separated or as a list; empty means every field and tag of the row. It narrows the payload only, not what a topic placeholder may refer to. | | `payload_template` | *(empty)* |`{"setpoint": 180, "reason": "temp {temp}"}`| Required when payload\_format=template. Uses the same placeholders as topic; a brace group that is not a placeholder stays literal, and {{ }} produce literal braces. | | `layout` | `row` | `field` | ‘row’ (default) publishes one message per row, ‘batch’ one message per topic per invocation, ‘field’ one message per field carrying the bare value. | | `time_format` | `rfc3339` | `ms` |How a timestamp is rendered in a json payload and in a {time} placeholder, which payload\_template accepts and topic refuses: ‘rfc3339’ (default), ‘ms’ or ’ns’. ’ns’ exceeds the largest integer JavaScript represents exactly. Ignored by lineprotocol, whose grammar requires integer nanoseconds.| | `qos` | `0` | `1` | Delivery guarantee asked of the broker: 0 (default), 1 or 2. QoS 2 is not recommended, being not exactly-once under a clean session and hiding a refusal at PUBREC. | | `retain` | `false` | `true` | Set the MQTT retain flag, so a subscriber arriving later sees the current value at once. Defaults to false. | **Limits and buffering** | Parameter | Default | Example | What it does | |-----------------------|-------------|-------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `max_messages` | `10000` | `2000` |Source rows one query reads at a time, which is what LIMIT receives: the page size of a scheduled tick and of a backfill call. The number of messages follows from layout. It does not apply to a write trigger, whose batch is encoded whole and bounded by max\_queue\_bytes instead. Defaults to 10000.| | `max_payload_bytes` | `122880` | `262144` | Largest message built, counting topic and properties. A batch is split to stay under it; a single row that still does not fit is dropped, counted and logged. Defaults to 122880, eight bytes short of AWS IoT Core’s cap. | |`max_delivery_attempts`| `3` | `5` | Hand-overs before a message is treated as undeliverable, then dropped, counted and logged. An attempt is charged only to the oldest unconfirmed message. Defaults to 3. | | `max_queue_bytes` | `67108864` | `16777216` | Bound on the in-memory buffer, counted as payload plus topic plus a per-message overhead so that it tracks real memory. Per trigger, and every trigger shares one engine process. Defaults to 67108864. | | `drop_policy` |`drop_oldest`|`drop_newest`| What happens at max\_queue\_bytes: ‘drop\_oldest’ (default) keeps the freshest data, ‘drop\_newest’ preserves the earliest events. | | `max_message_age` | `0` | `5min` | Discard a message whose row is older than this, measured from the row’s timestamp and applied on both the direct and the buffered path. On a schedule it also clamps the watermark and must exceed settle\_delay. A backfill is exempt. Defaults to 0, which never discards. | |`max_unconfirmed_bytes`| `16777216` | `4194304` | Back-pressure threshold, as the total size of messages handed over and not yet confirmed. Defaults to 16777216. | | `max_inflight` | `20` | `100` | How many QoS \>= 1 messages may await acknowledgement at once. Nothing past it is handed over, because paho would hold it outside every policy here. Defaults to 20. | **Common** | Parameter | Default | Example | What it does | |---------------------|------------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `config_file_path` | *(empty)* |`mqtt_publisher_config.toml`|TOML file under PLUGIN\_DIR holding any of these settings. Read after the environment and the trigger arguments and winning over both; the place for values containing commas and for credentials.| | `stats_interval` | `1min` | `5min` | How often counters are written to mqtt\_publisher\_stats and summarised in the log. Defaults to 1min; 0 writes no measurement. | | `instance_name` |hash of source and topic| `press-line` | Tag on mqtt\_publisher\_stats telling several triggers on one broker apart. Defaults to a stable hash of source\_measurements and topic. | |`enable_full_logging`| `false` | `true` | When false (the default), an exception’s own message is withheld so that broker URLs and paths cannot reach the log. Reason codes, errno and the setting an error names are logged either way. | ## HTTP trigger Replays a time range on demand. One call handles one table and **never touches the scheduled watermark**, so a backfill and a live scheduled trigger can run against the same table without interfering. ### How it behaves **The range is `[start, end)`** — `backfill_start` inclusive, `backfill_end`exclusive — so adjacent calls chain without republishing the row on the seam. Both may be omitted, and then the whole table is published, up to`max_backfill_messages`. **The call works through the range in pages**, waiting between them for the unconfirmed total to fall below `max_unconfirmed_bytes`, so a large range cannot outrun the broker and overflow the buffer. **The answer carries two timestamps**, and they are not the same thing: * `reached` — the last **fully confirmed** timestamp group. A group of which some messages are confirmed and some are not cannot be resumed from, so it does not count; * `resume_from` — what to pass next time, already adjusted for the direction and the inclusivity of the bound it goes into, so nobody has to do nanosecond arithmetic on an RFC 3339 string. **`max_backfill_messages` caps the whole call**, bounding each page by the budget left so it holds per call, not only between pages, at timestamp-group granularity. Reaching it with rows still to publish is an ordinary stop, not an error: the answer says `complete: false` and carries a `resume_from`. Reaching it with the range already exhausted answers `complete: true`, so an exact hit on the last page costs no spurious resume. That cap is what makes a request with no range safe to issue by accident. **A long backfill outlasts its HTTP response.** The plugin finishes regardless and logs progress — there is no way to stop it short of removing the trigger, which is why the cap’s default is low enough to notice. **This trigger ignores `status_topic`.** Its client exists for one request, and its coming and going says nothing about availability. ### Setup ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_publisher/mqtt_publisher.py \ --trigger-spec "request:mqtt_backfill" \ --trigger-arguments 'broker_url=mqtt://broker:1883,source_measurements=press_line,topic=replay/{machine}/{_table}' \ mqtt_backfill influxdb3 enable trigger --database mydb mqtt_backfill ``` ### Making a request ```bash curl -X POST "http://localhost:8181/api/v3/engine/mqtt_backfill" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{"backfill_start": "2026-10-05T00:00:00Z", "backfill_end": "2026-10-05T06:00:00Z"}' ``` ```json { "status": "ok", "table": "press_line", "published": 184320, "confirmed": 184320, "dropped": 0, "reached": "2026-10-05T05:59:59.000000000Z", "resume_from": "2026-10-05T05:59:59.000000001Z", "complete": true } ``` A run that stops with rows still to publish — interrupted, or capped with the range not yet exhausted — answers `"status": "aborted"` with `"complete": false`; pass its `resume_from` as `backfill_start` to continue from exactly where it stopped: ```bash curl -X POST "http://localhost:8181/api/v3/engine/mqtt_backfill" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" -H 'Content-Type: application/json' \ -d '{"backfill_start": "2026-10-05T05:59:59.000000001Z"}' ``` **Replaying into a different shape.** A request may change which rows are taken, where they go and what the message says — which is what a replay for a new consumer needs: ```bash curl -X POST "http://localhost:8181/api/v3/engine/mqtt_backfill?backfill_order=newest_first" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" -H 'Content-Type: application/json' \ -H 'X-Influxdb3-Mqtt-Publisher-Topic: replay/{machine}/{_table}' \ -d '{ "source_measurements": "weld_line", "backfill_start": "2026-10-05T00:00:00Z", "payload_fields": ["temp", "pressure"], "time_format": "ms", "filters": [["temp", ">=", 185]] }' ``` ### What a request may set Only these, in the body, a header or a query parameter: `backfill_start`, `backfill_end`, `backfill_order`, `source_measurements`,`filters`, `allow_values`, `deny_values`, `topic`, `topic_missing_value`,`payload_format`, `payload_template`, `payload_fields`, `layout`,`time_format`. Which rows to take, where they go and what the message says stay open because that is what a backfill varies. The broker, the credentials, the QoS, the TLS material and every limit belong to the trigger, and **a request naming one is refused** — the connection is the trigger’s identity, and a caller who could supply `broker_url` would be using the plugin to reach a broker of their own. ### Parameters **Connection** | Parameter | Default | Example | What it does | |-----------------------|-----------------------|-----------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `broker_url` | **required** | `mqtt://broker.example.com:1883` | Broker address and transport in one value: mqtt://, mqtts://, ws://, wss://, unix:///path, or a bare host[:port]. The port defaults by scheme (1883, 8883, 80, 443) and TLS is chosen by the scheme. | | `client_id` |`influxdb3-pub-<8 hex>`| `plant-publisher` |MQTT client identifier, defaulting to influxdb3-pub- and kept across reconnects. The broker allows one session per identifier and evicts the previous holder, so set it only when ACLs require a known one, per trigger rather than in a shared file, and only on a trigger pinned to one node.| | `username` | *(empty)* | `publisher` | User name for the broker. | | `password` | *(empty)* | `` | Password for the broker. Prefer the TOML file or the environment: trigger arguments are readable from the system tables. | | `mqtt_version` | `3.1.1` | `5` | Protocol version: ‘3.1.1’ (default), ‘5’ or ‘3.1’. Version 5 adds reason codes on acknowledgements, message expiry and broker-declared limits; without it a refused message cannot be told from a delivered one. | | `keep_alive` | `300` | `60` | Seconds declared in CONNECT, 300 by default. Large on purpose because nothing pings between invocations, but it also caps a TLS handshake and a WebSocket upgrade, so lower it on an unreliable TLS path. | | `max_silence` | half of `keep_alive` | `150s` | How long the connection may be quiet before an invocation replaces it rather than writing into it blind. Defaults to half of keep\_alive, and follows an MQTT 5 broker’s Server Keep Alive when one is imposed. | | `connect_timeout` | `2s` | `5s` | Bounds the TCP connect and the wait for CONNACK. It does not bound name resolution or a TLS handshake. Defaults to 2s. | |`reconnect_min_backoff`| `1s` | `2s` | Shortest gap between connection attempts, doubling on each failure. Defaults to 1s. | |`reconnect_max_backoff`| `120s` | `60s` | Ceiling for the backoff doubling, reset once a CONNACK succeeds. Defaults to 120s. | | `tls_ca_cert` | *(empty)* | `certs/ca.pem` | CA certificate for mqtts:// or wss://. A relative path resolves under PLUGIN\_DIR; omit to use the system trust store. | | `tls_cert` | *(empty)* | `certs/client.pem` | Client certificate for mutual TLS. | | `tls_key` | *(empty)* | `certs/client.key` | Client private key for mutual TLS. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | `tls_insecure` | `false` | `true` | Skip server certificate verification. Defaults to false and should stay there outside a laboratory. | | `ws_path` | from `broker_url` | `/mqtt` | Path for a WebSocket connection when it differs from the one in broker\_url. Ignored outside ws:// and wss://. | | `ws_headers` | *(empty)* |`Authorization:'Bearer abc' X-Tenant:'plant-1'`| Extra headers on the WebSocket upgrade, for brokers that authenticate there: space-separated name:‘value’ pairs, or a table of name to value in a TOML file or a request body. Ignored outside ws:// and wss://. | **Source** | Parameter | Default | Example | What it does | |---------------------|------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`source_measurements`|**required**|`press_line weld_line`|Tables to publish, space-separated or as a list. On a write trigger, batches for any other table are ignored, so the trigger specification may safely be all\_tables. On a schedule each is read in turn; on HTTP the first is the default table a request may override.| | `max_group_rows` | `100000` | `500000` | Most rows read for a single timestamp when a whole page carries one, since a page cannot stop inside a timestamp group without losing the rest of it. Beyond it the surplus is dropped, counted and logged, and the cursor advances anyway. Defaults to 100000. | **Row selection** | Parameter | Default | Example | What it does | |--------------|---------|-----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `filters` |*(empty)*| `temp:'>=185' state:'!=error'` | Comparisons applied before any message is built, with =, !=, \>, \<, \>= and \<=. As a string, space-separated column:‘operator value’ pairs with each value quoted; in a TOML file or a request body, also a list of [column, operator, value] triples. A row lacking the column fails a positive test and passes ‘!=’. | |`allow_values`|*(empty)*|`machine:'press7 press8' line:'l1'`|Membership tests: as a string, space-separated column:‘values’ pairs with each list quoted; in a TOML file or a request body, also a table of column to values. Only rows whose value is listed are published, and a column not named here is unconstrained. Values are matched as strings, so for a numeric or boolean column use filters instead.| |`deny_values` |*(empty)*| `machine:'press9' line:'l9'` | Same forms as allow\_values. Rows whose value is listed are skipped, a deny match wins over an allow match, and a row lacking the column is published. | **Message** | Parameter | Default | Example | What it does | |---------------------|------------|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `topic` |**required**| `plant/{machine}/{_table}` | Topic to publish to. A plain string is valid; {column} is replaced by that column’s value in the row, {\_table} by the table name and {\_field} by the field name under layout=field. A substituted value may not contain /, + or #. | |`topic_missing_value`| *(unset)* | `unknown` | Substitute for a placeholder whose column is empty on a row. Unset, such a row is skipped and counted. | | `payload_format` | `json` | `lineprotocol` | Encoder: ‘json’ (default), ’lineprotocol’ or ’template’. Ignored under layout=field, whose payload is the bare value. | | `payload_fields` | *(empty)* | `temp pressure` | Columns that reach the payload, space-separated or as a list; empty means every field and tag of the row. It narrows the payload only, not what a topic placeholder may refer to. | | `payload_template` | *(empty)* |`{"setpoint": 180, "reason": "temp {temp}"}`| Required when payload\_format=template. Uses the same placeholders as topic; a brace group that is not a placeholder stays literal, and {{ }} produce literal braces. | | `layout` | `row` | `field` | ‘row’ (default) publishes one message per row, ‘batch’ one message per topic per invocation, ‘field’ one message per field carrying the bare value. | | `time_format` | `rfc3339` | `ms` |How a timestamp is rendered in a json payload and in a {time} placeholder, which payload\_template accepts and topic refuses: ‘rfc3339’ (default), ‘ms’ or ’ns’. ’ns’ exceeds the largest integer JavaScript represents exactly. Ignored by lineprotocol, whose grammar requires integer nanoseconds.| | `qos` | `0` | `1` | Delivery guarantee asked of the broker: 0 (default), 1 or 2. QoS 2 is not recommended, being not exactly-once under a clean session and hiding a refusal at PUBREC. | | `retain` | `false` | `true` | Set the MQTT retain flag, so a subscriber arriving later sees the current value at once. Defaults to false. | **Limits and buffering** | Parameter | Default | Example | What it does | |-----------------------|-------------|-------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `max_messages` | `10000` | `2000` |Source rows one query reads at a time, which is what LIMIT receives: the page size of a scheduled tick and of a backfill call. The number of messages follows from layout. It does not apply to a write trigger, whose batch is encoded whole and bounded by max\_queue\_bytes instead. Defaults to 10000.| | `max_payload_bytes` | `122880` | `262144` | Largest message built, counting topic and properties. A batch is split to stay under it; a single row that still does not fit is dropped, counted and logged. Defaults to 122880, eight bytes short of AWS IoT Core’s cap. | |`max_delivery_attempts`| `3` | `5` | Hand-overs before a message is treated as undeliverable, then dropped, counted and logged. An attempt is charged only to the oldest unconfirmed message. Defaults to 3. | | `max_queue_bytes` | `67108864` | `16777216` | Bound on the in-memory buffer, counted as payload plus topic plus a per-message overhead so that it tracks real memory. Per trigger, and every trigger shares one engine process. Defaults to 67108864. | | `drop_policy` |`drop_oldest`|`drop_newest`| What happens at max\_queue\_bytes: ‘drop\_oldest’ (default) keeps the freshest data, ‘drop\_newest’ preserves the earliest events. | | `max_message_age` | `0` | `5min` | Discard a message whose row is older than this, measured from the row’s timestamp and applied on both the direct and the buffered path. On a schedule it also clamps the watermark and must exceed settle\_delay. A backfill is exempt. Defaults to 0, which never discards. | |`max_unconfirmed_bytes`| `16777216` | `4194304` | Back-pressure threshold, as the total size of messages handed over and not yet confirmed. Defaults to 16777216. | | `max_inflight` | `20` | `100` | How many QoS \>= 1 messages may await acknowledgement at once. Nothing past it is handed over, because paho would hold it outside every policy here. Defaults to 20. | **Backfill range** | Parameter | Default | Example | What it does | |-----------------------|--------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `backfill_start` | *(empty)* |`2026-10-05T00:00:00Z`| Inclusive lower bound of the range. Omitted, the range starts at the beginning of the table. | | `backfill_end` | *(empty)* |`2026-10-05T06:00:00Z`| Exclusive upper bound of the range, so adjacent calls chain without republishing the row on the seam. Omitted, the moment the request arrives. | | `backfill_order` |`oldest_first`| `newest_first` | ‘oldest\_first’ (default) or ’newest\_first’. | |`max_backfill_messages`| `1000000` | `50000` |Most messages one call publishes, honored at timestamp-group granularity (a group is published whole even if it overshoots). Reached with rows still remaining, the call answers complete=false with a resume\_from; reached with the range exhausted, it answers complete=true. Defaults to 1000000; 0 removes the cap. Not settable from a request.| | `publish_timeout` | `5s` | `30s` | How long the call waits at the end for its last page to be confirmed before answering. Defaults to 5s. | **Common** | Parameter | Default | Example | What it does | |---------------------|------------------------|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `config_file_path` | *(empty)* |`mqtt_publisher_config.toml`|TOML file under PLUGIN\_DIR holding any of these settings. Read after the environment and the trigger arguments and winning over both; the place for values containing commas and for credentials.| | `stats_interval` | `1min` | `5min` | How often counters are written to mqtt\_publisher\_stats and summarised in the log. Defaults to 1min; 0 writes no measurement. | | `instance_name` |hash of source and topic| `press-line` | Tag on mqtt\_publisher\_stats telling several triggers on one broker apart. Defaults to a stable hash of source\_measurements and topic. | |`enable_full_logging`| `false` | `true` | When false (the default), an exception’s own message is withheld so that broker URLs and paths cannot reach the log. Reason codes, errno and the setting an error names are logged either way. | ## Topics A plain string is valid and is the common case: ``` topic = test4 ``` ### A topic is not something that exists Nothing has to be created in advance, and the plugin creates nothing. In MQTT a topic is just a string in the PUBLISH packet: the broker matches it against whatever subscriptions exist at that moment and delivers it to those. If nobody is subscribed, the message is **discarded** and that is not an error — under MQTT 5 the broker says so with reason code `0x10` *no matching subscribers*, which this plugin counts as delivered, because it is. Three things do constrain which topics you may use: * **The broker’s ACLs.** Publishing to a topic the client is not allowed is refused, which is authorisation rather than existence. Under MQTT 5 it comes back as `0x87` and the message is dropped and logged; under 3.1.1 Mosquitto acknowledges it and discards it silently, and nothing here can tell. * **`retain = true` makes a topic persist.** The broker stores the last retained message per topic and hands it to every later subscriber. That is the point of the UNS pattern and of `status_topic` — and it is also the reason a topic template must produce a **bounded** set of topics. One topic per machine is a few hundred; one per timestamp would be one per row, stored for ever. That is why `{time}` is refused in a topic and offered only in the payload. * **Broker-specific rules.** Managed brokers cap topic length and depth — AWS IoT Core allows seven forward slashes and 256 bytes — and reserve the `$`prefix for their own topics, which this plugin refuses outright. A placeholder is `{`, a name matching `[A-Za-z_][A-Za-z0-9_]*`, and `}`. The template is scanned for exactly that; every other character, braces included, is literal, so a JSON-shaped template needs no escaping. `{{` and `}}` produce literal braces. ``` topic = plant/{machine}/line/{line}/{_table}/{_field} ``` |Placeholder| Resolves to | |-----------|------------------------------------------------------------------------------------------------------------| |`{column}` |The **value** of that tag or field in the row — `{machine}` on a row tagged `machine=press7` yields `press7`| |`{_table}` | The measurement name | |`{_field}` | The field name; meaningful only under `layout = field` | | `{time}` | The row’s timestamp, rendered by `time_format`. **`payload_template` only** — refused in a topic | Rules that apply only when a placeholder is used: * **Every placeholder must name a column of at least one source table.** The check happens when the schema is first read; requiring it in every table would make a trigger over several unlike tables impossible. A table that does not exist yet cannot be checked at all — the normal order for a write trigger — so that is one warning, and the check waits for the table. * **An empty value is a runtime condition, not an error.** In InfluxDB a tag may legitimately be absent from part of a table. Such a row uses`topic_missing_value`; unset, the row is skipped and counted. * **A substituted value containing `/`, `+`, `#` or a null character is refused.** `/` is refused because a value is data, and data must not be able to add a topic level: a tag value of `../cmd/setpoint` would otherwise steer the message into somebody else’s command topic. * **A topic that can never be published is refused with the settings.** Empty, starting with `$` (the broker’s own namespace) or carrying an empty level — when no placeholder can change that, it is a configuration error rather than a line in the log for every row. A value that renders into one of those shapes is still refused and counted per row. A column whose name falls outside the grammar — InfluxDB allows spaces and dots in column names — cannot be interpolated, and there is no escape syntax.`payload_fields` still reaches it; a topic that needs it needs it renamed. ## Payloads |`payload_format`| Payload | |----------------|---------------------------------------------------------------------------------------------| | `json` | A JSON object per row, or an array under `layout = batch` | | `lineprotocol` |InfluxDB line protocol — what `mqtt_subscriber` already parses, so the two form a closed loop| | `template` | `payload_template`, interpolated by the grammar above | |`layout`| Behaviour |Telegraf equivalent| |--------|---------------------------------------------------------------------------------------------------------------------------------------------|-------------------| | `row` | One message per row | `non-batch` | |`batch` | One message per topic per invocation, carrying all its rows | `batch` | |`field` |One message per field, the payload being the bare value. `{_field}` is used where the topic places it and appended as a final level otherwise| `field` | **A payload carries the timestamp first and the remaining columns in name order**, in JSON and in line protocol alike. A write trigger receives a row in the engine’s order and a query returns it in the statement’s, so without one order the same row would encode two ways depending on which trigger published it — a difference that matters to anyone comparing or hashing payloads. The two interact in two places that would otherwise be guesswork: * **`layout = field` ignores `payload_format`.** The payload is the value and nothing else, so there is no object to shape and no template to fill. Setting one is not an error, but it has no effect. * **`template` under `batch` renders once per row and joins with a newline**, the same shape `lineprotocol` takes in a batch. A JSON-shaped template then produces NDJSON, which a consumer can actually read. **A column empty on a row does not appear in that row’s payload, and a row left with no selected values is skipped.** One rule settles two things: a write row simply has no such key while a query returns every column and fills the gaps with `None`, so without it the same logical row would be published as`{"temp":21.5}` by one trigger and `{"temp":21.5,"pressure":null}` by another — and a line protocol payload must carry at least one field. **`lineprotocol` carries types, taken from the catalog**: `i` for int64, `u`for uint64, `t`/`f` for boolean, quotes for a string. A bare number in line protocol is a *float*, so an integer published without its suffix would be re-ingested as a float and collide with the column it came from. A Python value cannot tell int64 from uint64 — both are `int` — which is why the schema is read rather than guessed. Escaping follows the line protocol rules exactly. `payload_fields` narrows the **payload only**. A topic placeholder still resolves against the whole row, so `topic=plant/{machine}` works with`payload_fields=temp`. ## Row selection Two mechanisms, deliberately non-overlapping: * **`filters`** — scalar comparisons on column values. It selects **rows, not transitions**: every row satisfying the condition is published, for as long as it keeps satisfying it. Publishing on the crossing instead is what`state_change` and `threshold_deadman_checks` are for — they write an event row, and this plugin publishes that table. * **`allow_values` / `deny_values`** — membership tests, meant for **string tags**. The list values are matched as strings, so on a numeric or boolean column a write trigger (`str(185.0) == "185.0"`) and a scheduled or backfill trigger (SQL `IN ('185')`, coerced by the engine) can disagree. For a numeric or boolean column use `filters` (`temp:'>=185'`), which is typed and matches the same rows on every trigger type. A row that does **not carry the column** is treated differently by each, and the difference is the one people get wrong: | |Row without the column| Why | |----------------------------------------|----------------------|--------------------------------------------------------------| |`filters` with `=`, `>`, `<`, `>=`, `<=`| skipped |a positive match cannot be shown for a value that is not there| | `filters` with `!=` | **published** | it cannot equal a value it does not carry | | `allow_values` | skipped | membership in the list is not demonstrable | | `deny_values` | **published** | it cannot match a value it does not carry | The two that publish are the ones SQL gets wrong on its own: `NULL != 'x'` and`NULL NOT IN (…)` are both `NULL` rather than `TRUE`, so a bare `!=` or`NOT IN` drops exactly those rows. Each negative test therefore compiles with its own `IS NULL` arm, and both paths — compiled SQL and per-row Python — select the same rows. Without it, adding one `deny_values` would silently stop publishing every row lacking that tag, which on a table written by more than one source can be most of it. Filter columns and literals are **checked against the schema** when it is first read: a column no source table has, an ordering comparison on a text column, or a number compared against a text column are all refused with both names in the message. Values are always **bound as query parameters**, never written into the statement as text. ## Where settings come from The write and scheduled triggers read three layers; the HTTP trigger reads six: ``` query parameters > headers > request body > TOML file > trigger arguments > environment ``` | Layer | Spelling | Notes | |-----------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------| | environment | `INFLUXDB3_MQTT_PUBLISHER_BROKER_URL` | Any setting, upper-cased behind the prefix | |trigger arguments| `broker_url=mqtt://host:1883` | Any setting. **No value may contain a comma** — the CLI splits on it | | TOML file | `broker_url = "mqtt://host:1883"` | Any setting but `config_file_path` itself; the place for commas and for credentials that should stay out of the trigger catalogue | | request body | `{"topic": "replay/{machine}"}` | HTTP only, and only the settings the HTTP section lists | | headers |`X-Influxdb3-Mqtt-Publisher-Topic: replay/{machine}`|HTTP only; the same settings, matched without regard to case. An unknown header is ignored, because every client sends headers of its own| |query parameters | `?topic=replay/{machine}` | HTTP only; the same settings | An unrecognised name is **refused with the list of accepted ones** rather than ignored, in every layer the operator controls — a trigger argument, a key in the TOML file, a body field or a query parameter. A misspelled setting that silently did nothing would be found much later. `config_file_path` names a layer rather than setting a value, so it is accepted only from the environment and the trigger arguments: neither the file nor a request can point the plugin at another file. ### Two ways to write a setting A trigger argument can only be a string, so every setting has a compact string form. A TOML file and a JSON request body carry real data structures, and four settings accept those too — which reads better and keeps the types you wrote. | Setting | Compact string | Structured | |---------------------|--------------------------------|-------------------------------------------------| | `filters` |`"temp:'>=185' state:'!=error'"`|`[["temp", ">=", 185], ["state", "!=", "error"]]`| | `allow_values` | `"machine:'press7 press8'"` | `{ machine = ["press7", "press8"] }` | | `deny_values` | `"machine:'press9'"` | `{ machine = ["press9"] }` | | `ws_headers` | `"Authorization:'Bearer abc'"` | `{ Authorization = "Bearer abc" }` | |`source_measurements`| `"press_line weld_line"` | `["press_line", "weld_line"]` | | `payload_fields` | `"temp pressure"` | `["temp", "pressure"]` | `filters` also accepts a list of objects — `[{"column": "temp", "operator": ">=", "value": 185}]` — for a caller that generates the request. In the structured form the value keeps the type you wrote, so `185` is a number; in the compact form, and for a quoted `"185"`, it is read as one. **The compact form’s grammar**: pairs are separated by **whitespace outside quotes**, and quotes say which spaces belong to a value. Either quote character may be the outer pair, so a value carrying one kind is written with the other — `state:"=o'brien"`. There is no escape for a quote inside the same kind; a value needing both belongs in the structured form. One separator then serves both levels — the pairs and the lists inside them — and a decimal literal stays intact, which a dot separator could not:`temp:'>=1.5' state:'!=in progress'`. An unbalanced quote is a configuration error naming the setting. ## Installation * **InfluxDB 3 Enterprise**: 3.8.2 or later, with the Processing Engine enabled * **Python packages**: `influxdata-plugin-utils>=0.5.0`, `paho-mqtt>=2.1,<3` > > > One virtual environment serves every plugin on a node, so a requirement > pinned here can still be overridden by one pinned elsewhere. This plugin > needs paho 2.x: it registers `CallbackAPIVersion.VERSION2` and reads reason > codes from `on_publish`, neither of which exists in 1.x. > > 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled: ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install the packages: ``` influxdb3 install package influxdata-plugin-utils influxdb3 install package paho-mqtt ``` ## Observability Counters reach the log as a summary line and the `mqtt_publisher_stats` table in the trigger’s own database, at most once per `stats_interval`. | Field | Meaning | |-------------------------------------------|----------------------------------------------------------------------------------------------------| | `published` | Messages that reached the socket | | `confirmed` | Acknowledged — at QoS 0, written to the socket | | `failed` | Publishes the client refused, and rows whose topic could not be built | | `buffered` | Messages that entered the buffer | | `dropped_policy` | Discarded by `drop_policy` at `max_queue_bytes` | | `dropped_age` | Discarded by `max_message_age` | | `dropped_undeliverable` | Retired after `max_delivery_attempts`, or refused by a reason code | | `dropped_oversized` | A single row that did not fit `max_payload_bytes` | | `lost_disconnect` | QoS 0 messages a dropped connection took with it | | `reconnects` | Clients built after the first | | `skipped_reads` | Scheduled ticks that read nothing because the client could not take it | | `quota_refusals` | Times the broker answered `0x97 quota exceeded` | |`queue_depth`, `queue_bytes`, `unconfirmed`| The buffer and the in-flight window right now | | `client_id` |A **field**, not a tag — it changes on every reload and would otherwise leave a trail of dead series| Rows are tagged with `broker` (`host:port`) and `instance_name`. **“Published” is not a delivery count.** What it proves depends on the QoS: at QoS 0 nothing about the broker having received it, at QoS 1 that the broker accepted it at least once, at QoS 2 that the four-way handshake completed.`PUBCOMP` is **not** a promise of exactly-once here — that needs session state surviving a disconnection, and the plugin connects with a clean session by design. QoS 2 costs two extra round trips and adds nothing to QoS 1. ## Code overview | File | Contents | |----------------------------------------|-------------------------------------------| | `mqtt_publisher.py` | The plugin, with the metadata docstring | |`mqtt_publisher_config_data_writes.toml`| Write-trigger configuration template | | `mqtt_publisher_config_scheduler.toml` | Scheduled-trigger template | | `mqtt_publisher_config_http.toml` | HTTP-trigger template | | `test_mqtt_publisher.py` |Tests with a mocked MQTT client and runtime| | Function | Role | |--------------------------------|--------------------------------------------------------| | `process_writes` | Publishes the rows of a write | | `process_scheduled_call` |Publishes what each table gained since the previous tick| | `process_request` | Backfills a time range on demand | |`ensure_client` / `build_client`| The connection’s life, including when to replace it | | `drain` | Hands the buffer to the client within a time budget | | `build_messages` | Rows into topics and payloads | | `publish_scheduled` | The watermark, paging and the tick budget | | `run_backfill` | The backfill cursor and its answer | ## Troubleshooting ### Nothing arrives, and the log says nothing is wrong On a **scheduled** trigger over a table another plugin writes, check`settle_delay`. A derived row carries the timestamp of the period it describes and lands a whole interval later, so under the default `settle_delay` (2s) it is below the watermark by the time it appears and is never published. Use a write trigger on that table, or set `settle_delay` above the producing plugin’s interval plus its offset and run time. An empty tick is also indistinguishable from a quiet table — the plugin does not warn about it, because telling the two apart would cost an extra query per tick and the right configuration is known in advance from how the source writes. ### On InfluxDB 3 Enterprise.11, a scheduled trigger or a backfill publishes only part of a table If the table has **no tag columns at all**, this is a known server defect rather than the plugin: on the upgraded storage engine — the default for new clusters since 3.11.0 — a query carrying a `time` predicate against a tagless table returns only the rows still in the write buffer, and the ones already flushed to disk are missing with no error. Both query-driven triggers read exactly that way; a **write** trigger is unaffected because it runs no query. Fixed in 3.12.0, where a time-filtered query returns every row of a tagless table. On 3.11.x, add any tag column to the table so its rows stay visible to the query. ### `max_message_age` and `settle_delay` are refused together `max_message_age` must exceed `settle_delay`, or the clamp puts the lower bound of every window above its upper bound and nothing is ever published; with the default `settle_delay` of 2s, a `max_message_age` of 2s or less is refused. It should also exceed `settle_delay` plus the schedule interval, or each tick skips the part of its window the next tick’s clamp has already passed. The plugin cannot check the second, because the trigger specification never reaches it, but the clamp logs every range it skips. ### The scheduled trigger falls further and further behind The source writes more rows per interval than one tick publishes. Raise`tick_budget`, raise `max_messages`, or shorten the schedule. A tick that spends its budget with a full page still waiting says so in the log. ### The retained status says `offline` but the publisher is healthy The trigger is invoked less often than 1.5 × `keep_alive`, so the broker closes the connection between calls and publishes the will. Either leave`status_topic` unset, or arrange for the trigger to fire often enough — for a write trigger that means `all_tables` rather than one sparse table. ### The retained status says `online` after the trigger was deleted Deleting a trigger does not close its client’s socket; the broker notices when its keepalive expires, 1.5 × `keep_alive` after the last packet, and publishes the will then. At the default that is seven and a half minutes. Deleting and recreating releases it in about three seconds instead. ### A connection attempt blocks the trigger for minutes `connect_timeout` bounds the TCP connect and the CONNACK wait, but a TLS handshake and a WebSocket upgrade are bounded by `keep_alive` — paho’s doing, not the plugin’s — and name resolution by the system resolver alone. Against a TLS endpoint that accepts the connection and then goes silent, lower`keep_alive`. ### At QoS 1 a large burst drains slowly One invocation confirms only about `max_inflight` messages per round trip within`pump_budget`, then returns; the rest stay in the in-memory buffer and drain on the next invocation. At the defaults (`max_inflight = 20`, `pump_budget = 50ms`) a single 50 000-row write at `qos = 1` confirmed \~50 on the first invocation and climbed \~50 per later write — nothing is lost, but a one-shot burst on a table written rarely delivers slowly. Raising the two together helps dramatically: the same 50 000 rows at `pump_budget = 2s`, `max_inflight = 200` confirmed \~12 000 in one invocation. So for high QoS 1 volume, raise `max_inflight` and `pump_budget`(and `max_unconfirmed_bytes` with them), watch `all_tables` so any write drains the buffer, or publish at `qos = 0`, where a whole batch goes out in one invocation. Remember the buffer is in memory: an undrained backlog is lost if the server restarts. ### The broker throttles and the plugin does not say why Under `mqtt_version = 5` a quota refusal arrives as `0x97` and is logged, and the invocation stops handing messages over without dead-lettering any of them. MQTT 3.1.1 has no reason codes, so the broker simply closes the connection and it is indistinguishable from any other disconnection — a reason to prefer version 5 against a broker with a published rate limit, AWS IoT Core’s hundred publishes a second per connection being the common one. ### A message the broker’s ACL forbids is counted as delivered Under MQTT 3.1.1 `PUBACK` carries no reason code, and Mosquitto answers a forbidden publish with an ordinary one rather than closing, so the broker discards the message and the plugin cannot know. Under `mqtt_version = 5` the same publish returns `0x87` and is dropped, counted and logged. ### Memory grows while the broker is down That is a write trigger holding what it cannot publish, bounded by`max_queue_bytes`. Lower it, or set `max_message_age` so stale readings are discarded rather than queued. A scheduled trigger does not do this — it stops reading instead. ### Two triggers keep disconnecting each other They share an explicit `client_id`. The broker allows one session per identifier and must evict the previous holder, so give each trigger its own in its own trigger arguments, or point them at separate files with`config_file_path` — a shared TOML file ranks above the trigger arguments and cannot be overridden from them. ### On a cluster every message arrives several times `--node-spec` defaults to `all`, so every process node owns the trigger and runs it: a schedule fires once per owning node on every tick, a write trigger once per owning node on every WAL flush. Pin it with`--node-spec nodes:`. Each node holds its own cache, so its own watermark, buffer and client, and nothing reconciles them. ### A trigger created before its table exists logs a warning That is expected and is the normal order for a write trigger: placeholders and filter types cannot be checked against a schema that does not exist yet. They are checked once the table appears. ## Report an issue For plugin issues, see the[Plugins repository issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Mqtt publisher plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/mqtt_publisher) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | | Write ( table:… , all_tables ) | Scheduled ( every:… , cron:… ) | HTTP ( request:… ) | | --- | --- | --- | --- | | | Write ( table:… , all_tables ) | Scheduled ( every:… , cron:… ) | HTTP ( request:… ) | | Rows come from | the batch being written | a query over a moving window | a query over a range you name | | Fires when | data lands | the schedule says so | a request arrives | | Latency | milliseconds | up to one interval | on demand | | Position is kept by | nothing — the batch is all there is | a watermark per table, in the cache | the request’s own range | | While the broker is down | rows are buffered in memory under max_queue_bytes | nothing is read; the rows stay in the database | the call stops and reports how far it got | | Risk to understand | memory during a long outage | settle_delay versus how late the source writes | a range with no bounds replaying a whole table | | Publishes status_topic | yes | yes | no | | Your source | Trigger | Why | | --- | --- | --- | | Your source | Trigger | Why | | A table your clients write | write | Publishes within milliseconds, and nothing is read back from the database | | A table another plugin derives — a rollup, a forecast, an anomaly score | write | A derived row carries the timestamp of the period it describes and is written well after it, so a schedule would find it below the watermark and never publish it | | A table you want sampled rather than mirrored | scheduled | One pass per tick, coalesced, paced by the schedule | | History, on demand | HTTP | A bounded replay that leaves the watermark alone | | | What follows | | --- | --- | | | What follows | | Nothing runs between invocations | A trigger invoked less often than 1.5 × keep_alive loses its connection between calls and reconnects on the next one — cheaply, but status_topic then reports the invocation rhythm rather than availability | | Acknowledgements are collected only while an invocation runs | At QoS ≥ 1 the confirmed count trails the published one by up to one invocation, and no more than max_inflight messages are outstanding at a time | | A dropped connection is replaced, never repaired | Its unconfirmed messages go back to the head of the buffer, which is what lets a message the broker keeps refusing be retired after max_delivery_attempts | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | broker_url | required | mqtt://broker.example.com:1883 | Broker address and transport in one value: mqtt://, mqtts://, ws://, wss://, unix:///path, or a bare host[:port]. The port defaults by scheme (1883, 8883, 80, 443) and TLS is chosen by the scheme. | | client_id | influxdb3-pub-<8 hex> | plant-publisher | MQTT client identifier, defaulting to influxdb3-pub- and kept across reconnects. The broker allows one session per identifier and evicts the previous holder, so set it only when ACLs require a known one, per trigger rather than in a shared file, and only on a trigger pinned to one node. | | username | (empty) | publisher | User name for the broker. | | password | (empty) | | Password for the broker. Prefer the TOML file or the environment: trigger arguments are readable from the system tables. | | mqtt_version | 3.1.1 | 5 | Protocol version: ‘3.1.1’ (default), ‘5’ or ‘3.1’. Version 5 adds reason codes on acknowledgements, message expiry and broker-declared limits; without it a refused message cannot be told from a delivered one. | | keep_alive | 300 | 60 | Seconds declared in CONNECT, 300 by default. Large on purpose because nothing pings between invocations, but it also caps a TLS handshake and a WebSocket upgrade, so lower it on an unreliable TLS path. | | max_silence | half of keep_alive | 150s | How long the connection may be quiet before an invocation replaces it rather than writing into it blind. Defaults to half of keep_alive, and follows an MQTT 5 broker’s Server Keep Alive when one is imposed. | | connect_timeout | 2s | 5s | Bounds the TCP connect and the wait for CONNACK. It does not bound name resolution or a TLS handshake. Defaults to 2s. | | pump_budget | 50ms | 200ms | How long one write invocation may spend running the network loop collecting acknowledgements. Raise it together with max_inflight when a QoS 1 trigger buffers against a healthy broker. Defaults to 50ms. | | reconnect_min_backoff | 1s | 2s | Shortest gap between connection attempts, doubling on each failure. Defaults to 1s. | | reconnect_max_backoff | 120s | 60s | Ceiling for the backoff doubling, reset once a CONNACK succeeds. Defaults to 120s. | | tls_ca_cert | (empty) | certs/ca.pem | CA certificate for mqtts:// or wss://. A relative path resolves under PLUGIN_DIR; omit to use the system trust store. | | tls_cert | (empty) | certs/client.pem | Client certificate for mutual TLS. | | tls_key | (empty) | certs/client.key | Client private key for mutual TLS. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | tls_insecure | false | true | Skip server certificate verification. Defaults to false and should stay there outside a laboratory. | | ws_path | from broker_url | /mqtt | Path for a WebSocket connection when it differs from the one in broker_url. Ignored outside ws:// and wss://. | | ws_headers | (empty) | Authorization:'Bearer abc' X-Tenant:'plant-1' | Extra headers on the WebSocket upgrade, for brokers that authenticate there: space-separated name:‘value’ pairs, or a table of name to value in a TOML file or a request body. Ignored outside ws:// and wss://. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | status_topic | (empty) | plant/publisher/status | Retained availability topic. The online payload is published on every connect and the offline one is registered as the MQTT will. Meaningful only when invocations are more frequent than keep_alive; the HTTP trigger never touches it. | | status_online_payload | online | up | Payload published retained on status_topic after every successful connect. Defaults to ‘online’. | | status_offline_payload | offline | down | Payload registered as the MQTT will, so the broker emits it when the connection dies unexpectedly. Defaults to ‘offline’. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | source_measurements | required | press_line weld_line | Tables to publish, space-separated or as a list. On a write trigger, batches for any other table are ignored, so the trigger specification may safely be all_tables. On a schedule each is read in turn; on HTTP the first is the default table a request may override. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | filters | (empty) | temp:'>=185' state:'!=error' | Comparisons applied before any message is built, with =, !=, >, <, >= and <=. As a string, space-separated column:‘operator value’ pairs with each value quoted; in a TOML file or a request body, also a list of [column, operator, value] triples. A row lacking the column fails a positive test and passes ‘!=’. | | allow_values | (empty) | machine:'press7 press8' line:'l1' | Membership tests: as a string, space-separated column:‘values’ pairs with each list quoted; in a TOML file or a request body, also a table of column to values. Only rows whose value is listed are published, and a column not named here is unconstrained. Values are matched as strings, so for a numeric or boolean column use filters instead. | | deny_values | (empty) | machine:'press9' line:'l9' | Same forms as allow_values. Rows whose value is listed are skipped, a deny match wins over an allow match, and a row lacking the column is published. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | topic | required | plant/{machine}/{_table} | Topic to publish to. A plain string is valid; {column} is replaced by that column’s value in the row, {_table} by the table name and {_field} by the field name under layout=field. A substituted value may not contain /, + or #. | | topic_missing_value | (unset) | unknown | Substitute for a placeholder whose column is empty on a row. Unset, such a row is skipped and counted. | | payload_format | json | lineprotocol | Encoder: ‘json’ (default), ’lineprotocol’ or ’template’. Ignored under layout=field, whose payload is the bare value. | | payload_fields | (empty) | temp pressure | Columns that reach the payload, space-separated or as a list; empty means every field and tag of the row. It narrows the payload only, not what a topic placeholder may refer to. | | payload_template | (empty) | {"setpoint": 180, "reason": "temp {temp}"} | Required when payload_format=template. Uses the same placeholders as topic; a brace group that is not a placeholder stays literal, and {{ }} produce literal braces. | | layout | row | field | ‘row’ (default) publishes one message per row, ‘batch’ one message per topic per invocation, ‘field’ one message per field carrying the bare value. | | time_format | rfc3339 | ms | How a timestamp is rendered in a json payload and in a {time} placeholder, which payload_template accepts and topic refuses: ‘rfc3339’ (default), ‘ms’ or ’ns’. ’ns’ exceeds the largest integer JavaScript represents exactly. Ignored by lineprotocol, whose grammar requires integer nanoseconds. | | qos | 0 | 1 | Delivery guarantee asked of the broker: 0 (default), 1 or 2. QoS 2 is not recommended, being not exactly-once under a clean session and hiding a refusal at PUBREC. | | retain | false | true | Set the MQTT retain flag, so a subscriber arriving later sees the current value at once. Defaults to false. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | max_messages | 10000 | 2000 | Source rows one query reads at a time, which is what LIMIT receives: the page size of a scheduled tick and of a backfill call. The number of messages follows from layout. It does not apply to a write trigger, whose batch is encoded whole and bounded by max_queue_bytes instead. Defaults to 10000. | | max_payload_bytes | 122880 | 262144 | Largest message built, counting topic and properties. A batch is split to stay under it; a single row that still does not fit is dropped, counted and logged. Defaults to 122880, eight bytes short of AWS IoT Core’s cap. | | max_delivery_attempts | 3 | 5 | Hand-overs before a message is treated as undeliverable, then dropped, counted and logged. An attempt is charged only to the oldest unconfirmed message. Defaults to 3. | | max_queue_bytes | 67108864 | 16777216 | Bound on the in-memory buffer, counted as payload plus topic plus a per-message overhead so that it tracks real memory. Per trigger, and every trigger shares one engine process. Defaults to 67108864. | | drop_policy | drop_oldest | drop_newest | What happens at max_queue_bytes: ‘drop_oldest’ (default) keeps the freshest data, ‘drop_newest’ preserves the earliest events. | | max_message_age | 0 | 5min | Discard a message whose row is older than this, measured from the row’s timestamp and applied on both the direct and the buffered path. On a schedule it also clamps the watermark and must exceed settle_delay. A backfill is exempt. Defaults to 0, which never discards. | | max_unconfirmed_bytes | 16777216 | 4194304 | Back-pressure threshold, as the total size of messages handed over and not yet confirmed. Defaults to 16777216. | | max_inflight | 20 | 100 | How many QoS >= 1 messages may await acknowledgement at once. Nothing past it is handed over, because paho would hold it outside every policy here. Defaults to 20. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | config_file_path | (empty) | mqtt_publisher_config.toml | TOML file under PLUGIN_DIR holding any of these settings. Read after the environment and the trigger arguments and winning over both; the place for values containing commas and for credentials. | | stats_interval | 1min | 5min | How often counters are written to mqtt_publisher_stats and summarised in the log. Defaults to 1min; 0 writes no measurement. | | instance_name | hash of source and topic | press-line | Tag on mqtt_publisher_stats telling several triggers on one broker apart. Defaults to a stable hash of source_measurements and topic. | | enable_full_logging | false | true | When false (the default), an exception’s own message is withheld so that broker URLs and paths cannot reach the log. Reason codes, errno and the setting an error names are logged either way. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | broker_url | required | mqtt://broker.example.com:1883 | Broker address and transport in one value: mqtt://, mqtts://, ws://, wss://, unix:///path, or a bare host[:port]. The port defaults by scheme (1883, 8883, 80, 443) and TLS is chosen by the scheme. | | client_id | influxdb3-pub-<8 hex> | plant-publisher | MQTT client identifier, defaulting to influxdb3-pub- and kept across reconnects. The broker allows one session per identifier and evicts the previous holder, so set it only when ACLs require a known one, per trigger rather than in a shared file, and only on a trigger pinned to one node. | | username | (empty) | publisher | User name for the broker. | | password | (empty) | | Password for the broker. Prefer the TOML file or the environment: trigger arguments are readable from the system tables. | | mqtt_version | 3.1.1 | 5 | Protocol version: ‘3.1.1’ (default), ‘5’ or ‘3.1’. Version 5 adds reason codes on acknowledgements, message expiry and broker-declared limits; without it a refused message cannot be told from a delivered one. | | keep_alive | 300 | 60 | Seconds declared in CONNECT, 300 by default. Large on purpose because nothing pings between invocations, but it also caps a TLS handshake and a WebSocket upgrade, so lower it on an unreliable TLS path. | | max_silence | half of keep_alive | 150s | How long the connection may be quiet before an invocation replaces it rather than writing into it blind. Defaults to half of keep_alive, and follows an MQTT 5 broker’s Server Keep Alive when one is imposed. | | connect_timeout | 2s | 5s | Bounds the TCP connect and the wait for CONNACK. It does not bound name resolution or a TLS handshake. Defaults to 2s. | | reconnect_min_backoff | 1s | 2s | Shortest gap between connection attempts, doubling on each failure. Defaults to 1s. | | reconnect_max_backoff | 120s | 60s | Ceiling for the backoff doubling, reset once a CONNACK succeeds. Defaults to 120s. | | tls_ca_cert | (empty) | certs/ca.pem | CA certificate for mqtts:// or wss://. A relative path resolves under PLUGIN_DIR; omit to use the system trust store. | | tls_cert | (empty) | certs/client.pem | Client certificate for mutual TLS. | | tls_key | (empty) | certs/client.key | Client private key for mutual TLS. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | tls_insecure | false | true | Skip server certificate verification. Defaults to false and should stay there outside a laboratory. | | ws_path | from broker_url | /mqtt | Path for a WebSocket connection when it differs from the one in broker_url. Ignored outside ws:// and wss://. | | ws_headers | (empty) | Authorization:'Bearer abc' X-Tenant:'plant-1' | Extra headers on the WebSocket upgrade, for brokers that authenticate there: space-separated name:‘value’ pairs, or a table of name to value in a TOML file or a request body. Ignored outside ws:// and wss://. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | status_topic | (empty) | plant/publisher/status | Retained availability topic. The online payload is published on every connect and the offline one is registered as the MQTT will. Meaningful only when invocations are more frequent than keep_alive; the HTTP trigger never touches it. | | status_online_payload | online | up | Payload published retained on status_topic after every successful connect. Defaults to ‘online’. | | status_offline_payload | offline | down | Payload registered as the MQTT will, so the broker emits it when the connection dies unexpectedly. Defaults to ‘offline’. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | source_measurements | required | press_line weld_line | Tables to publish, space-separated or as a list. On a write trigger, batches for any other table are ignored, so the trigger specification may safely be all_tables. On a schedule each is read in turn; on HTTP the first is the default table a request may override. | | settle_delay | 2s | 30s | Hold back the upper bound of the range, so rows still below the server’s write-to-visibility lag (up to --wal-flush-interval , default 1s) are not skipped and rows a batching source writes late are still above the watermark when they appear. Keep it at or above --wal-flush-interval ; a table another plugin derives needs that plugin’s whole interval plus its offset and run time. Defaults to 2s. | | tick_budget | 5s | 20s | How long one tick may keep working through pages before leaving the rest to the next one. One page per tick cannot catch up with a source writing more than max_messages per interval. Defaults to 5s. | | max_group_rows | 100000 | 500000 | Most rows read for a single timestamp when a whole page carries one, since a page cannot stop inside a timestamp group without losing the rest of it. Beyond it the surplus is dropped, counted and logged, and the cursor advances anyway. Defaults to 100000. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | filters | (empty) | temp:'>=185' state:'!=error' | Comparisons applied before any message is built, with =, !=, >, <, >= and <=. As a string, space-separated column:‘operator value’ pairs with each value quoted; in a TOML file or a request body, also a list of [column, operator, value] triples. A row lacking the column fails a positive test and passes ‘!=’. | | allow_values | (empty) | machine:'press7 press8' line:'l1' | Membership tests: as a string, space-separated column:‘values’ pairs with each list quoted; in a TOML file or a request body, also a table of column to values. Only rows whose value is listed are published, and a column not named here is unconstrained. Values are matched as strings, so for a numeric or boolean column use filters instead. | | deny_values | (empty) | machine:'press9' line:'l9' | Same forms as allow_values. Rows whose value is listed are skipped, a deny match wins over an allow match, and a row lacking the column is published. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | topic | required | plant/{machine}/{_table} | Topic to publish to. A plain string is valid; {column} is replaced by that column’s value in the row, {_table} by the table name and {_field} by the field name under layout=field. A substituted value may not contain /, + or #. | | topic_missing_value | (unset) | unknown | Substitute for a placeholder whose column is empty on a row. Unset, such a row is skipped and counted. | | payload_format | json | lineprotocol | Encoder: ‘json’ (default), ’lineprotocol’ or ’template’. Ignored under layout=field, whose payload is the bare value. | | payload_fields | (empty) | temp pressure | Columns that reach the payload, space-separated or as a list; empty means every field and tag of the row. It narrows the payload only, not what a topic placeholder may refer to. | | payload_template | (empty) | {"setpoint": 180, "reason": "temp {temp}"} | Required when payload_format=template. Uses the same placeholders as topic; a brace group that is not a placeholder stays literal, and {{ }} produce literal braces. | | layout | row | field | ‘row’ (default) publishes one message per row, ‘batch’ one message per topic per invocation, ‘field’ one message per field carrying the bare value. | | time_format | rfc3339 | ms | How a timestamp is rendered in a json payload and in a {time} placeholder, which payload_template accepts and topic refuses: ‘rfc3339’ (default), ‘ms’ or ’ns’. ’ns’ exceeds the largest integer JavaScript represents exactly. Ignored by lineprotocol, whose grammar requires integer nanoseconds. | | qos | 0 | 1 | Delivery guarantee asked of the broker: 0 (default), 1 or 2. QoS 2 is not recommended, being not exactly-once under a clean session and hiding a refusal at PUBREC. | | retain | false | true | Set the MQTT retain flag, so a subscriber arriving later sees the current value at once. Defaults to false. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | max_messages | 10000 | 2000 | Source rows one query reads at a time, which is what LIMIT receives: the page size of a scheduled tick and of a backfill call. The number of messages follows from layout. It does not apply to a write trigger, whose batch is encoded whole and bounded by max_queue_bytes instead. Defaults to 10000. | | max_payload_bytes | 122880 | 262144 | Largest message built, counting topic and properties. A batch is split to stay under it; a single row that still does not fit is dropped, counted and logged. Defaults to 122880, eight bytes short of AWS IoT Core’s cap. | | max_delivery_attempts | 3 | 5 | Hand-overs before a message is treated as undeliverable, then dropped, counted and logged. An attempt is charged only to the oldest unconfirmed message. Defaults to 3. | | max_queue_bytes | 67108864 | 16777216 | Bound on the in-memory buffer, counted as payload plus topic plus a per-message overhead so that it tracks real memory. Per trigger, and every trigger shares one engine process. Defaults to 67108864. | | drop_policy | drop_oldest | drop_newest | What happens at max_queue_bytes: ‘drop_oldest’ (default) keeps the freshest data, ‘drop_newest’ preserves the earliest events. | | max_message_age | 0 | 5min | Discard a message whose row is older than this, measured from the row’s timestamp and applied on both the direct and the buffered path. On a schedule it also clamps the watermark and must exceed settle_delay. A backfill is exempt. Defaults to 0, which never discards. | | max_unconfirmed_bytes | 16777216 | 4194304 | Back-pressure threshold, as the total size of messages handed over and not yet confirmed. Defaults to 16777216. | | max_inflight | 20 | 100 | How many QoS >= 1 messages may await acknowledgement at once. Nothing past it is handed over, because paho would hold it outside every policy here. Defaults to 20. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | config_file_path | (empty) | mqtt_publisher_config.toml | TOML file under PLUGIN_DIR holding any of these settings. Read after the environment and the trigger arguments and winning over both; the place for values containing commas and for credentials. | | stats_interval | 1min | 5min | How often counters are written to mqtt_publisher_stats and summarised in the log. Defaults to 1min; 0 writes no measurement. | | instance_name | hash of source and topic | press-line | Tag on mqtt_publisher_stats telling several triggers on one broker apart. Defaults to a stable hash of source_measurements and topic. | | enable_full_logging | false | true | When false (the default), an exception’s own message is withheld so that broker URLs and paths cannot reach the log. Reason codes, errno and the setting an error names are logged either way. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | broker_url | required | mqtt://broker.example.com:1883 | Broker address and transport in one value: mqtt://, mqtts://, ws://, wss://, unix:///path, or a bare host[:port]. The port defaults by scheme (1883, 8883, 80, 443) and TLS is chosen by the scheme. | | client_id | influxdb3-pub-<8 hex> | plant-publisher | MQTT client identifier, defaulting to influxdb3-pub- and kept across reconnects. The broker allows one session per identifier and evicts the previous holder, so set it only when ACLs require a known one, per trigger rather than in a shared file, and only on a trigger pinned to one node. | | username | (empty) | publisher | User name for the broker. | | password | (empty) | | Password for the broker. Prefer the TOML file or the environment: trigger arguments are readable from the system tables. | | mqtt_version | 3.1.1 | 5 | Protocol version: ‘3.1.1’ (default), ‘5’ or ‘3.1’. Version 5 adds reason codes on acknowledgements, message expiry and broker-declared limits; without it a refused message cannot be told from a delivered one. | | keep_alive | 300 | 60 | Seconds declared in CONNECT, 300 by default. Large on purpose because nothing pings between invocations, but it also caps a TLS handshake and a WebSocket upgrade, so lower it on an unreliable TLS path. | | max_silence | half of keep_alive | 150s | How long the connection may be quiet before an invocation replaces it rather than writing into it blind. Defaults to half of keep_alive, and follows an MQTT 5 broker’s Server Keep Alive when one is imposed. | | connect_timeout | 2s | 5s | Bounds the TCP connect and the wait for CONNACK. It does not bound name resolution or a TLS handshake. Defaults to 2s. | | reconnect_min_backoff | 1s | 2s | Shortest gap between connection attempts, doubling on each failure. Defaults to 1s. | | reconnect_max_backoff | 120s | 60s | Ceiling for the backoff doubling, reset once a CONNACK succeeds. Defaults to 120s. | | tls_ca_cert | (empty) | certs/ca.pem | CA certificate for mqtts:// or wss://. A relative path resolves under PLUGIN_DIR; omit to use the system trust store. | | tls_cert | (empty) | certs/client.pem | Client certificate for mutual TLS. | | tls_key | (empty) | certs/client.key | Client private key for mutual TLS. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | tls_insecure | false | true | Skip server certificate verification. Defaults to false and should stay there outside a laboratory. | | ws_path | from broker_url | /mqtt | Path for a WebSocket connection when it differs from the one in broker_url. Ignored outside ws:// and wss://. | | ws_headers | (empty) | Authorization:'Bearer abc' X-Tenant:'plant-1' | Extra headers on the WebSocket upgrade, for brokers that authenticate there: space-separated name:‘value’ pairs, or a table of name to value in a TOML file or a request body. Ignored outside ws:// and wss://. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | source_measurements | required | press_line weld_line | Tables to publish, space-separated or as a list. On a write trigger, batches for any other table are ignored, so the trigger specification may safely be all_tables. On a schedule each is read in turn; on HTTP the first is the default table a request may override. | | max_group_rows | 100000 | 500000 | Most rows read for a single timestamp when a whole page carries one, since a page cannot stop inside a timestamp group without losing the rest of it. Beyond it the surplus is dropped, counted and logged, and the cursor advances anyway. Defaults to 100000. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | filters | (empty) | temp:'>=185' state:'!=error' | Comparisons applied before any message is built, with =, !=, >, <, >= and <=. As a string, space-separated column:‘operator value’ pairs with each value quoted; in a TOML file or a request body, also a list of [column, operator, value] triples. A row lacking the column fails a positive test and passes ‘!=’. | | allow_values | (empty) | machine:'press7 press8' line:'l1' | Membership tests: as a string, space-separated column:‘values’ pairs with each list quoted; in a TOML file or a request body, also a table of column to values. Only rows whose value is listed are published, and a column not named here is unconstrained. Values are matched as strings, so for a numeric or boolean column use filters instead. | | deny_values | (empty) | machine:'press9' line:'l9' | Same forms as allow_values. Rows whose value is listed are skipped, a deny match wins over an allow match, and a row lacking the column is published. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | topic | required | plant/{machine}/{_table} | Topic to publish to. A plain string is valid; {column} is replaced by that column’s value in the row, {_table} by the table name and {_field} by the field name under layout=field. A substituted value may not contain /, + or #. | | topic_missing_value | (unset) | unknown | Substitute for a placeholder whose column is empty on a row. Unset, such a row is skipped and counted. | | payload_format | json | lineprotocol | Encoder: ‘json’ (default), ’lineprotocol’ or ’template’. Ignored under layout=field, whose payload is the bare value. | | payload_fields | (empty) | temp pressure | Columns that reach the payload, space-separated or as a list; empty means every field and tag of the row. It narrows the payload only, not what a topic placeholder may refer to. | | payload_template | (empty) | {"setpoint": 180, "reason": "temp {temp}"} | Required when payload_format=template. Uses the same placeholders as topic; a brace group that is not a placeholder stays literal, and {{ }} produce literal braces. | | layout | row | field | ‘row’ (default) publishes one message per row, ‘batch’ one message per topic per invocation, ‘field’ one message per field carrying the bare value. | | time_format | rfc3339 | ms | How a timestamp is rendered in a json payload and in a {time} placeholder, which payload_template accepts and topic refuses: ‘rfc3339’ (default), ‘ms’ or ’ns’. ’ns’ exceeds the largest integer JavaScript represents exactly. Ignored by lineprotocol, whose grammar requires integer nanoseconds. | | qos | 0 | 1 | Delivery guarantee asked of the broker: 0 (default), 1 or 2. QoS 2 is not recommended, being not exactly-once under a clean session and hiding a refusal at PUBREC. | | retain | false | true | Set the MQTT retain flag, so a subscriber arriving later sees the current value at once. Defaults to false. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | max_messages | 10000 | 2000 | Source rows one query reads at a time, which is what LIMIT receives: the page size of a scheduled tick and of a backfill call. The number of messages follows from layout. It does not apply to a write trigger, whose batch is encoded whole and bounded by max_queue_bytes instead. Defaults to 10000. | | max_payload_bytes | 122880 | 262144 | Largest message built, counting topic and properties. A batch is split to stay under it; a single row that still does not fit is dropped, counted and logged. Defaults to 122880, eight bytes short of AWS IoT Core’s cap. | | max_delivery_attempts | 3 | 5 | Hand-overs before a message is treated as undeliverable, then dropped, counted and logged. An attempt is charged only to the oldest unconfirmed message. Defaults to 3. | | max_queue_bytes | 67108864 | 16777216 | Bound on the in-memory buffer, counted as payload plus topic plus a per-message overhead so that it tracks real memory. Per trigger, and every trigger shares one engine process. Defaults to 67108864. | | drop_policy | drop_oldest | drop_newest | What happens at max_queue_bytes: ‘drop_oldest’ (default) keeps the freshest data, ‘drop_newest’ preserves the earliest events. | | max_message_age | 0 | 5min | Discard a message whose row is older than this, measured from the row’s timestamp and applied on both the direct and the buffered path. On a schedule it also clamps the watermark and must exceed settle_delay. A backfill is exempt. Defaults to 0, which never discards. | | max_unconfirmed_bytes | 16777216 | 4194304 | Back-pressure threshold, as the total size of messages handed over and not yet confirmed. Defaults to 16777216. | | max_inflight | 20 | 100 | How many QoS >= 1 messages may await acknowledgement at once. Nothing past it is handed over, because paho would hold it outside every policy here. Defaults to 20. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | backfill_start | (empty) | 2026-10-05T00:00:00Z | Inclusive lower bound of the range. Omitted, the range starts at the beginning of the table. | | backfill_end | (empty) | 2026-10-05T06:00:00Z | Exclusive upper bound of the range, so adjacent calls chain without republishing the row on the seam. Omitted, the moment the request arrives. | | backfill_order | oldest_first | newest_first | ‘oldest_first’ (default) or ’newest_first’. | | max_backfill_messages | 1000000 | 50000 | Most messages one call publishes, honored at timestamp-group granularity (a group is published whole even if it overshoots). Reached with rows still remaining, the call answers complete=false with a resume_from; reached with the range exhausted, it answers complete=true. Defaults to 1000000; 0 removes the cap. Not settable from a request. | | publish_timeout | 5s | 30s | How long the call waits at the end for its last page to be confirmed before answering. Defaults to 5s. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | config_file_path | (empty) | mqtt_publisher_config.toml | TOML file under PLUGIN_DIR holding any of these settings. Read after the environment and the trigger arguments and winning over both; the place for values containing commas and for credentials. | | stats_interval | 1min | 5min | How often counters are written to mqtt_publisher_stats and summarised in the log. Defaults to 1min; 0 writes no measurement. | | instance_name | hash of source and topic | press-line | Tag on mqtt_publisher_stats telling several triggers on one broker apart. Defaults to a stable hash of source_measurements and topic. | | enable_full_logging | false | true | When false (the default), an exception’s own message is withheld so that broker URLs and paths cannot reach the log. Reason codes, errno and the setting an error names are logged either way. | | Placeholder | Resolves to | | --- | --- | | Placeholder | Resolves to | | {column} | The value of that tag or field in the row — {machine} on a row tagged machine=press7 yields press7 | | {_table} | The measurement name | | {_field} | The field name; meaningful only under layout = field | | {time} | The row’s timestamp, rendered by time_format . payload_template only — refused in a topic | | payload_format | Payload | | --- | --- | | payload_format | Payload | | json | A JSON object per row, or an array under layout = batch | | lineprotocol | InfluxDB line protocol — what mqtt_subscriber already parses, so the two form a closed loop | | template | payload_template , interpolated by the grammar above | | layout | Behaviour | Telegraf equivalent | | --- | --- | --- | | layout | Behaviour | Telegraf equivalent | | row | One message per row | non-batch | | batch | One message per topic per invocation, carrying all its rows | batch | | field | One message per field, the payload being the bare value. {_field} is used where the topic places it and appended as a final level otherwise | field | | | Row without the column | Why | | --- | --- | --- | | | Row without the column | Why | | filters with = , > , < , >= , <= | skipped | a positive match cannot be shown for a value that is not there | | filters with != | published | it cannot equal a value it does not carry | | allow_values | skipped | membership in the list is not demonstrable | | deny_values | published | it cannot match a value it does not carry | | Layer | Spelling | Notes | | --- | --- | --- | | Layer | Spelling | Notes | | environment | INFLUXDB3_MQTT_PUBLISHER_BROKER_URL | Any setting, upper-cased behind the prefix | | trigger arguments | broker_url=mqtt://host:1883 | Any setting. No value may contain a comma — the CLI splits on it | | TOML file | broker_url = "mqtt://host:1883" | Any setting but config_file_path itself; the place for commas and for credentials that should stay out of the trigger catalogue | | request body | {"topic": "replay/{machine}"} | HTTP only, and only the settings the HTTP section lists | | headers | X-Influxdb3-Mqtt-Publisher-Topic: replay/{machine} | HTTP only; the same settings, matched without regard to case. An unknown header is ignored, because every client sends headers of its own | | query parameters | ?topic=replay/{machine} | HTTP only; the same settings | | Setting | Compact string | Structured | | --- | --- | --- | | Setting | Compact string | Structured | | filters | "temp:'>=185' state:'!=error'" | [["temp", ">=", 185], ["state", "!=", "error"]] | | allow_values | "machine:'press7 press8'" | { machine = ["press7", "press8"] } | | deny_values | "machine:'press9'" | { machine = ["press9"] } | | ws_headers | "Authorization:'Bearer abc'" | { Authorization = "Bearer abc" } | | source_measurements | "press_line weld_line" | ["press_line", "weld_line"] | | payload_fields | "temp pressure" | ["temp", "pressure"] | | Field | Meaning | | --- | --- | | Field | Meaning | | published | Messages that reached the socket | | confirmed | Acknowledged — at QoS 0, written to the socket | | failed | Publishes the client refused, and rows whose topic could not be built | | buffered | Messages that entered the buffer | | dropped_policy | Discarded by drop_policy at max_queue_bytes | | dropped_age | Discarded by max_message_age | | dropped_undeliverable | Retired after max_delivery_attempts , or refused by a reason code | | dropped_oversized | A single row that did not fit max_payload_bytes | | lost_disconnect | QoS 0 messages a dropped connection took with it | | reconnects | Clients built after the first | | skipped_reads | Scheduled ticks that read nothing because the client could not take it | | quota_refusals | Times the broker answered 0x97 quota exceeded | | queue_depth , queue_bytes , unconfirmed | The buffer and the in-flight window right now | | client_id | A field , not a tag — it changes on every reload and would otherwise leave a trail of dead series | | File | Contents | | --- | --- | | File | Contents | | mqtt_publisher.py | The plugin, with the metadata docstring | | mqtt_publisher_config_data_writes.toml | Write-trigger configuration template | | mqtt_publisher_config_scheduler.toml | Scheduled-trigger template | | mqtt_publisher_config_http.toml | HTTP-trigger template | | test_mqtt_publisher.py | Tests with a mocked MQTT client and runtime | | Function | Role | | --- | --- | | Function | Role | | process_writes | Publishes the rows of a write | | process_scheduled_call | Publishes what each table gained since the previous tick | | process_request | Backfills a time range on demand | | ensure_client / build_client | The connection’s life, including when to replace it | | drain | Hands the buffer to the client within a time budget | | build_messages | Rows into topics and payloads | | publish_scheduled | The watermark, paging and the tick budget | | run_backfill | The backfill cursor and its answer | --- # Mqtt sparkplug publisher plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/mqtt-sparkplug-publisher/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later, `paho-mqtt>=2.1,<3`, `protobuf>=5` and`influxdata-plugin-utils>=0.5.0`. > > The MQTT Sparkplug B Publisher turns InfluxDB 3 Enterprise into a **Sparkplug B Edge Node**. As rows are written, it publishes them to an MQTT broker as a compliant`spBv1.0` session — Protocol Buffers payloads, birth and death certificates, sequence numbers and aliases — so a SCADA system, Ignition, or any Sparkplug host application consumes the data natively. * **A real Edge Node, not a passthrough**: `NBIRTH` announces the metric set,`NDATA`/`DDATA` carry the values, `NDEATH`/`DDEATH` report loss, `bdSeq` and`seq` keep the session ordered, and aliases keep each message compact * **Devices from your data**: publish at node level, one device per source table, or one device per distinct tag value * **Store-and-forward**: a broker outage buffers rows in memory under a byte budget and a drop policy, then replays them with `is_historical` so old data never overwrites the host’s live value * **Survives across writes**: one background sender owns the connection, the sequence numbers and the buffer, so a slow or down broker never blocks writes * **Rebirth, STATE and takeover handled**: answers a host’s rebirth `NCMD`, gates on a primary host’s `STATE`, and detects an orphan takeover ## Contents * [How it works](#how-it-works) — the Edge Node session and the background sender * [Deployment: one session per node](#deployment-one-session-per-node) — read this first * [Devices and metrics](#devices-and-metrics) — how rows become Sparkplug metrics * [Store-and-forward](#store-and-forward) — the buffer, historical data and draining * [Rebirth, STATE and takeover](#rebirth-state-and-takeover) * [Setup](#setup) · [Examples](#examples) · [Parameters](#parameters) * [Where settings come from](#where-settings-come-from) · [Installation](#installation) * [Observability](#observability) · [Troubleshooting](#troubleshooting) ## How it works ### The Edge Node session The plugin owns one Sparkplug Edge Node, identified by `sparkplug_group_id` and`sparkplug_edge_node_id`. Every topic is `spBv1.0///`, with a device appended for device-level messages: |Message | Topic | When | |--------|------------------------------------------|---------------------------------------------| |`NBIRTH`| `spBv1.0//NBIRTH/` | On connect: declares every node metric | |`NDATA` | `spBv1.0//NDATA/` | Node-level values | |`NDEATH`| `spBv1.0//NDEATH/` |Registered as the MQTT will; also on offline | |`DBIRTH`| `spBv1.0//DBIRTH//` | Declares a device’s metrics | |`DDATA` | `spBv1.0//DDATA//` | Device values | |`DDEATH`| `spBv1.0//DDEATH//` |A device idle past `sparkplug_device_timeout`| | `NCMD` |`spBv1.0//NCMD/` (subscribed)| A host asks for a rebirth | |`STATE` | `spBv1.0/STATE/` (subscribed) | Primary host online/offline | Each connect carries a fresh `bdSeq` (matching the will’s) and starts `NBIRTH`at `seq=0`; every message after it increments `seq`, wrapping at 255. Each metric is declared once in a birth with a name and an **alias**, and after that every `NDATA`/`DDATA` carries only the alias, which keeps messages small. ### The background sender Publishing does not happen on the write path. A write validates the configuration, turns the batch into compact buffered entries, and returns immediately. A **background sender** owns the MQTT connection, the sequence numbers and the buffer: it connects, publishes the births, drains the buffer, answers rebirths and runs the watchdog. A broker that is slow or down therefore never blocks writes into InfluxDB. If no write arrives for `idle_disconnect`, the sender publishes `NDEATH` and shuts the connection down, so a deleted or idle trigger cannot leave an orphan session alive until the server restarts. ## Deployment: one session per node **A Sparkplug Edge Node is a single identity; two live sessions for one`group`/`node` pair fight over the broker.** Deploy accordingly: * **Create the trigger WITHOUT `--run-asynchronous`.** A synchronous trigger ensures exactly one sender is ever built. * **Run exactly one process node for this trigger.** On a multi-node cluster the node specification picks only the owner, but runs are placed round-robin across **every** process node, so two process nodes would stand up two sessions. A single `--mode all` node (the Docker default) or one standalone `--mode process` node is the supported shape. * **Keep `client_id` stable** across trigger recreations (the default already is, derived from the group and node). The broker then evicts an orphan of the same identity by takeover. ## Devices and metrics `sparkplug_device_from` decides how a row maps to a Sparkplug device: |`sparkplug_device_from`| Result | Metric name | Needs | |-----------------------|---------------------------------------------|-------------|------------------------| | *(unset)* | Node-level `NDATA` |`table/field`| — | | `table` | One device per source table | `field` | — | | `tag:` |One device per distinct value of tag ``| `field` |exactly one source table| `payload_fields` is the **declared metric set** of the birth certificate and is mandatory. Each entry is a column, optionally `name:type` to override the datatype inferred from the schema; supported types are `int8/16/32/64`,`uint8/16/32/64`, `float`, `double`, `boolean`, `string` and `datetime`. Signed integers are encoded as two’s-complement in Sparkplug’s unsigned fields. Sparkplug `datetime` is milliseconds since the epoch, but a source column may hold another unit, so a datetime entry may name it: `name:datetime:unit` with`unit` one of `s`, `ms` (default), `us` or `ns` — the value is converted to milliseconds. For example `event_ts:datetime:ns` reads a nanosecond column. A column written but not in `payload_fields` is not published (and is logged once at startup). A tag that is neither the device dimension nor pinned to a single value in `allow_values` is refused at startup, because its series would otherwise collapse onto one metric name. ## Store-and-forward Nothing is handed to a disconnected client. While the broker is unreachable, entries accumulate in the buffer up to `max_queue_bytes`, measured by their in-memory size. At the cap, `drop_policy` decides what goes: `drop_oldest`(default) keeps the freshest data, `drop_newest` the earliest. **Historical data.** A row whose own timestamp is older than `now − live_window`is published with `is_historical = true` rather than as the node’s current value, so a backfill, late data or a WAL replay after a restart never rolls the host’s live value back or re-fires an alarm. `sparkplug_replay_historical = false`discards those rows instead of replaying them. Rows merely delayed by back-pressure are seconds old and stay live. `live_window = 0` turns the test off. **Drain order.** `fifo` (default) drains strictly in order, so after a long outage live data waits behind the backlog. `live_first` publishes recent rows first and drains the historical ones with the leftover rate budget, so current values stay prompt. **Report by exception.** With `report_by_exception = true` a metric is published only when its value changed from the last published value — the Sparkplug-recommended mode for periodically polled sources. `deadband` suppresses a numeric change smaller than itself. After a reconnect the buffered data is replayed on the new session with correct, continuous sequence numbers. ## Rebirth, STATE and takeover * **Rebirth.** A host publishes `Node Control/Rebirth` on `NCMD`; the sender answers with the full `NBIRTH` + `DBIRTH` sequence, coalescing a burst of requests into one. A rebirth is held while a configured primary host is offline, just like the first birth. * **Primary host STATE.** Set `sparkplug_primary_host_id` to subscribe to`spBv1.0/STATE/` and hold the first `NBIRTH` until the host is online. On`offline` the node publishes `NDEATH`, stops cleanly, and the next write rebuilds it — reconnecting and again waiting for the host to come back online. * **Takeover.** With MQTT 5 the `0x8E` reason code is exact; on 3.1.1 a run of quick post-connect closes (`takeover_attempts`) is read as another client taking the identity, after which this client stops and cools down. paho-mqtt 2.1 leaves a reason-only DISCONNECT undecoded (mosquitto sends `e0 01 8e`), so the plugin reads the reason byte itself; if it cannot, MQTT 5 falls back to the 3.1.1 heuristic. ## Setup ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_sparkplug_publisher/mqtt_sparkplug_publisher.py \ --trigger-spec "all_tables" \ --trigger-arguments 'broker_url=mqtt://broker:1883,sparkplug_group_id=plant1,sparkplug_edge_node_id=influxdb3,source_measurements=press_line,payload_fields=temp pressure' \ spb_press influxdb3 enable trigger --database mydb spb_press ``` Batches for any table other than `source_measurements` are ignored, so`all_tables` is a safe trigger specification and keeps a sparse source table from idling the client out. ## Examples Source row used throughout: ``` press_line,machine=press7 temp=21.5,pressure=4.2 1791019630123456789 ``` **Node-level metrics.** No device; metric names are `table/field`: ```bash --trigger-arguments 'broker_url=mqtt://broker:1883,sparkplug_group_id=plant1,sparkplug_edge_node_id=influxdb3,source_measurements=press_line,payload_fields=temp pressure' ``` ``` spBv1.0/plant1/NBIRTH/influxdb3 bdSeq, Node Control/Rebirth, press_line/temp, press_line/pressure spBv1.0/plant1/NDATA/influxdb3 press_line/temp=21.5, press_line/pressure=4.2 (is_historical if old) ``` **A device per machine.** One source table, device from the `machine` tag; metric names are the bare field names: ```bash --trigger-arguments 'broker_url=mqtt://broker:1883,sparkplug_group_id=plant1,sparkplug_edge_node_id=influxdb3,source_measurements=press_line,payload_fields=temp pressure,sparkplug_device_from=tag:machine' ``` ``` spBv1.0/plant1/DBIRTH/influxdb3/press7 temp, pressure spBv1.0/plant1/DDATA/influxdb3/press7 temp=21.5, pressure=4.2 ``` **Publish only changed values, with a deadband.** Typed metric and filters contain commas, so this one goes in a TOML file: ```toml broker_url = "mqtts://broker:8883" sparkplug_group_id = "plant1" sparkplug_edge_node_id = "influxdb3" source_measurements = ["press_line"] payload_fields = ["temp", "pressure", "count:int64"] report_by_exception = true deadband = 0.5 filters = [["temp", ">=", 185]] tls_ca_cert = "certs/ca.pem" ``` ```bash --trigger-arguments 'config_file_path=mqtt_sparkplug_publisher_config.toml' ``` > > > The trigger CLI splits on commas, so any value containing a comma — a typed > metric list, `filters`, `allow_values` — belongs in the TOML file, which is > also the right place for credentials. > > ## Parameters **Required** | Parameter | Default | Example | What it does | |------------------------|------------|--------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `broker_url` |**required**|`mqtt://broker.example.com:1883`| Broker address and transport in one value: `mqtt://`, `mqtts://`, `ws://`, `wss://`, `unix:///path`, or a bare `host[:port]`. Port defaults by scheme. | | `sparkplug_group_id` |**required**| `plant1` | Sparkplug Group ID, the second topic token. No `/`, `+` or `#`. With the edge node id, must be unique across the infrastructure. | |`sparkplug_edge_node_id`|**required**| `influxdb3` | Sparkplug Edge Node ID, the fourth topic token. No `/`, `+` or `#`. | | `source_measurements` |**required**| `press_line weld_line` | Space-separated source tables; exactly one when `sparkplug_device_from` names a tag. Other tables are ignored, so `all_tables` is a safe spec. | | `payload_fields` |**required**| `temp pressure count:int64` |Space-separated metric columns, each optionally `name:type` to override the inferred datatype; a datetime may add a source unit as `name:datetime:unit` (`s`/`ms`/`us`/`ns`, default `ms`), converted to the milliseconds Sparkplug requires. The declared metric set of the birth certificate.| **Connection** | Parameter | Default | Example | What it does | |-----------------------|-----------------------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `client_id` |`influxdb3-spb-<8 hex>`|`plant-spb` | MQTT client identifier, stable by default so the broker evicts an orphan by takeover. Set only when ACLs require a known one; keep it stable. | | `mqtt_version` | `3.1.1` | `5` | `3.1.1` (default) or `5`. MQTT 5 is strongly recommended: it carries the `0x8E` takeover reason code and the broker’s Maximum Packet Size. | | `username` | *(empty)* |`publisher` | User name sent in CONNECT. Never logged. | | `password` | *(empty)* |``| Password. Prefer the TOML file: trigger arguments are readable from the system tables. Never logged. | | `keep_alive` | `60` | `30` | Seconds declared in CONNECT, minimum 1. A dropped link is detected after about 1.5× this; QoS 0 data sent meanwhile is lost but counted published. | |`reconnect_min_backoff`| `1s` | `2s` | Shortest gap between connection attempts, doubling to the ceiling, reset on a successful CONNACK. | |`reconnect_max_backoff`| `120s` | `60s` | Ceiling for the backoff doubling. | | `takeover_attempts` | `3` | `5` |Consecutive quick post-connect closes read as a takeover, after which the client stops. Integer ≥ 2. Used on MQTT 3.1.1; MQTT 5 reads the `0x8E` reason code instead and falls back to this only when it cannot be read.| | `idle_disconnect` | `1h` | `30min` | After this long with no invocation the sender publishes `NDEATH` and exits. Minimum 1min. Use the `all_tables` spec so a sparse table does not idle it out. | **Sparkplug** | Parameter | Default | Example | What it does | |-----------------------------|---------|-------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `sparkplug_device_from` |*(empty)*|`tag:machine`| Empty (node-level `NDATA`, names `table/field`), `table` (per-table device), or `tag:` (one device per tag value, needs one source table). | | `sparkplug_device_timeout` | `0` | `5min` | Publish `DDEATH` for a device idle this long, and `DBIRTH` again when it returns. `0` never does, since an idle device cannot be told from a removed one. | | `sparkplug_primary_host_id` |*(empty)*| `scada1` |Set to subscribe to `spBv1.0/STATE/`, wait for `online` before the first `NBIRTH`, and on `offline` publish `NDEATH`, stop, and rebuild on the next write (awaiting `online` again).| | `live_window` | `5min` | `0` | A row older than `now − live_window` is published `is_historical = true` instead of as the current value. `0` turns the test off (every row live). | |`sparkplug_replay_historical`| `true` | `false` | What happens to rows marked historical by `live_window`: `true` republishes them `is_historical = true`; `false` discards them. Live rows unaffected. | | `report_by_exception` | `false` | `true` | Publish a metric only when its value changed from the last published for that alias. The Sparkplug-recommended mode for periodically polled sources. | | `deadband` | `0` | `0.5` | With `report_by_exception`, a numeric change smaller than this is treated as no change, to suppress analog noise. | **TLS** (for `mqtts://` and `wss://`; a relative path resolves under `PLUGIN_DIR`) | Parameter | Default | Example | What it does | |--------------------------|---------|------------------|-----------------------------------------------------------------------------------------------------| | `tls_ca_cert` |*(empty)*| `certs/ca.pem` | CA certificate; omit to use the system trust store. | | `tls_client_cert` |*(empty)*|`certs/client.pem`| Client certificate for mutual TLS. | | `tls_client_key` |*(empty)*|`certs/client.key`|Client private key. Must be unencrypted; a passphrase-protected key is refused at configuration time.| |`tls_insecure_skip_verify`| `false` | `true` | Skip certificate chain and hostname verification. Leave `false` outside a lab. | | `tls_alpn` |*(empty)*| `x-amzn-mqtt-ca` | ALPN protocol name, how AWS IoT Core serves MQTT on port 443. | **WebSocket** (for `ws://` and `wss://`) | Parameter | Default | Example | What it does | |------------|-----------------|----------------------------|------------------------------------------------------------------------------------| | `ws_path` |from `broker_url`| `/mqtt` | Path for the WebSocket connection. | |`ws_headers`| *(empty)* |`Authorization:'Bearer abc'`|Extra headers on the upgrade: space-separated `name:'value'` pairs, or a TOML table.| **Row selection** | Parameter | Default | Example | What it does | |--------------|---------|------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `filters` |*(empty)*|`temp:'>=185' state:'!=error'`|Per-row comparisons with `=`, `!=`, `>`, `<`, `>=`, `<=`. As a string, space-separated `column:'op value'` pairs; in a file, `[column, op, value]` triples. A numeric value compares numerically; quote it (`'185'`) to compare as text.| |`allow_values`|*(empty)*| `machine:'press7 press8'` | Membership tests, matched as strings. Only listed rows pass. Also the way to pin a non-device tag to one value so several series do not merge onto one metric. | |`deny_values` |*(empty)*| `machine:'press9'` | Rows whose value is listed are skipped; a deny wins over an allow. | **Delivery and buffering** | Parameter | Default | Example | What it does | |-------------------------|-------------|-------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `max_payload_bytes` | `122880` | `262144` |Largest message built, counting topic and properties. A per-device batch is split to stay under it; the birth size is checked before connecting. Integer ≥ 1024. Clamped by an MQTT 5 broker’s Maximum Packet Size.| | `max_queue_bytes` | `67108864` | `16777216` | The only bound on the in-memory buffer, measured by the entries’ in-memory size. Integer ≥ 1024. Per trigger; every trigger shares one engine process. | | `drop_policy` |`drop_oldest`|`drop_newest`| At the buffer cap: `drop_oldest` (default) keeps the freshest data, `drop_newest` the earliest. | | `drain_order` | `fifo` |`live_first` | `fifo` (default) drains in order; `live_first` publishes recent rows first and drains historical ones with the leftover rate budget. | | `max_message_age` | `0` | `10min` | On drain, a buffered entry older than this is discarded instead of published. Duration or `0` (never). | |`max_messages_per_second`| `0` | `500` | Token-bucket cap on how fast the sender hands messages over, and on the `DBIRTH` burst. Integer ≥ 1, or `0` (no limit). | | `max_unconfirmed_bytes` | `16777216` | `4194304` | Back-pressure threshold, as the total size handed over and not yet reported back. Integer ≥ 65536. | **Common** | Parameter | Default | Example | What it does | |---------------------|---------|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `config_file_path` |*(empty)*|`mqtt_sparkplug_publisher_config.toml`|TOML file under `PLUGIN_DIR` holding any of these settings. Read after the environment and the arguments and winning over both; the place for commas and credentials.| | `stats_interval` | `1min` | `5min` | How often counters are written to `mqtt_sparkplug_publisher_stats` and summarised in the log. Duration or `0` (log summary only, no measurement). | |`enable_full_logging`| `false` | `true` | When `false`, an exception is logged as its type only; reason codes and errno are logged either way. Credentials are never logged. | ## Where settings come from Every setting may come from three layers, applied in order, each winning over the one before: 1. **Environment** — `INFLUXDB3_MQTT_SPARKPLUG_PUBLISHER_` (uppercase). 2. **Trigger arguments** — `--trigger-arguments 'key=value,key=value'`. 3. **TOML file** — `config_file_path`, read from `PLUGIN_DIR`. A list is space-separated in an argument (`payload_fields=temp pressure`) and a native array in TOML (`payload_fields = ["temp", "pressure"]`). Because the CLI splits arguments on commas, put any value containing a comma — and all credentials — in the TOML file. See `mqtt_sparkplug_publisher_config.toml` for a documented template. ## Installation * **InfluxDB 3 Enterprise**: 3.8.2 or later, with the Processing Engine enabled * **Python packages**: `influxdata-plugin-utils>=0.5.0`, `paho-mqtt>=2.1,<3`, `protobuf>=5` > > > One virtual environment serves every plugin on a node, so a package pinned by > one plugin affects the others. This plugin requires `paho-mqtt` 2.x and runs > alongside the other MQTT plugins without conflict. Its Sparkplug codec is built > from a descriptor embedded in the plugin rather than a generated module or a > library such as `pysparkplug`, so the plugin ships as a single file and pins no`protobuf` version — it will not downgrade packages the other plugins rely on. > > 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled: ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install the packages: ``` influxdb3 install package influxdata-plugin-utils influxdb3 install package paho-mqtt influxdb3 install package protobuf ``` ## Observability Counters reach the log as a summary line and the `mqtt_sparkplug_publisher_stats`table in the trigger’s own database, at most once per `stats_interval`. | Field | Meaning | |---------------------------------------------|----------------------------------------------------------------| | `published` | Data messages handed to the socket | | `confirmed` | Message ids the broker took (at QoS 0, the socket write) | | `buffered` | Metrics that entered the buffer | | `dropped_policy` | Discarded by `drop_policy` at `max_queue_bytes` | | `dropped_age` | Discarded by `max_message_age` | | `dropped_historical` | Discarded by `sparkplug_replay_historical = false` | | `dropped_unencodable` | A metric that could not be encoded, or alone exceeds the limit | | `dropped_shutdown` |Buffered metrics dropped when the watchdog reaped an idle client| | `lost` | QoS 0 messages a dropped connection took with it | | `reconnects` | Connections after the first | | `rebirths_served` | Rebirth requests answered | | `births` | Birth sequences published | | `command_errors` | `NCMD`/`STATE` messages that failed to decode | | `sender_errors` | Unexpected errors the sender caught and carried on from | |`queue_depth`, `queue_bytes`, `devices_known`| The buffer and the live device set right now | Rows are tagged with `broker` (`host:port`), `group` and `edge_node`. **“Published” is not a delivery count.** At QoS 0 it means the message reached the socket, nothing about the broker having received it. A low `keep_alive`shortens the window in which a silently dropped link counts QoS 0 data as published while it is actually lost. ## Troubleshooting ### The host sees no data, but the counters stay green A broker ACL may be dropping the QoS 0 publishes silently. Subscribe to the node’s own birth topic — `mosquitto_sub -t 'spBv1.0//NBIRTH/'` — to confirm delivery. MQTT 5 surfaces an authorization refusal as a reason code; on 3.1.1 it is silent. ### Two sessions keep evicting each other Two live senders share one `group`/`node` identity. Make sure the trigger runs on exactly **one** process node and was created **without** `--run-asynchronous`(see [Deployment](#deployment-one-session-per-node)). A stable `client_id` lets the broker evict the orphan cleanly rather than flapping. ### Old data overwrites the host’s live value `live_window` is too short or `0`. A backfill or a WAL replay carries old timestamps; with a `live_window` that covers your backfill horizon those rows go out `is_historical = true` and the host keeps its current value. Set`sparkplug_replay_historical = false` to drop them entirely instead. ### A device never disappears after it stops reporting `sparkplug_device_timeout` defaults to `0`, which never publishes `DDEATH`because an idle device cannot be told from a removed one. Set it to a duration longer than the device’s normal reporting gap. ### The client stops and will not reconnect for a while A refused birth (packet too large or not authorized) cools down for five minutes; a detected takeover cools down for ten seconds, before the next invocation rebuilds the client. The log line names which. Fix the underlying cause (payload size, ACL, or the duplicate session) rather than recreating the trigger in a loop. ### A trigger created before its source table exists That is expected. The schema cannot be read until the table is written, so the plugin logs that the table is not found yet and starts without its metrics; they join the birth automatically on the first write to that table, with a rebirth. Other source tables that do exist are unaffected. ### The host still shows an old node after I renamed it Changing `sparkplug_group_id`, `sparkplug_edge_node_id` or `broker_url` publishes`NDEATH` on the old topics before switching, so the host marks the old node dead. Changing only a non-identity setting (password, TLS, keep-alive) keeps the same identity and corrects the host with a fresh `NBIRTH` instead. ### A low `keep_alive` costs more PINGREQ That is the trade-off: a lower `keep_alive` shortens the window in which a silently dropped link loses QoS 0 data counted as published, at the cost of more keep-alive traffic. 15–30s is a reasonable floor on an unreliable link. ## Report an issue For plugin issues, see the[Plugins repository issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Mqtt sparkplug publisher plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/mqtt_sparkplug_publisher) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Message | Topic | When | | --- | --- | --- | | Message | Topic | When | | NBIRTH | spBv1.0//NBIRTH/ | On connect: declares every node metric | | NDATA | spBv1.0//NDATA/ | Node-level values | | NDEATH | spBv1.0//NDEATH/ | Registered as the MQTT will; also on offline | | DBIRTH | spBv1.0//DBIRTH// | Declares a device’s metrics | | DDATA | spBv1.0//DDATA// | Device values | | DDEATH | spBv1.0//DDEATH// | A device idle past sparkplug_device_timeout | | NCMD | spBv1.0//NCMD/ (subscribed) | A host asks for a rebirth | | STATE | spBv1.0/STATE/ (subscribed) | Primary host online/offline | | sparkplug_device_from | Result | Metric name | Needs | | --- | --- | --- | --- | | sparkplug_device_from | Result | Metric name | Needs | | (unset) | Node-level NDATA | table/field | — | | table | One device per source table | field | — | | tag: | One device per distinct value of tag | field | exactly one source table | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | broker_url | required | mqtt://broker.example.com:1883 | Broker address and transport in one value: mqtt:// , mqtts:// , ws:// , wss:// , unix:///path , or a bare host[:port] . Port defaults by scheme. | | sparkplug_group_id | required | plant1 | Sparkplug Group ID, the second topic token. No / , + or # . With the edge node id, must be unique across the infrastructure. | | sparkplug_edge_node_id | required | influxdb3 | Sparkplug Edge Node ID, the fourth topic token. No / , + or # . | | source_measurements | required | press_line weld_line | Space-separated source tables; exactly one when sparkplug_device_from names a tag. Other tables are ignored, so all_tables is a safe spec. | | payload_fields | required | temp pressure count:int64 | Space-separated metric columns, each optionally name:type to override the inferred datatype; a datetime may add a source unit as name:datetime:unit ( s / ms / us / ns , default ms ), converted to the milliseconds Sparkplug requires. The declared metric set of the birth certificate. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | client_id | influxdb3-spb-<8 hex> | plant-spb | MQTT client identifier, stable by default so the broker evicts an orphan by takeover. Set only when ACLs require a known one; keep it stable. | | mqtt_version | 3.1.1 | 5 | 3.1.1 (default) or 5 . MQTT 5 is strongly recommended: it carries the 0x8E takeover reason code and the broker’s Maximum Packet Size. | | username | (empty) | publisher | User name sent in CONNECT. Never logged. | | password | (empty) | | Password. Prefer the TOML file: trigger arguments are readable from the system tables. Never logged. | | keep_alive | 60 | 30 | Seconds declared in CONNECT, minimum 1. A dropped link is detected after about 1.5× this; QoS 0 data sent meanwhile is lost but counted published. | | reconnect_min_backoff | 1s | 2s | Shortest gap between connection attempts, doubling to the ceiling, reset on a successful CONNACK. | | reconnect_max_backoff | 120s | 60s | Ceiling for the backoff doubling. | | takeover_attempts | 3 | 5 | Consecutive quick post-connect closes read as a takeover, after which the client stops. Integer ≥ 2. Used on MQTT 3.1.1; MQTT 5 reads the 0x8E reason code instead and falls back to this only when it cannot be read. | | idle_disconnect | 1h | 30min | After this long with no invocation the sender publishes NDEATH and exits. Minimum 1min. Use the all_tables spec so a sparse table does not idle it out. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | sparkplug_device_from | (empty) | tag:machine | Empty (node-level NDATA , names table/field ), table (per-table device), or tag: (one device per tag value, needs one source table). | | sparkplug_device_timeout | 0 | 5min | Publish DDEATH for a device idle this long, and DBIRTH again when it returns. 0 never does, since an idle device cannot be told from a removed one. | | sparkplug_primary_host_id | (empty) | scada1 | Set to subscribe to spBv1.0/STATE/ , wait for online before the first NBIRTH , and on offline publish NDEATH , stop, and rebuild on the next write (awaiting online again). | | live_window | 5min | 0 | A row older than now − live_window is published is_historical = true instead of as the current value. 0 turns the test off (every row live). | | sparkplug_replay_historical | true | false | What happens to rows marked historical by live_window : true republishes them is_historical = true ; false discards them. Live rows unaffected. | | report_by_exception | false | true | Publish a metric only when its value changed from the last published for that alias. The Sparkplug-recommended mode for periodically polled sources. | | deadband | 0 | 0.5 | With report_by_exception , a numeric change smaller than this is treated as no change, to suppress analog noise. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | tls_ca_cert | (empty) | certs/ca.pem | CA certificate; omit to use the system trust store. | | tls_client_cert | (empty) | certs/client.pem | Client certificate for mutual TLS. | | tls_client_key | (empty) | certs/client.key | Client private key. Must be unencrypted; a passphrase-protected key is refused at configuration time. | | tls_insecure_skip_verify | false | true | Skip certificate chain and hostname verification. Leave false outside a lab. | | tls_alpn | (empty) | x-amzn-mqtt-ca | ALPN protocol name, how AWS IoT Core serves MQTT on port 443. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | ws_path | from broker_url | /mqtt | Path for the WebSocket connection. | | ws_headers | (empty) | Authorization:'Bearer abc' | Extra headers on the upgrade: space-separated name:'value' pairs, or a TOML table. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | filters | (empty) | temp:'>=185' state:'!=error' | Per-row comparisons with = , != , > , < , >= , <= . As a string, space-separated column:'op value' pairs; in a file, [column, op, value] triples. A numeric value compares numerically; quote it ( '185' ) to compare as text. | | allow_values | (empty) | machine:'press7 press8' | Membership tests, matched as strings. Only listed rows pass. Also the way to pin a non-device tag to one value so several series do not merge onto one metric. | | deny_values | (empty) | machine:'press9' | Rows whose value is listed are skipped; a deny wins over an allow. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | max_payload_bytes | 122880 | 262144 | Largest message built, counting topic and properties. A per-device batch is split to stay under it; the birth size is checked before connecting. Integer ≥ 1024. Clamped by an MQTT 5 broker’s Maximum Packet Size. | | max_queue_bytes | 67108864 | 16777216 | The only bound on the in-memory buffer, measured by the entries’ in-memory size. Integer ≥ 1024. Per trigger; every trigger shares one engine process. | | drop_policy | drop_oldest | drop_newest | At the buffer cap: drop_oldest (default) keeps the freshest data, drop_newest the earliest. | | drain_order | fifo | live_first | fifo (default) drains in order; live_first publishes recent rows first and drains historical ones with the leftover rate budget. | | max_message_age | 0 | 10min | On drain, a buffered entry older than this is discarded instead of published. Duration or 0 (never). | | max_messages_per_second | 0 | 500 | Token-bucket cap on how fast the sender hands messages over, and on the DBIRTH burst. Integer ≥ 1, or 0 (no limit). | | max_unconfirmed_bytes | 16777216 | 4194304 | Back-pressure threshold, as the total size handed over and not yet reported back. Integer ≥ 65536. | | Parameter | Default | Example | What it does | | --- | --- | --- | --- | | Parameter | Default | Example | What it does | | config_file_path | (empty) | mqtt_sparkplug_publisher_config.toml | TOML file under PLUGIN_DIR holding any of these settings. Read after the environment and the arguments and winning over both; the place for commas and credentials. | | stats_interval | 1min | 5min | How often counters are written to mqtt_sparkplug_publisher_stats and summarised in the log. Duration or 0 (log summary only, no measurement). | | enable_full_logging | false | true | When false , an exception is logged as its type only; reason codes and errno are logged either way. Credentials are never logged. | | Field | Meaning | | --- | --- | | Field | Meaning | | published | Data messages handed to the socket | | confirmed | Message ids the broker took (at QoS 0, the socket write) | | buffered | Metrics that entered the buffer | | dropped_policy | Discarded by drop_policy at max_queue_bytes | | dropped_age | Discarded by max_message_age | | dropped_historical | Discarded by sparkplug_replay_historical = false | | dropped_unencodable | A metric that could not be encoded, or alone exceeds the limit | | dropped_shutdown | Buffered metrics dropped when the watchdog reaped an idle client | | lost | QoS 0 messages a dropped connection took with it | | reconnects | Connections after the first | | rebirths_served | Rebirth requests answered | | births | Birth sequences published | | command_errors | NCMD / STATE messages that failed to decode | | sender_errors | Unexpected errors the sender caught and carried on from | | queue_depth , queue_bytes , devices_known | The buffer and the live device set right now | --- # Mqtt subscriber plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/mqtt-subscriber/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The MQTT Subscriber Plugin enables real-time ingestion of MQTT messages into InfluxDB 3 Enterprise. Subscribe to MQTT broker topics and automatically transform messages into time-series data with support for JSON, Line Protocol, and custom text formats. The plugin uses persistent MQTT sessions (`clean_session=False`) to support message delivery between executions and provides comprehensive error tracking and statistics. Delivery is at least once: if the broker misses an acknowledgment during disconnect, it can redeliver a message and the plugin can write that message twice. Map `timestamp_field` from the payload to make redeliveries idempotent. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. This plugin supports TOML configuration files for complex mapping scenarios, which can be specified using the `config_file_path` parameter. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters In TOML configuration, `broker_host` and `topics` are placed under the `[mqtt]` section; `table_name` and `table_name_field` are placed under `[mapping.json]` or `[mapping.text]` section. | Parameter | Type | Default | Description | |------------------|------|-------------------------|-------------------------------------------------------------------------------------------------------------------------| | `broker_host` |string| required | MQTT broker hostname or IP address | | `topics` |string| required | Space-separated list of topics (for example, “sensors/temp sensors/humidity”) | | `table_name` |string|required (json/text only)| InfluxDB measurement name for storing data. Not required for `lineprotocol` format or when `table_name_field` is set. | |`table_name_field`|string| none |JSON field name or regex pattern to extract table name dynamically from each message. Alternative to static `table_name`.| ### Connection parameters In TOML configuration, these parameters are placed under the `[mqtt]` section. | Parameter | Type | Default | Description | |-----------------|-------|--------------|----------------------------------------------------------------------------------------| | `broker_port` |integer| 1883 | MQTT broker port (1883 for non-TLS, 8883 for TLS) | | `qos` |integer| 1 | MQTT Quality of Service level (0, 1, or 2) | | `client_id` |string |auto-generated| MQTT client identifier (must be unique per broker) | |`max_queue_bytes`|integer| 67108864 |Maximum total bytes of MQTT payloads buffered between scheduled drains (default: 64 MiB)| **Recommendation:** Use QoS 1 for most IoT scenarios. It provides reliable delivery with minimal overhead. **About `max_queue_bytes`:** Between two scheduled fires, paho’s network thread buffers incoming MQTT messages in an in-memory queue inside the plugin. To protect InfluxDB from a misbehaving or malicious broker, the plugin tracks the running total of buffered payload sizes; once it would exceed `max_queue_bytes`, new messages are dropped (the existing queue is still processed normally). Drop counts are reported per-topic in the `mqtt_stats` table (`messages_dropped` field) and a summary error is logged at the end of each cycle when any drops occurred. Raise this value if you expect high-throughput bursts; lower it on memory-constrained hosts. ### Authentication parameters In TOML configuration, `username` and `password` are placed under the `[mqtt.auth]` section. |Parameter | Type |Default| Description | |----------|------|-------|---------------------------------------------| |`username`|string| none |MQTT broker username (required with password)| |`password`|string| none |MQTT broker password (required with username)| **Note:** Both `username` and `password` must be provided together for authentication. `allow_insecure_auth` is placed directly under the `[mqtt]` section (not `[mqtt.auth]`): | Parameter | Type |Default| Description | |---------------------|-------|-------|---------------------------------------------------------------| |`allow_insecure_auth`|boolean| false |Permit sending `username`/`password` when TLS is not configured| **Security note:** When `username`/`password` are provided without a `ca_cert` (TLS), credentials are transmitted in cleartext over an unencrypted connection. The plugin refuses this by default and raises a configuration error. Configure TLS (`ca_cert`), or explicitly set `allow_insecure_auth = true` to permit cleartext credentials — only do this on a trusted network. ### TLS/SSL parameters In TOML configuration, these parameters are placed under the `[mqtt.tls]` section. | Parameter | Type |Default| Description | |-------------|------|-------|-----------------------------------------| | `ca_cert` |string| none | Path to CA certificate file | |`client_cert`|string| none |Path to client certificate for mutual TLS| |`client_key` |string| none |Path to client private key for mutual TLS| **Note:** For mutual TLS, both `client_cert` and `client_key` must be provided together. ### Logging parameters In TOML configuration, `enable_full_logging` is placed directly under the `[mqtt]` section. | Parameter | Type |Default| Description | |---------------------|-------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`enable_full_logging`|boolean| false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths) into log output. Enable temporarily for debugging.| ### Message format parameters In TOML configuration, `format` is placed under the `[mqtt]` section; `timestamp_field` is placed under `[mapping.json]` or `[mapping.text]` section. | Parameter | Type |Default| Description | |-----------------|------|-------|-------------------------------------------------------------| | `format` |string|“json” | Message format: json, lineprotocol, or text | |`timestamp_field`|string| none |Field containing timestamp (format depends on message format)| **Format-specific timestamp\_field syntax:** The timestamp field format differs between JSON and Text formats: |Format| Syntax | Split Method | Example (CLI) | Example (TOML) | |------|-------------------|------------------|----------------|------------------| | JSON |`field_name:format`|Split by first `:`|`"timestamp:ms"`|`"$.timestamp:ms"`| | Text | `regex:format` |Split by last `:` |`"ts:(\\d+):ms"`| `"ts:(\\d+):ms"` | **Note:** In CLI arguments, JSON paths are specified without `$.` prefix (added automatically). In TOML configuration, use full JSONPath syntax with `$.` prefix. **Note:** Text format uses the last colon to split, allowing regex patterns to contain colons (for example, time patterns). **Supported timestamp formats:** * `ns` - nanoseconds (Unix timestamp) * `ms` - milliseconds (Unix timestamp) * `s` - seconds (Unix timestamp) * `datetime` - ISO 8601 string (for example, “2021-12-01T12:00:00Z”) ### JSON format parameters In TOML configuration, `tags` are placed under `[mapping.json.tags]` section and `fields` under `[mapping.json.fields]` section. |Parameter| Type |Default | Description | |---------|------|--------|-------------------------------------------------------------------------| | `tags` |string| none | Space-separated tag names. Example: “room sensor location” | |`fields` |string|required|Space-separated field mappings. Format: “name:type=jsonpath” without `$.`| **Field specification format:** `"temp:float=temperature hum:int=humidity status:bool=online"` **Supported field types:** `int`, `uint`, `float`, `string`, `bool` **JSONPath restriction:** The regex-based JSONPath operators `=~` (filter regex-match) and `sub()` (regex substitution) are disabled. They run a configured regex against untrusted broker data and are vulnerable to backtracking (ReDoS). A configuration that uses them is rejected at startup. All other JSONPath constructs — nesting, recursive descent (`..`), wildcards (`*`), and comparison filters such as `[?(@.type=="temp")]` — are fully supported. ### Text format parameters In TOML configuration, `tags` are placed under `[mapping.text.tags]` section and `fields` under `[mapping.text.fields]` section. |Parameter| Type |Default | Description | |---------|------|--------|------------------------------------------------------------------| | `tags` |string| none | Space-separated tag patterns. Format: “name=regex\_pattern” | |`fields` |string|required|Space-separated field patterns. Format: “name:type=regex\_pattern”| **Tag specification format:** `"room=room:([^,\\s]+) sensor=sensor:(\\w+)"` **Field specification format:** `"temp:float=temp:([\\d.]+) status:bool=(true|false)"` ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|-----------------------------------------------| |`config_file_path`|string| none |Path to TOML config file (absolute or relative)| *To use a TOML configuration file, specify the `config_file_path` in the trigger arguments.* ### File path resolution All file paths in the plugin (configuration file, TLS certificates) follow the same resolution logic: * **Absolute paths** (for example, `/etc/mqtt/config.toml`) are used as-is * **Relative paths** (for example, `config.toml`, `certs/ca.crt`) are resolved from `PLUGIN_DIR` environment variable If a relative path is specified and `PLUGIN_DIR` is not set, the plugin will return an error. #### Example TOML configuration [mqtt\_config\_example.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/mqtt_subscriber/mqtt_config_example.toml) - comprehensive configuration example with all three message formats ## Data requirements The plugin automatically creates the target measurement table on first write. Field mappings are required for JSON and Text formats to specify which fields to extract and their data types. ### Message encoding requirements * **UTF-8 text only**: The plugin only processes UTF-8 encoded text messages. Binary messages are automatically skipped with a warning logged. * **Non-empty payloads**: Empty or whitespace-only messages are automatically skipped. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: * `paho-mqtt` (MQTT client library) * `jsonpath-ng` (JSON path parsing for JSON format) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package paho-mqtt influxdb3 install package jsonpath-ng ``` ## Trigger setup ### Scheduled ingestion with TOML configuration Recommended for production use with complex mappings: ```bash # 1. Set PLUGIN_DIR environment variable export PLUGIN_DIR=~/.plugins # 2. Copy and edit configuration file cp mqtt_config_example.toml $PLUGIN_DIR/my_mqtt_config.toml # Edit my_mqtt_config.toml with your broker and mapping settings # 3. Create the trigger influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_subscriber/mqtt_subscriber.py \ --trigger-spec "every:10m" \ --trigger-arguments config_file_path=my_mqtt_config.toml \ mqtt_ingestion ``` ### Scheduled ingestion with command-line arguments For simple JSON message ingestion: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_subscriber/mqtt_subscriber.py \ --trigger-spec "every:5m" \ --trigger-arguments 'broker_host=broker.hivemq.com,topics=sensors/temperature sensors/humidity,format=json,table_name=sensor_data,fields=temp:float=temperature hum:int=humidity,tags=location sensor_id' \ mqtt_sensors ``` ### Secure MQTT connection with TLS ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/mqtt_subscriber/mqtt_subscriber.py \ --trigger-spec "every:10m" \ --trigger-arguments 'broker_host=secure-broker.example.com,broker_port=8883,topics=secure/data,format=json,table_name=secure_data,ca_cert=certs/ca.crt,username=myuser,password=mypass,fields=value:float=value' \ secure_mqtt ``` ## MQTT Topic Wildcards The plugin supports standard MQTT wildcard patterns in topic subscriptions: * `+` - Single level wildcard (matches one topic level) * `#` - Multi-level wildcard (matches any number of levels, must be last) **Examples:** * `sensors/+/temperature` - matches `sensors/room1/temperature`, `sensors/room2/temperature` * `sensors/#` - matches `sensors/temp`, `sensors/room1/humidity`, `sensors/building/floor1/co2` * `+/+/data` - matches `home/kitchen/data`, `office/room1/data` **Note:** Wildcards are processed by the MQTT broker, not the plugin. Ensure your broker supports the wildcard patterns you use. ## Message Formats ### JSON Format The primary use case for structured IoT data. Supports nested fields using JSONPath expressions. #### TOML Configuration ```toml [mqtt] broker_host = "broker.hivemq.com" broker_port = 1883 topics = ["sensors/temperature"] qos = 1 format = "json" [mapping.json] table_name = "sensor_data" timestamp_field = "$.timestamp:ms" [mapping.json.tags] location = "$.location" sensor_id = "$.sensor.id" [mapping.json.fields] temperature = ["$.temp", "float"] humidity = ["$.humidity", "int"] status = ["$.online", "bool"] ``` #### Example Message ```json { "timestamp": 1638360000000, "location": "warehouse_a", "sensor": { "id": "sensor_001" }, "temp": 22.5, "humidity": 65, "online": true } ``` #### Resulting Data ``` sensor_data,location=warehouse_a,sensor_id=sensor_001 temperature=22.5,humidity=65i,status=true 1638360000000000000 ``` #### JSON Array Support Process batch messages containing arrays of JSON objects: ```json [ {"timestamp": 1638360000000, "sensor_id": "001", "temperature": 22.5}, {"timestamp": 1638360001000, "sensor_id": "002", "temperature": 23.1}, {"timestamp": 1638360002000, "sensor_id": "003", "temperature": 21.8} ] ``` **Array processing behavior:** * Each array element is processed independently as a separate data point * If one element fails to parse, the others continue processing (partial success) * Parse errors for individual elements are logged to `mqtt_exceptions` table * Statistics count 1 MQTT message = 1 unit (regardless of array size) ### Line Protocol Format For messages already in InfluxDB line protocol format, use passthrough mode. No mapping configuration needed - messages are validated and written directly. #### TOML Configuration ```toml [mqtt] broker_host = "broker.example.com" topics = ["influxdb/metrics"] format = "lineprotocol" ``` #### CLI Configuration ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename mqtt_subscriber.py \ --trigger-spec "every:10m" \ --trigger-arguments 'broker_host=broker.example.com,topics=influxdb/metrics,format=lineprotocol' \ mqtt_lineprotocol ``` #### Example Message ``` sensor_data,location=warehouse_a,sensor_id=001 temperature=22.5,humidity=65i 1638360000000000000 ``` #### Supported Line Protocol Types | Type | Suffix | Example | |----------------|--------------|------------------| | Float | none |`temperature=22.5`| | Integer | `i` | `count=100i` | |Unsigned Integer| `u` | `bytes=1024u` | | String | `"..."` |`status="running"`| | Boolean |`true`/`false`| `active=true` | ### Text Format Parse plain text messages using regular expressions: #### TOML Configuration ```toml [mqtt] format = "text" [mapping.text] table_name = "sensor_logs" timestamp_field = "ts:(\\d+):ms" [mapping.text.tags] location = "location=([^,\\s]+)" [mapping.text.fields] temperature = ["temp:([\\d.]+)", "float"] humidity = ["hum:(\\d+)", "int"] status = ["status:(true|false)", "bool"] ``` #### Example Message ``` location=warehouse_a,temp:22.5,hum:65,status:true,ts:1638360000000 ``` ## Statistics and Monitoring The plugin tracks comprehensive statistics and writes them to the `mqtt_stats` table on every plugin invocation. **Important notes:** * Statistics are written **on every plugin invocation** * Each topic is tracked separately with independent statistics * Statistics persist across plugin restarts using the InfluxDB cache ### mqtt\_stats Table | Field |Type | Description | |--------------------|-----|-------------------------------------------------------------------------------------------------| | `topic` (tag) | tag | MQTT topic name | |`broker_host` (tag) | tag | MQTT broker address | |`messages_received` | int | Total messages received on this topic (includes dropped messages) | |`messages_processed`| int | Successfully processed messages | | `messages_failed` | int | Failed messages | | `messages_dropped` | int | Messages dropped because the `max_queue_bytes` budget was exhausted | | `success_rate` |float|Percentage of successfully processed messages: `processed / (processed + failed + dropped) * 100`| ### Querying Statistics ```bash # Get latest statistics influxdb3 query --database mydb \ "SELECT * FROM mqtt_stats ORDER BY time DESC LIMIT 10" # Success rate over time influxdb3 query --database mydb \ "SELECT topic, success_rate, messages_processed, messages_failed FROM mqtt_stats WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC" ``` ## Error Handling Parse errors and message processing failures are logged to the `mqtt_exceptions` table: ### mqtt\_exceptions Table | Field | Type | Description | |------------------|------|--------------------------------------------| | `topic` (tag) | tag | MQTT topic where error occurred | |`error_type` (tag)| tag |Type of error (for example, JSONDecodeError)| | `error_message` |string| Detailed error message | | `raw_message` |string| Original MQTT message (truncated to 1KB) | ### Checking for Errors ```bash influxdb3 query --database mydb \ "SELECT * FROM mqtt_exceptions ORDER BY time DESC LIMIT 10" ``` ## Troubleshooting ### Check Plugin Logs ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'mqtt_ingestion' ORDER BY time DESC LIMIT 20" ``` ### Common Issues #### “paho-mqtt library not installed” ```bash influxdb3 install package paho-mqtt ``` #### “Configuration file not found” * For relative paths, ensure `PLUGIN_DIR` environment variable is set * For absolute paths, verify the file exists at the specified location ```bash # For relative paths export PLUGIN_DIR=~/.plugins ls $PLUGIN_DIR/my_mqtt_config.toml # Or use absolute path ls /etc/mqtt/my_mqtt_config.toml ``` #### “Failed to connect to MQTT broker” * Verify broker address and port * Check network connectivity * For TLS connections, verify certificate paths * Verify authentication credentials (both username and password required) #### “Both username and password must be provided for authentication” Either provide both `username` and `password`, or omit both for anonymous connection. #### “Refusing to send username/password over an unencrypted connection” `username`/`password` were provided without a `ca_cert`, so the credentials would be sent in cleartext. Configure TLS by providing `ca_cert` (and `client_cert`/`client_key`for mutual TLS), or set `allow_insecure_auth = true` to permit cleartext credentials on a trusted network. #### “Both client\_cert and client\_key must be provided for mutual TLS” For mutual TLS authentication, both client certificate and key are required. #### “No fields were mapped from JSON data” * Verify JSONPath expressions in field mappings (use `$.` prefix) * Check that JSON structure matches your paths * Review `mqtt_exceptions` table for detailed errors #### “Unsupported JSONPath operator ‘=\~’ / ‘sub’” A tag, field, timestamp, or `table_name_field` mapping uses a regex-based JSONPath operator. These are disabled for security (ReDoS). Remove the `=~`filter or `sub()` call — use a comparison filter (`[?(@.x=="value")]`) instead, or extract the raw value and transform it downstream. #### Messages not being processed * Check trigger status: `influxdb3 show summary --database mydb` * Verify MQTT connection in plugin logs * Increase trigger frequency (for example, from `every:5s` to `every:1s`) ## Architecture ### How It Works 1. **Scheduled Trigger**: Plugin runs on schedule (for example, `every:10m`) 2. **Configuration Caching**: Plugin configuration is parsed once and cached between executions 3. **Message Queue**: Incoming MQTT messages are queued in-memory during callback execution 4. **Batch Processing**: Each trigger execution processes all queued messages 5. **Parse & Write**: Messages parsed according to format and written to InfluxDB 6. **Error Tracking**: Parse errors logged to `mqtt_exceptions` table 7. **Statistics**: Written to `mqtt_stats` table on every plugin invocation ### Performance Optimization The plugin includes several optimizations for high-throughput scenarios: * **Configuration Caching**: Plugin configuration is parsed once and reused across all trigger executions * **Pre-compiled Patterns**: JSONPath expressions and regex patterns are compiled once during parser initialization, not per-message * **Persistent Session**: MQTT client uses `clean_session=False` - broker preserves subscriptions and queued messages between connections ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Mqtt subscriber plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/mqtt_subscriber) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | broker_host | string | required | MQTT broker hostname or IP address | | topics | string | required | Space-separated list of topics (for example, “sensors/temp sensors/humidity”) | | table_name | string | required (json/text only) | InfluxDB measurement name for storing data. Not required for lineprotocol format or when table_name_field is set. | | table_name_field | string | none | JSON field name or regex pattern to extract table name dynamically from each message. Alternative to static table_name . | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | broker_port | integer | 1883 | MQTT broker port (1883 for non-TLS, 8883 for TLS) | | qos | integer | 1 | MQTT Quality of Service level (0, 1, or 2) | | client_id | string | auto-generated | MQTT client identifier (must be unique per broker) | | max_queue_bytes | integer | 67108864 | Maximum total bytes of MQTT payloads buffered between scheduled drains (default: 64 MiB) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | username | string | none | MQTT broker username (required with password) | | password | string | none | MQTT broker password (required with username) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | allow_insecure_auth | boolean | false | Permit sending username / password when TLS is not configured | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | ca_cert | string | none | Path to CA certificate file | | client_cert | string | none | Path to client certificate for mutual TLS | | client_key | string | none | Path to client private key for mutual TLS | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths) into log output. Enable temporarily for debugging. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | format | string | “json” | Message format: json, lineprotocol, or text | | timestamp_field | string | none | Field containing timestamp (format depends on message format) | | Format | Syntax | Split Method | Example (CLI) | Example (TOML) | | --- | --- | --- | --- | --- | | Format | Syntax | Split Method | Example (CLI) | Example (TOML) | | JSON | field_name:format | Split by first : | "timestamp:ms" | "$.timestamp:ms" | | Text | regex:format | Split by last : | "ts:(\\d+):ms" | "ts:(\\d+):ms" | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tags | string | none | Space-separated tag names. Example: “room sensor location” | | fields | string | required | Space-separated field mappings. Format: “name:type=jsonpath” without $. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tags | string | none | Space-separated tag patterns. Format: “name=regex_pattern” | | fields | string | required | Space-separated field patterns. Format: “name:type=regex_pattern” | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | Path to TOML config file (absolute or relative) | | Type | Suffix | Example | | --- | --- | --- | | Type | Suffix | Example | | Float | none | temperature=22.5 | | Integer | i | count=100i | | Unsigned Integer | u | bytes=1024u | | String | "..." | status="running" | | Boolean | true / false | active=true | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | topic (tag) | tag | MQTT topic name | | broker_host (tag) | tag | MQTT broker address | | messages_received | int | Total messages received on this topic (includes dropped messages) | | messages_processed | int | Successfully processed messages | | messages_failed | int | Failed messages | | messages_dropped | int | Messages dropped because the max_queue_bytes budget was exhausted | | success_rate | float | Percentage of successfully processed messages: processed / (processed + failed + dropped) * 100 | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | topic (tag) | tag | MQTT topic where error occurred | | error_type (tag) | tag | Type of error (for example, JSONDecodeError) | | error_message | string | Detailed error message | | raw_message | string | Original MQTT message (truncated to 1KB) | --- # Notifier plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/notifier/ The Notifier Plugin provides multi-channel notification capabilities for InfluxDB 3 Enterprise, enabling real-time alert delivery through various communication channels. Send notifications via Slack, Discord, HTTP webhooks, SMS, or WhatsApp based on incoming HTTP requests. Acts as a centralized notification dispatcher that receives data from other plugins or external systems and routes notifications to the appropriate channels. ## Configuration This HTTP plugin receives all configuration via the request body. No trigger arguments are required. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Request body parameters Send these parameters as JSON in the HTTP POST request body: | Parameter | Type |Default | Description | |-------------------|------|--------|-------------------------------------------| |`notification_text`|string|required| Text content of the notification message | | `senders_config` |object|required|Configuration for each notification channel| ### Sender-specific configuration (in request body) The `senders_config` object accepts channel configurations where keys are sender names and values contain channel-specific settings: #### Slack notifications | Parameter | Type |Default | Description | |-------------------|------|--------|---------------------------| |`slack_webhook_url`|string|required| Slack webhook URL | | `slack_headers` |string| none |Base64-encoded JSON headers| #### Discord notifications | Parameter | Type |Default | Description | |---------------------|------|--------|---------------------------| |`discord_webhook_url`|string|required| Discord webhook URL | | `discord_headers` |string| none |Base64-encoded JSON headers| #### HTTP webhook notifications | Parameter | Type |Default | Description | |------------------|------|--------|--------------------------------| |`http_webhook_url`|string|required|Custom webhook URL for HTTP POST| | `http_headers` |string| none | Base64-encoded JSON headers | #### SMS notifications (via Twilio) | Parameter | Type |Default | Description | |--------------------|------|--------|-------------------------------------------------| | `twilio_sid` |string|required|Twilio Account SID (or use `TWILIO_SID` env var) | | `twilio_token` |string|required|Twilio Auth Token (or use `TWILIO_TOKEN` env var)| |`twilio_from_number`|string|required| Sender phone number in E.164 format | | `twilio_to_number` |string|required| Recipient phone number in E.164 format | #### WhatsApp notifications (via Twilio) | Parameter | Type |Default | Description | |--------------------|------|--------|-------------------------------------------------| | `twilio_sid` |string|required|Twilio Account SID (or use `TWILIO_SID` env var) | | `twilio_token` |string|required|Twilio Auth Token (or use `TWILIO_TOKEN` env var)| |`twilio_from_number`|string|required| Sender WhatsApp number in E.164 format | | `twilio_to_number` |string|required| Recipient WhatsApp number in E.164 format | ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Python packages**: * `httpx` (for HTTP requests) * `twilio` (for SMS/WhatsApp notifications) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package httpx influxdb3 install package twilio ``` ## Trigger setup ### HTTP trigger Create an HTTP trigger to handle notification requests: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/notifier/notifier_plugin.py" \ --trigger-spec "request:notify" \ notification_trigger ``` This registers an HTTP endpoint at `/api/v3/engine/notify`. ### Enable trigger ```bash influxdb3 enable trigger --database mydb notification_trigger ``` ## Example usage ### Example 1: Slack notification Send a notification to Slack: ```bash curl -X POST http://localhost:8181/api/v3/engine/notify \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "notification_text": "Alert: High CPU usage detected on server1", "senders_config": { "slack": { "slack_webhook_url": "'"$SLACK_WEBHOOK_URL"'" } } }' ``` Set `INFLUXDB3_AUTH_TOKEN` and `SLACK_WEBHOOK_URL` to your credentials. **Expected output** Notification sent to Slack channel with message: “Alert: High CPU usage detected on server1” ### Example 2: SMS notification Send an SMS via Twilio: ```bash curl -X POST http://localhost:8181/api/v3/engine/notify \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "notification_text": "Critical alert: System down", "senders_config": { "sms": { "twilio_from_number": "'"$TWILIO_FROM_NUMBER"'", "twilio_to_number": "'"$TWILIO_TO_NUMBER"'" } } }' ``` Set `TWILIO_FROM_NUMBER` and `TWILIO_TO_NUMBER` to your phone numbers. Twilio credentials can be set via `TWILIO_SID` and `TWILIO_TOKEN` environment variables. ### Example 3: Multi-channel notification Send notifications via multiple channels simultaneously: ```bash curl -X POST http://localhost:8181/api/v3/engine/notify \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "notification_text": "Performance warning: Memory usage above threshold", "senders_config": { "slack": { "slack_webhook_url": "'"$SLACK_WEBHOOK_URL"'" }, "discord": { "discord_webhook_url": "'"$DISCORD_WEBHOOK_URL"'" } } }' ``` Set `SLACK_WEBHOOK_URL` and `DISCORD_WEBHOOK_URL` to your webhook URLs. ## Code overview ### Files * `notifier_plugin.py`: The main plugin code containing the HTTP handler for notification dispatch ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'notification_trigger'" ``` ### Main functions #### `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)` Handles incoming HTTP notification requests. Parses the request body, extracts notification text and sender configurations, and dispatches notifications to configured channels. Key operations: 1. Validates request body for required `notification_text` and `senders_config` 2. Iterates through sender configurations (Slack, Discord, HTTP, SMS, WhatsApp) 3. Dispatches notifications with built-in retry logic and error handling 4. Returns success/failure status for each channel ## Troubleshooting ### Common issues #### Issue: Notification not delivered **Solution**: Verify webhook URLs are correct and accessible. Check Twilio credentials and phone number formats. Review logs for specific error messages. #### Issue: Authentication errors **Solution**: Ensure Twilio credentials are set via environment variables or request parameters. Verify webhook URLs have proper authentication if required. #### Issue: Rate limiting **Solution**: Plugin includes built-in retry logic with exponential backoff. Consider implementing client-side rate limiting for high-frequency notifications. ### Environment variables For security, set Twilio credentials as environment variables: ```bash export TWILIO_SID=your_account_sid export TWILIO_TOKEN=your_auth_token ``` ### Viewing logs Check processing logs in the InfluxDB system tables: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%notifier%' ORDER BY event_time DESC LIMIT 10" ``` ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Notifier plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/notifier) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[notifications](/influxdb3/enterprise/tags/notifications/)[alerting](/influxdb3/enterprise/tags/alerting/)[integration](/influxdb3/enterprise/tags/integration/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | notification_text | string | required | Text content of the notification message | | senders_config | object | required | Configuration for each notification channel | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | slack_webhook_url | string | required | Slack webhook URL | | slack_headers | string | none | Base64-encoded JSON headers | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | discord_webhook_url | string | required | Discord webhook URL | | discord_headers | string | none | Base64-encoded JSON headers | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | http_webhook_url | string | required | Custom webhook URL for HTTP POST | | http_headers | string | none | Base64-encoded JSON headers | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | twilio_sid | string | required | Twilio Account SID (or use TWILIO_SID env var) | | twilio_token | string | required | Twilio Auth Token (or use TWILIO_TOKEN env var) | | twilio_from_number | string | required | Sender phone number in E.164 format | | twilio_to_number | string | required | Recipient phone number in E.164 format | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | twilio_sid | string | required | Twilio Account SID (or use TWILIO_SID env var) | | twilio_token | string | required | Twilio Auth Token (or use TWILIO_TOKEN env var) | | twilio_from_number | string | required | Sender WhatsApp number in E.164 format | | twilio_to_number | string | required | Recipient WhatsApp number in E.164 format | --- # NWS weather plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/nws-weather/ The US NWS Weather Sampler Plugin provides real-time weather data from the National Weather Service API for demonstration and sample data purposes. Fetch live observations from multiple weather stations across the United States with zero authentication required. Perfect for demos, training, testing, and providing realistic IoT-like time-series data streams. * **Zero authentication**: No API keys or signup required * **Real-time data**: Live weather updates from NOAA weather stations * **Multiple metrics**: Temperature, humidity, wind, pressure, visibility, and more * **Configurable stations**: Choose from thousands of US weather stations * **Parallel fetching**: Efficient concurrent data retrieval from multiple stations * **Statistics tracking**: Built-in monitoring of plugin performance ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters |Parameter | Type | Default | Description | |----------|------|-------------------------------------------------|-------------------------------------| |`stations`|string|KSEA.KORD.KJFK.KDEN.KATL.KDFW.KLAX.KMIA.KPHX.KBOS|Dot-separated list of NWS station IDs| ### Optional parameters | Parameter | Type | Default | Description | |---------------------|-------|------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `measurement` |string | weather\_observations | Destination measurement for weather data | | `user_agent` |string |InfluxDB3-NWS-Plugin/1.0| Custom User-Agent header for NWS API requests | |`use_data_timestamp` |boolean| true | Use timestamp from weather observation data instead of current time when writing data | |`enable_full_logging`|boolean| false |When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging.| **Note:** Station IDs are separated by dots (`.`) in trigger arguments. Example: `stations=KSEA.KSFO.KLAX` ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: No additional packages required (uses Python standard library only) * **Network access**: Outbound HTTPS access to `api.weather.gov` ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. No additional Python packages are required for this plugin. ## Trigger setup ### Scheduled trigger Create a trigger for periodic weather data collection: ```bash influxdb3 create trigger \ --database weather_demo \ --path "gh:influxdata/nws_weather/nws_weather_sampler.py" \ --trigger-spec "every:5m" \ nws_weather_trigger # Enable the trigger influxdb3 enable trigger --database weather_demo nws_weather_trigger ``` ## Example usage ### Example 1: Basic weather data collection Collect weather data from multiple stations every 5 minutes: ```bash # Create the trigger influxdb3 create trigger \ --database weather_demo \ --path "gh:influxdata/nws_weather/nws_weather_sampler.py" \ --trigger-spec "every:5m" \ --trigger-arguments "stations=KSEA.KSFO.KORD" \ nws_basic_weather # Enable the trigger influxdb3 enable trigger --database weather_demo nws_basic_weather # Query weather data (after a few minutes) influxdb3 query \ --database weather_demo \ "SELECT station_id, temperature_c, relative_humidity_percent, time FROM weather_observations ORDER BY time DESC LIMIT 10" ``` ### Expected output |station\_id|temperature\_c|relative\_humidity\_percent| time | |-----------|--------------|---------------------------|--------------------| | KSEA | 15.6 | 72.0 |2025-11-26T10:00:00Z| | KSFO | 18.3 | 65.0 |2025-11-26T10:00:00Z| | KORD | 12.2 | 68.0 |2025-11-26T10:00:00Z| ### Example 2: Custom measurement name and user agent ```bash # Create trigger with custom configuration influxdb3 create trigger \ --database weather_demo \ --path "gh:influxdata/nws_weather/nws_weather_sampler.py" \ --trigger-spec "every:10m" \ --trigger-arguments "stations=KJFK.KATL.KDFW,measurement=us_weather,user_agent=MyDemo/1.0" \ nws_custom_weather ``` ### Example 3: Using current time instead of API observation timestamp By default (`use_data_timestamp=true`), the plugin uses the timestamp from the weather observation data provided by the NWS API. However, since `api.weather.gov` may not update data regularly, this can lead to scenarios where no new records are written: **Scenario example:** * Weather station updates data every 10 minutes * Plugin runs every 3 minutes * First run: New record written (timestamp from API: 4:10 PM) * Second run (3 min later): No new record (API still returns data timestamped 4:10 PM - a duplicate that will overwrite the existing record) * Third run (6 min later): No new record (API still returns data timestamped 4:10 PM - a duplicate that will overwrite the existing record) * Fourth run (9 min later): New record written (timestamp from API: 4:20 PM) To ensure data is written on every plugin execution (useful for demos or populating tables with regular data), set `use_data_timestamp=false`: ```bash # Create trigger that writes data on every execution using current time influxdb3 create trigger \ --database weather_demo \ --path "gh:influxdata/nws_weather/nws_weather_sampler.py" \ --trigger-spec "every:3m" \ --trigger-arguments "stations=KSEA.KSFO,use_data_timestamp=false" \ nws_current_time ``` **Note:** The NWS API data is typically delayed. For example: * Request at 4:30 PM → Returns data timestamped 4:10 PM * Request at 4:37 PM → Returns data timestamped 4:25 PM ## Output data structure ### Measurement: `weather_observations` **Tags:** * `station_id`: NWS station identifier (for example, “KSEA”) * `station`: Full station name * `conditions`: Current weather conditions (for example, “Fair”, “Partly Cloudy”) * `longitude`: Station longitude (4 decimal places) * `latitude`: Station latitude (4 decimal places) **Fields:** * `temperature_c` (float): Temperature in Celsius * `dewpoint_c` (float): Dew point in Celsius * `wind_speed_kmh` (float): Wind speed in km/h * `wind_direction_degrees` (float): Wind direction (0-360°) * `wind_gust_kmh` (float): Wind gust speed in km/h * `barometric_pressure_pa` (float): Barometric pressure in Pascals * `visibility_m` (float): Visibility in meters * `relative_humidity_percent` (float): Relative humidity (0-100%) * `elevation_m` (float): Station elevation in meters **Timestamp:** * Automatically set from NWS observation time (nanosecond precision) ### Measurement: `nws_plugin_stats` Tracks plugin execution statistics: **Tags:** * `plugin`: Always “nws\_weather\_sampler” **Fields:** * `success_count` (int64): Number of successful station fetches * `error_count` (int64): Number of failed station fetches * `total_count` (int64): Total stations attempted * `success_rate` (float64): Success percentage (0-100) * `task_id` (string): Unique identifier for each plugin execution ## Finding Weather Stations ### Popular Station IDs Major US airports make excellent weather stations: **West Coast:** * `KSEA` - Seattle-Tacoma International, WA * `KPDX` - Portland International, OR * `KSFO` - San Francisco International, CA * `KLAX` - Los Angeles International, CA * `KSAN` - San Diego International, CA **East Coast:** * `KBOS` - Boston Logan International, MA * `KJFK` - New York JFK International, NY * `KEWR` - Newark Liberty International, NJ * `KPHL` - Philadelphia International, PA * `KATL` - Atlanta Hartsfield-Jackson International, GA * `KMIA` - Miami International, FL **Central:** * `KORD` - Chicago O’Hare International, IL * `KDFW` - Dallas/Fort Worth International, TX * `KDEN` - Denver International, CO * `KPHX` - Phoenix Sky Harbor International, AZ * `KLAS` - Las Vegas McCarran International, NV ### How to Find Stations 1. **Airport Codes**: Most US airport weather stations use the ICAO code (4 letters starting with ‘K’) * Example: Seattle-Tacoma airport code SEA → Station ID KSEA 2. **NWS Station List**: Browse available stations at: * https://www.weather.gov/documentation/services-web-api * API endpoint: `https://api.weather.gov/stations?state=WA` (replace WA with your state) 3. **Test a Station**: ``` curl -H "User-Agent: test" https://api.weather.gov/stations/KSEA/observations/latest ``` ## Example Queries ### Current temperature across all stations ``` SELECT station_id, temperature_c, time FROM weather_observations WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC; ``` ### Average temperature by location ``` SELECT station_id, AVG(temperature_c) as avg_temp, MAX(temperature_c) as max_temp, MIN(temperature_c) as min_temp FROM weather_observations WHERE time > now() - INTERVAL '24 hours' GROUP BY station_id ORDER BY avg_temp DESC; ``` ### Wind speed trends ``` SELECT time_bucket(time, INTERVAL '1 hour') as hour, station_id, AVG(wind_speed_kmh) as avg_wind_speed, MAX(wind_gust_kmh) as max_gust FROM weather_observations WHERE time > now() - INTERVAL '24 hours' GROUP BY hour, station_id ORDER BY hour, station_id; ``` ### Check plugin statistics ``` SELECT time, success_count, error_count, success_rate FROM nws_plugin_stats WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC; ``` ## Code overview ### Files * `nws_weather_sampler.py`: The main plugin code containing the scheduled handler for weather data collection ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'nws_weather_trigger'" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled weather data fetching. Fetches observations from configured NWS stations in parallel and writes to InfluxDB. Key operations: 1. Parses configuration from trigger arguments 2. Fetches weather data from stations in parallel using ThreadPoolExecutor 3. Writes weather observations and plugin statistics to InfluxDB 4. Implements comprehensive error handling and logging ## Troubleshooting ### Common issues #### Issue: No data appearing **Solution**: Check trigger status, review plugin logs, and verify network connectivity: ```bash # Check trigger status influxdb3 show summary --database weather_demo --token YOUR_TOKEN # Check plugin logs influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE log_text LIKE '%NWS%' ORDER BY event_time DESC LIMIT 10" # Verify network connectivity curl -H "User-Agent: test" https://api.weather.gov/stations/KSEA/observations/latest ``` #### Issue: Rate limiting from NWS API **Solution**: The NWS API has generous rate limits. If you encounter rate limiting, reduce polling frequency, reduce number of stations, or add delays between station requests. #### Issue: HTTP 404 errors for specific stations **Solution**: Station may not exist or be decommissioned. Check station ID spelling and try a different station from the same area. #### Issue: JSON decode errors **Solution**: NWS API may be temporarily unavailable. Check if station is reporting data: `curl -H "User-Agent: test" https://api.weather.gov/stations/STATION_ID/observations/latest` ### Debugging tips 1. **Check trigger status**: ``` influxdb3 show summary --database weather_demo --token YOUR_TOKEN ``` 2. **Enable/Disable trigger**: ``` influxdb3 disable trigger nws_weather --database weather_demo --token YOUR_TOKEN influxdb3 enable trigger nws_weather --database weather_demo --token YOUR_TOKEN ``` ## Integration with Grafana 1. **Add InfluxDB 3 Enterprise Data Source** in Grafana * URL: `http://localhost:8181` * Database: `weather_demo` * Token: Your admin token 2. **Create a Dashboard Panel**: **Query:** ``` SELECT time, station_id, temperature_c FROM weather_observations WHERE time > now() - INTERVAL '1 hour' ORDER BY time ``` **Visualization**: Time series graph 3. **Watch real-time weather data update** every 5 minutes! ## Common Customizations ### Change Polling Frequency **Every 1 minute** (for faster demos): ```bash influxdb3 delete trigger nws_weather --database weather_demo --token YOUR_TOKEN influxdb3 create trigger \ --trigger-spec "every:1m" \ --path "gh:influxdata/nws_weather/nws_weather_sampler.py" \ --database weather_demo \ --trigger-arguments "use_data_timestamp=false" \ --token YOUR_TOKEN \ nws_weather_fast ``` ### Select Different Cities Pick from major US cities: ```bash # West Coast --trigger-arguments "stations=KSEA.KPDX.KSFO.KLAX.KSAN" # East Coast --trigger-arguments "stations=KBOS.KJFK.KPHL.KBWI.KATL" # Central --trigger-arguments "stations=KORD.KMSP.KSTL.KCLE.KDEN" # Texas --trigger-arguments "stations=KDFW.KIAH.KAUS.KSAT.KHOU" ``` ## API Reference **National Weather Service API** * Base URL: https://api.weather.gov * Documentation: https://www.weather.gov/documentation/services-web-api * Authentication: None required (User-Agent header required) * Rate Limits: Not publicly documented, but generous * Data Update Frequency: Typically every 1-5 minutes * Coverage: Thousands of stations across the United States ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [NWS weather plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/nws_weather) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | stations | string | KSEA.KORD.KJFK.KDEN.KATL.KDFW.KLAX.KMIA.KPHX.KBOS | Dot-separated list of NWS station IDs | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | weather_observations | Destination measurement for weather data | | user_agent | string | InfluxDB3-NWS-Plugin/1.0 | Custom User-Agent header for NWS API requests | | use_data_timestamp | boolean | true | Use timestamp from weather observation data instead of current time when writing data | | enable_full_logging | boolean | false | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values. Enable temporarily for debugging. | | station_id | temperature_c | relative_humidity_percent | time | | --- | --- | --- | --- | | station_id | temperature_c | relative_humidity_percent | time | | KSEA | 15.6 | 72.0 | 2025-11-26T10:00:00Z | | KSFO | 18.3 | 65.0 | 2025-11-26T10:00:00Z | | KORD | 12.2 | 68.0 | 2025-11-26T10:00:00Z | --- # Opcua plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/opcua/ scheduled opcua, ingestion, iot, industrial InfluxDB 3 Enterprise > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The OPC UA Plugin enables periodic ingestion of OPC UA node values into InfluxDB 3 Enterprise. Connect to an OPC UA server, read current values from configured nodes, and automatically write them as time-series data with support for automatic data type detection. Designed for industrial IoT scenarios — PLCs, SCADA systems, CNC machines, and other OPC UA-enabled equipment. **Key characteristics:** * **Polling-based**: Reads current node values on each scheduled trigger call (not subscription-based) * **Two operating modes**: Explicit node listing for precise control, or browse mode for auto-discovery of thousands of devices * **Auto type detection**: OPC UA data types are automatically mapped to InfluxDB field types * **Namespace URI support**: Use stable namespace URIs instead of numeric indexes that may change on server restart * **Quality filtering**: Accept or reject OPC UA values based on quality category (good, uncertain, bad) * **Persistent connection**: OPC UA connection is cached and reused across scheduled calls with automatic reconnect on failure * **Scalar values only**: Arrays, structures, and other complex OPC UA types are not supported ## Operating Modes The plugin supports two mutually exclusive modes for specifying which OPC UA nodes to read: ### Explicit nodes mode Specify each node ID manually. Best for small setups where you need precise control over field names and types. All nodes are written as fields in a **single data point** per trigger execution. ### Browse mode Automatically discover devices and their variables by browsing the OPC UA address space from a root node. Best for large-scale deployments with hundreds or thousands of devices sharing the same variable structure. The plugin maps the Object hierarchy to InfluxDB tags and writes **one data point per unique tag combination** per trigger execution. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. This plugin supports TOML configuration files for complex setups, which can be specified using the `config_file_path` parameter. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |TOML Section| Description | |------------|------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`server_url`|string| `[opcua]` |OPC UA server endpoint URL (for example, `opc.tcp://localhost:4840`). Must use the `opc.tcp://` or `opc.tls://` scheme; other schemes (`file://`, `http://`, etc.) are rejected.| |`table_name`|string| `[opcua]` | InfluxDB measurement name for storing data | **One of the following is required** (mutually exclusive): | Parameter | Type | TOML Section | Description | |-------------|------|----------------|------------------------------------------------------------------------------------------------------------------------------------------| | `nodes` |string|`[opcua.nodes]` | Space-separated node mappings. Format: `field_name:namespace:identifier[:type]`. See [Node mapping](#node-mapping-format). | |`browse_root`|string|`[opcua.browse]`|Root node ID for auto-discovery mode (for example, `ns=2;s=Devices` or `nsu=;s=Devices`). See [Browse mode](#browse-mode-parameters).| ### Namespace parameters | Parameter | Type |Default| TOML Section | Description | |------------|------|-------|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`namespaces`|string| none |`[opcua.namespaces]`|Space-separated namespace alias mappings. Format: `alias=namespace_uri`. Aliases can be used instead of numeric namespace indexes in `nodes` and `tag_nodes` for stable configuration that survives server restarts. The plugin resolves URIs to numeric indexes at connection time.| **CLI format:** ```bash namespaces="siemens=urn:vendor:s7 beckhoff=urn:beckhoff:ua" nodes="temperature:siemens:s=SpindleTemp pressure:beckhoff:s=CoolantPressure:float" ``` **TOML format:** In TOML configuration, namespace aliases are not used — write node IDs directly with `nsu=;...` format: ```toml [opcua.nodes] temperature = "nsu=urn:vendor:s7;s=SpindleTemp" pressure = ["nsu=urn:beckhoff:ua;s=CoolantPressure", "float"] ``` The plugin resolves `nsu=` URIs to numeric `ns=` indexes after connecting to the server. ### Browse mode parameters | Parameter | Type |Default| TOML Section | Description | |------------------|------|-------|----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `browse_root` |string| none |`[opcua.browse]`| Root node ID for auto-discovery. Accepts a numeric namespace (`ns=2;s=Devices`) or a namespace URI (`nsu=;s=Devices`) resolved at connection time for restart stability. See [Namespace URI resolution](#namespace-uri-resolution). | | `browse_depth` | int | 2 |`[opcua.browse]`| Maximum browse depth. 1 = direct children, 2 = children of children, etc. | |`browse_cache_ttl`| int | 3600 |`[opcua.browse]`|Seconds to cache the discovered node set before re-browsing. Discovery runs on the first call and every `browse_cache_ttl` seconds thereafter; cached node IDs are read in between. Set low for a fast-changing address space, high to run discovery infrequently while collecting on a short trigger interval. See [Discovery caching](#discovery-caching).| | `path_tags` | list | `[]` |`[opcua.browse]`| Tag names mapping Object hierarchy levels to InfluxDB tags. First entry maps to depth-1 Objects, second to depth-2, etc. Length must be strictly less than `browse_depth`. Objects beyond `path_tags` length become field name prefixes. **Required in TOML** (specify `[]` explicitly for no hierarchy tags); defaults to `[]` in CLI. | | `filter` |string| none |`[opcua.browse]`| Regex pattern to filter discovered variable names. Only matching variables are included. Matches the original Variable browse name, not the prefixed field name (see [filter in nested hierarchies](#filter-in-nested-hierarchies)). | |`exclude_branches`|string| none |`[opcua.browse]`| Regex pattern to exclude Object nodes (branches) by browse name. Matched Objects and their entire subtree are skipped. Works at all browse depths. See [Exclude branches](#exclude-branches). | | `browse_tags` | list | none |`[opcua.browse]`| Variable names that should be stored as InfluxDB tags instead of fields. Values are converted to strings. See [Browse tags](#browse-tags). | | `name_separator` |string| none |`[opcua.browse]`| Separator for splitting Variable browse names into segments for tag extraction. Required when `name_tags` is set. See [Name tags](#name-tags). | | `name_tags` | list | none |`[opcua.browse]`| Tag names extracted from leading segments of Variable browse names split by `name_separator`. Remaining segments form the field name (joined with `_`). See [Name tags](#name-tags). | > > > **Note:** integer parameters (`browse_depth`, `browse_cache_ttl`) also accept numeric strings, so `browse_depth = "2"` is valid in TOML. `browse_cache_ttl` is capped at 30 days. > > ### Tag parameters | Parameter | Type |Default| TOML Section | Description | |--------------|------|-------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`default_tags`|string| none |`[opcua.default_tags]`| Space-separated static tags added to every point. Format: `key=value`. | | `tag_nodes` |string| none | `[opcua.tag_nodes]` |Space-separated tag node mappings. Format: `tag_name:namespace:identifier`. Values are read from OPC UA nodes at each execution and used as string tags. See [Static and dynamic tags](#static-and-dynamic-tags).| ### Quality filter parameters | Parameter | Type |Default|TOML Section| Description | |----------------|------|-------|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`quality_filter`|string|“good” | `[opcua]` |Space-separated list of OPC UA quality categories to accept: `good`, `uncertain`, `bad`. Values with non-matching quality are skipped and logged to `opcua_exceptions`.| **CLI format:** ```bash quality_filter="good uncertain" ``` **TOML format:** ```toml [opcua] quality_filter = ["good", "uncertain"] ``` ### Authentication parameters |Parameter | Type |Default| TOML Section | Description | |----------|------|-------|--------------|-------------------------------------| |`username`|string| none |`[opcua.auth]`|Username for UserToken authentication| |`password`|string| none |`[opcua.auth]`|Password for UserToken authentication| **Note:** Both `username` and `password` must be provided together. **Security:** When `security_policy` is not set, the connection is unencrypted and credentials are sent in cleartext. The plugin refuses to send `username`/`password` in that case unless `allow_insecure_auth` is explicitly set to `true`. Prefer configuring a `security_policy` for an encrypted connection. ### Security parameters | Parameter | Type | Default | TOML Section | Description | |-----------------|------|----------------|------------------|------------------------------------------------------------------------------------------------------------------| |`security_policy`|string| none |`[opcua.security]`|OPC UA security policy: `Basic128Rsa15`, `Basic256`, `Basic256Sha256`, `Aes128Sha256RsaOaep`, `Aes256Sha256RsaPss`| | `security_mode` |string|“SignAndEncrypt”|`[opcua.security]`| OPC UA security mode: `Sign`, `SignAndEncrypt`. Used when `security_policy` is set. | | `certificate` |string| none |`[opcua.security]`| Path to client certificate (DER format). Required when `security_policy` is set. | | `private_key` |string| none |`[opcua.security]`| Path to client private key (PEM format). Required when `security_policy` is set. | **Note:** When `security_policy` is set, both `certificate` and `private_key` are required. The OPC UA server must trust the client certificate. ### Advanced parameters | Parameter | Type |Default|TOML Section| Description | |----------------------|------|-------|------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `config_file_path` |string| none | CLI only | Path to TOML config file (absolute or relative to `PLUGIN_DIR`) | | `config_cache_ttl` | int | 3600 | `[opcua]` | Seconds to cache the parsed configuration (max 2592000). Controls how quickly credential, endpoint, table, or filter changes take effect. Independent of `browse_cache_ttl`. | |`disable_config_cache`| bool | false | `[opcua]` |Reload configuration on every call instead of caching for `config_cache_ttl` seconds. Also disables caching of the discovered browse structure, so in browse mode the address space is re-walked on every call. Useful during development.| |`allow_insecure_auth` | bool | false | `[opcua]` | Permit sending `username`/`password` when `security_policy` is not set (credentials sent in cleartext over an unencrypted connection). Only enable on a trusted network. | |`enable_full_logging` | bool | false | `[opcua]` | When `true`, full exception messages are written to logs. When `false` (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths). Enable temporarily for debugging. | ### Static and dynamic tags The plugin supports two ways to attach tags to every written data point: * **Static tags** (`default_tags`): Fixed key-value pairs defined at configuration time. Do not change between trigger calls. * **Dynamic tags** (`tag_nodes`): Values read from OPC UA nodes on each trigger execution, then converted to strings and used as tag values. A node whose value is an empty string adds no tag. Useful for identifiers stored on the device itself — serial number, model name, firmware version, or any other variable property. Dynamic tags are merged with static tags before writing. If a key appears in both, the dynamic value takes precedence. **CLI format:** ```bash # Static tags default_tags="location=factory_1 line=A" # Dynamic tags — same syntax as nodes, type suffix is ignored (values always stored as strings) tag_nodes="machine_id:2:s=MachineID serial:2:s=SerialNumber" ``` **TOML format:** ```toml [opcua.default_tags] location = "factory_1" line = "A" [opcua.tag_nodes] machine_id = "ns=2;s=MachineID" serial = "ns=2;s=SerialNumber" # List format is accepted but type is ignored — values are always read as strings # firmware = ["ns=2;s=FirmwareVersion", "string"] ``` ### Browse tags In browse mode, some discovered variables may represent identifiers (room name, zone, asset ID) rather than measurements. Use `browse_tags` to store them as InfluxDB tags instead of fields. A variable whose value is an empty string adds no tag. **CLI format:** ```bash browse_tags="room zone" ``` **TOML format:** ```toml [opcua.browse] browse_root = "ns=2;s=Sensors" browse_depth = 2 browse_tags = ["room", "zone"] ``` **Example:** Given this OPC UA address space: ``` Sensors +-- Sensor_001 | +-- room (value="Building_A") -> tag | +-- zone (value="Zone_3") -> tag | +-- temperature (value=22.5) -> field | +-- humidity (value=45.0) -> field ``` The plugin writes: ``` sensor_data,device=Sensor_001,room=Building_A,zone=Zone_3 temperature=22.5,humidity=45.0 ``` ### Exclude branches Use `exclude_branches` to skip entire Object nodes (and all their children) during auto-discovery. The pattern is a Python regex matched against Object browse names using `re.search()` (partial match). Works at all browse depths — both within `path_tags` range (skipping entire device groups) and beyond (skipping sub-object branches that would become field prefixes). **CLI format:** ```bash exclude_branches="Debug_.*|Test_.*" ``` **TOML format:** ```toml [opcua.browse] browse_root = "ns=2;s=Factory" browse_depth = 3 path_tags = ["line", "station"] exclude_branches = "Debug_.*|Test_.*" ``` **Example:** Given this OPC UA address space with `path_tags = ["line", "station"]`: ``` Factory +-- Line_A | +-- Station_01 -> included | +-- Station_02 -> included | +-- Debug_Station -> EXCLUDED (matches "Debug_.*") +-- Test_Line -> EXCLUDED (matches "Test_.*") +-- Station_01 -> not discovered (parent excluded) ``` The plugin writes only data from non-excluded branches: ``` factory_data,line=Line_A,station=Station_01 Temperature=42.5,Pressure=3.2 factory_data,line=Line_A,station=Station_02 Temperature=38.1,Pressure=2.8 ``` `Debug_Station` and `Test_Line` (with all its children) are completely skipped. **Note:** `exclude_branches` filters Object nodes (branches), while `filter` filters Variable nodes (leaves). They are independent and can be used together. ### Name tags Use `name_tags` with `name_separator` when Variable browse names encode hierarchy in a flat format (for example, `BuildingA.Zone3.Temperature`). The plugin splits each Variable name by the separator, extracts leading segments as InfluxDB tags, and uses remaining segments as the field name. Variables are then grouped by extracted tag combinations — one point per unique combination. Variables that don’t have enough segments (fewer than `len(name_tags) + 1`) are skipped with a warning. **CLI format:** ```bash name_separator="." name_tags="building zone" ``` **TOML format:** ```toml [opcua.browse] browse_root = "ns=2;s=FlatSensors" browse_depth = 1 path_tags = [] name_separator = "." name_tags = ["building", "zone"] ``` **Example:** Given Variables directly under `browse_root`: ``` FlatSensors +-- BuildingA.Zone3.Temperature (value=22.5) +-- BuildingA.Zone3.Humidity (value=45.0) +-- BuildingB.Zone1.Temperature (value=20.1) +-- BuildingB.Zone1.Humidity (value=50.2) ``` The plugin splits by `.`, extracts 2 leading segments as tags, and groups: ``` flat_sensors,building=BuildingA,zone=Zone3 Temperature=22.5,Humidity=45.0 flat_sensors,building=BuildingB,zone=Zone1 Temperature=20.1,Humidity=50.2 ``` **Note:** Tag names in `name_tags` must not overlap with `path_tags` tag names. ### File path resolution All file paths in the plugin (configuration file, certificates) follow the same resolution logic: * **Absolute paths** (for example, `/etc/opcua/client.der`) are used as-is * **Relative paths** (for example, `certs/client.der`) are resolved from `PLUGIN_DIR` environment variable If a relative path is specified and `PLUGIN_DIR` is not set, the plugin will return an error. ### Example TOML configuration [opcua\_config\_example.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/opcua/opcua_config_example.toml) - comprehensive configuration example with multiple scenarios ### TOML section reference | TOML section | Description | |----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| | `[opcua]` | Required. `server_url`, `table_name`, `quality_filter`, `config_cache_ttl`, `disable_config_cache`, `allow_insecure_auth` | |`[opcua.default_tags]`| Static tags as `key = "value"` pairs | | `[opcua.namespaces]` | Namespace alias mappings as `alias = "uri"` pairs. Validated but not used for alias substitution in TOML — use `nsu=;...` directly in node IDs | | `[opcua.tag_nodes]` | Dynamic tags from OPC UA nodes | | `[opcua.nodes]` | Explicit node mappings (mutually exclusive with `[opcua.browse]`) | | `[opcua.browse]` |Browse mode settings: `browse_root`, `browse_depth`, `browse_cache_ttl`, `path_tags`, `filter`, `exclude_branches`, `browse_tags`, `name_separator`, `name_tags`| | `[opcua.security]` | `security_policy`, `security_mode`, `certificate`, `private_key` | | `[opcua.auth]` | `username`, `password` | ## Node Mapping Format ### CLI arguments format ``` field_name:namespace:identifier[:type] ``` | Component |Required| Description | |------------|--------|----------------------------------------------------------------------------------------| |`field_name`| yes | InfluxDB field name for this node value | |`namespace` | yes |OPC UA namespace index (non-negative integer) or alias defined in `namespaces` parameter| |`identifier`| yes | OPC UA node identifier with type prefix (`s=`, `i=`, `g=`, `b=`) | | `type` | no | Explicit type override: `int`, `uint`, `float`, `string`, `bool` | When namespace is a numeric index, the node ID is built as `ns=;`. When namespace is an alias, the corresponding URI is looked up in `namespaces` and the node ID is built as `nsu=;`, then resolved to a numeric index at connection time. **Examples:** ```bash # Auto-detected types "temperature:2:s=SpindleTemp pressure:2:s=CoolantPressure" # With explicit type override "rpm:2:i=1234:uint status:3:s=MachineStatus:string" # Using namespace alias (requires namespaces="siemens=urn:vendor:s7") "temperature:siemens:s=SpindleTemp" ``` ### TOML format ```toml [opcua.nodes] # Auto-detected type (string format) temperature = "ns=2;s=SpindleTemp" # Explicit type override (list format) rpm = ["ns=2;s=SpindleRPM", "uint"] # Using namespace URI (resolved at connection time) pressure = "nsu=urn:vendor:s7;s=CoolantPressure" ``` ### Type auto-detection When no explicit type is specified, the plugin automatically maps OPC UA data types to InfluxDB field types: | OPC UA VariantType |InfluxDB type|LineBuilder method| |-------------------------------------|-------------|------------------| | Boolean | bool | `bool_field()` | | SByte, Int16, Int32, Int64 | int | `int64_field()` | | Byte, UInt16, UInt32, UInt64 | uint | `uint64_field()` | | Float, Double | float |`float64_field()` | |String, DateTime, and all other types| string | `string_field()` | ## Browse Mode Browse mode automatically discovers devices and their variables by traversing the OPC UA address space hierarchy. This is designed for large-scale deployments where manually listing every node ID is impractical. ### How browse works The plugin uses `path_tags` to map Object hierarchy levels to InfluxDB tags: ``` browse_root (specified by browse_root parameter) +-- Device_001 (Object, depth 1) -> path_tags[0] = "device" -> tag: device=Device_001 | +-- Temperature (Variable) -> field | +-- Pressure (Variable) -> field | +-- Status (Variable) -> field +-- Device_002 (Object, depth 1) -> path_tags[0] = "device" -> tag: device=Device_002 | +-- Temperature (Variable) -> field +-- ... (hundreds/thousands of devices) ``` * **Object nodes** at depths within `path_tags` range become **tag values** — the Object browse name is stored as the value of the corresponding tag * **Object nodes** beyond `path_tags` range become **field name prefixes** (for example, `Position_X`) * **Variable nodes** become **fields** — their browse name (with optional prefix) becomes the field name * **Variable nodes** directly under `browse_root` (with `path_tags = []`) are written as a single point without hierarchy tags Each unique combination of tag values produces one data point. All Variables within that combination are collected as fields of that point. Tags on the final data point come from multiple sources (in merge order): `default_tags`, `tag_nodes`, `path_tags`, `browse_tags`, and `name_tags`. See the corresponding parameter sections below for details. ### Nested hierarchies For servers with deeper nesting, increase `browse_depth` and configure `path_tags` to control which Object levels become tags vs field prefixes: ```bash # path_tags = ["device"], browse_depth = 3 browse_root (depth 0) +-- Robot_001 (Object, depth 1) -> tag: device=Robot_001 +-- Speed (Variable, depth 2) -> field: "Speed" +-- Position (Object, depth 2) -> field prefix: "Position_" | +-- X (Variable, depth 3) -> field: "Position_X" | +-- Y (Variable, depth 3) -> field: "Position_Y" | +-- Z (Variable, depth 3) -> field: "Position_Z" +-- Gripper (Object, depth 2) -> field prefix: "Gripper_" +-- Force (Variable, depth 3) -> field: "Gripper_Force" ``` Result — one data point per device, all Variables (including nested) collected as fields: ``` robot_data,device=Robot_001 Speed=100i,Position_X=1.5,Position_Y=2.3,Position_Z=0.8,Gripper_Force=5.2 ``` With multi-level `path_tags = ["line", "station"]` and `browse_depth = 3`, each unique line×station combination produces one data point: ``` browse_root (depth 0) +-- Line_A (Object, depth 1) -> tag: line=Line_A | +-- Station_01 (Object, depth 2) -> tag: station=Station_01 | | +-- Temperature (Variable) -> field | | +-- Pressure (Variable) -> field | +-- Station_02 (Object, depth 2) -> tag: station=Station_02 | +-- Temperature (Variable) -> field | +-- Pressure (Variable) -> field +-- Line_B (Object, depth 1) -> tag: line=Line_B +-- Station_01 (Object, depth 2) -> tag: station=Station_01 +-- Temperature (Variable) -> field ``` Result — one data point per unique tag combination (3 points): ``` factory_data,line=Line_A,station=Station_01 Temperature=42.5,Pressure=3.2 factory_data,line=Line_A,station=Station_02 Temperature=38.1,Pressure=2.8 factory_data,line=Line_B,station=Station_01 Temperature=40.0 ``` Objects beyond `path_tags` length become field name prefixes with underscore separator (for example, `Position_X`). ### Browse structure caching The discovered node structure (which devices have which variables) is cached for **1 hour**. Only values are re-read on each trigger call. This means: * First call: browse + read (slower) * Subsequent calls: read only (fast) * After 1 hour: browse is repeated to pick up any new devices or variables * Config reload also invalidates the browse cache ### TOML configuration for browse mode ```toml [opcua] server_url = "opc.tcp://192.168.1.100:4840" table_name = "cnc_data" [opcua.browse] browse_root = "ns=2;s=Devices" browse_depth = 2 path_tags = ["device"] # filter = "Temperature|Pressure|Status" # optional regex # browse_tags = ["room", "zone"] # optional: store these as tags instead of fields [opcua.default_tags] location = "factory_1" ``` ### Browse mode example output Given 3 devices with 3 variables each, the plugin writes: ``` cnc_data,location=factory_1,device=Device_001 Temperature=42.5,Pressure=3.2,Status="running" cnc_data,location=factory_1,device=Device_002 Temperature=38.1,Pressure=2.8,Status="idle" cnc_data,location=factory_1,device=Device_003 Temperature=40.0,Pressure=3.0,Status="running" ``` ## Data requirements The plugin automatically creates the target measurement table on first write. No pre-existing schema is required. ### OPC UA node requirements * **Scalar values only**: The plugin reads scalar (single) values. Arrays, structures, ExtensionObjects, and other complex types are not supported — they will be converted to string representation. * **Readable nodes**: Nodes must have the `Read` access level. Nodes with `Bad` status codes are skipped with errors logged. * **Explicit mode**: Node IDs must be known in advance. * **Browse mode**: The `browse_root` node must exist and contain Object/Variable child nodes. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: * `asyncua` (async Python OPC UA client library) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package asyncua ``` ## Trigger setup ### Scheduled reading with TOML configuration Recommended for production use: ```bash # 1. Set PLUGIN_DIR environment variable export PLUGIN_DIR=~/.plugins # 2. Copy and edit configuration file cp opcua_config_example.toml $PLUGIN_DIR/my_opcua_config.toml # Edit my_opcua_config.toml with your OPC UA server and node settings # 3. Create the trigger influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments config_file_path=my_opcua_config.toml \ opcua_ingestion ``` ### Scheduled reading with command-line arguments (explicit mode) For simple setups: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:5s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=cnc_machine,nodes=temperature:2:s=SpindleTemp pressure:2:s=CoolantPressure:float rpm:2:i=1234:uint,default_tags=location=factory_1,tag_nodes=machine_id:2:s=MachineID serial:2:s=SerialNumber' \ opcua_cnc ``` ### Explicit mode with namespace aliases ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:5s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=cnc_machine,namespaces=siemens=urn:vendor:s7,nodes=temperature:siemens:s=SpindleTemp pressure:siemens:s=CoolantPressure:float' \ opcua_ns_alias ``` ### Browse mode with command-line arguments For auto-discovery of many devices: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=cnc_data,browse_root=ns=2;s=Devices,browse_depth=2,path_tags=device,default_tags=location=factory_1' \ opcua_browse ``` ### Browse mode with multi-level path\_tags Map multiple Object hierarchy levels to tags: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=factory_data,browse_root=ns=2;s=Factory,browse_depth=3,path_tags=line station,default_tags=plant=north' \ opcua_multi_level ``` ### Browse mode with filter and browse\_tags Read only specific variables from each device, storing some as tags: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=plc_data,browse_root=ns=3;s=Floor2,browse_depth=2,path_tags=device,filter=Temperature|Pressure|Status|room,browse_tags=room' \ opcua_filtered ``` ### Browse mode with exclude\_branches Skip specific Object branches during auto-discovery: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=factory_data,browse_root=ns=2;s=Factory,browse_depth=3,path_tags=line station,exclude_branches=Debug_.*|Test_.*,default_tags=plant=north' \ opcua_exclude ``` ### Browse mode with name\_tags (flat namespace) Extract tags from Variable browse names in flat namespaces: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=flat_sensors,browse_root=ns=2;s=FlatSensors,browse_depth=1,path_tags=,name_separator=.,name_tags=building zone' \ opcua_flat ``` ### Secure connection with certificates ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://secure-plc.company.com:4840,table_name=plc_data,nodes=speed:3:s=Motor.Speed torque:3:s=Motor.Torque,security_policy=Basic256Sha256,security_mode=SignAndEncrypt,certificate=certs/client.der,private_key=certs/client.pem,username=operator,password=secret' \ secure_opcua ``` ### Accept uncertain quality values ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/opcua/opcua.py \ --trigger-spec "every:10s" \ --trigger-arguments 'server_url=opc.tcp://192.168.1.100:4840,table_name=sensor_data,nodes=temperature:2:s=Temperature pressure:2:s=Pressure,quality_filter=good uncertain' \ opcua_with_uncertain ``` ## Statistics and Monitoring The plugin tracks statistics and writes them to the `opcua_stats` table every 10 plugin calls. ### opcua\_stats Table | Field |Type | Description | |----------------------|-----|-----------------------------------------| | `server_url` (tag) | tag | OPC UA server URL | | `table_name` (tag) | tag | Target InfluxDB measurement | | `nodes_read` | int |Nodes successfully read in current period| | `nodes_failed` | int | Nodes failed to read in current period | | `points_written` | int | Data points written in current period | | `success_rate` |float| Success rate for current period (%) | | `total_nodes_read` | int |Total nodes successfully read (all time) | | `total_nodes_failed` | int | Total nodes failed (all time) | |`total_points_written`| int | Total data points written (all time) | | `total_success_rate` |float| Total success rate (all time, %) | ### Querying Statistics ```bash # Get latest statistics influxdb3 query --database mydb \ "SELECT * FROM opcua_stats ORDER BY time DESC LIMIT 10" # Success rate over time influxdb3 query --database mydb \ "SELECT server_url, success_rate, nodes_read, nodes_failed, points_written FROM opcua_stats WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC" ``` ## Error Handling Node read errors, type conversion failures, and quality filter rejections are logged to the `opcua_exceptions` table: ### opcua\_exceptions Table | Field | Type | Description | |------------------|------|---------------------------------------------------------------------------------------------| | `node_id` (tag) | tag | OPC UA node ID that failed | |`field_name` (tag)| tag | Configured field name for this node | |`error_type` (tag)| tag |Error type: `NodeReadError`, `TypeConversionError`, `InvalidFieldError`, `QualityFilterError`| | `error_message` |string| Detailed error message | ### Checking for Errors ```bash influxdb3 query --database mydb \ "SELECT * FROM opcua_exceptions ORDER BY time DESC LIMIT 10" ``` ## Important Behaviors ### Timestamp handling All fields within a single data point share the **same timestamp** — the latest OPC UA timestamp among successfully read nodes that passed the quality filter. Per node, `SourceTimestamp` is preferred; if unavailable, `ServerTimestamp` is used. The maximum across all accepted nodes becomes the point timestamp. If no OPC UA timestamps are available, system time is used. ### Connection lifecycle The OPC UA connection is **persistent** — it is cached and reused across scheduled trigger calls. This avoids the overhead of connecting/disconnecting on every call. The connection is automatically re-established if it fails during a read operation (one reconnect attempt per call). The connection is dropped and all caches are cleared if a call fails after reconnect. ### Dynamic tags ignore quality\_filter The `tag_nodes` feature only accepts values with **“good”** quality, regardless of the `quality_filter` setting. This is by design — tag values are identifiers (serial number, machine ID) and must be reliable. An uncertain or bad serial number should not become an InfluxDB tag. ### Quality filter with “bad” category Including `"bad"` in `quality_filter` has no practical effect. Nodes with bad StatusCode have no value on the OPC UA server — the plugin cannot write a non-existent value. These nodes will still generate `NodeReadError` exceptions even if “bad” is in the filter. ### filter applies to browse\_tags variables The `filter` regex applies to **all** discovered variables, including those listed in `browse_tags`. If you use `filter` and `browse_tags` together, the browse\_tags variable names must also match the filter pattern, otherwise they won’t be discovered. Example: with `browse_tags = ["room"]` and `filter = "temp|hum"`, the `room` variable will **not** be found. Use `filter = "room|temp|hum"` instead. ### filter in nested hierarchies The `filter` regex is applied to the **original Variable browse name**, not to the resulting prefixed field name. In nested hierarchies where Objects beyond `path_tags` become field prefixes, `filter` matches the variable’s own name at its level in the tree. Example with `path_tags = ["device"]` and `browse_depth = 3`: ``` Robot_001 (Object, depth 1) -> tag: device=Robot_001 +-- Position (Object, depth 2) -> prefix: "Position_" | +-- X (Variable) -> field: "Position_X" | +-- Y (Variable) -> field: "Position_Y" +-- Speed (Variable) -> field: "Speed" ``` * `filter = "X|Speed"` matches `X` and `Speed` — writes fields `Position_X` and `Speed` * `filter = "Position_X"` matches **nothing** — `Position_X` is the field name, not the browse name ### exclude\_branches vs filter `exclude_branches` and `filter` are independent parameters that operate on different node types: * **`exclude_branches`** matches **Object** node browse names — skips the entire branch (Object + all children) * **`filter`** matches **Variable** node browse names — includes/excludes individual leaf values They can be used together. For example, `exclude_branches = "Debug_.*"` with `filter = "Temperature|Pressure"` will first skip all debug Object branches, then among remaining branches only discover Temperature and Pressure variables. ### browse\_tags follow quality\_filter Unlike `tag_nodes` which always require “good” quality, `browse_tags` variables are subject to the normal `quality_filter` setting. If `quality_filter = ["good", "uncertain"]`, browse\_tags with uncertain quality will be accepted. ### Groups with only browse\_tags produce no data point If all variables in a device group are listed in `browse_tags` (no regular fields remain), no data point is written for that group. InfluxDB requires at least one field per point — browse\_tags alone are not sufficient. ### Discovery caching Two independent caches, each with its own TTL: * **Parsed configuration** — cached for `config_cache_ttl` seconds (default **1 hour**). Controls how quickly a change to credentials, endpoint, table, or filters is picked up. * **Discovered browse structure** — cached for `browse_cache_ttl` seconds (default **1 hour**). Discovery runs on the first call and every `browse_cache_ttl` seconds thereafter; cached node IDs are read in between. This means: * Set a low `browse_cache_ttl` for a fast-changing address space, or set it much larger than the trigger interval so discovery runs rarely while collection stays frequent — the recommended pattern for large or high-latency servers where a full browse is expensive * New devices added to the OPC UA server appear after at most `browse_cache_ttl` seconds * Changes to the config take effect after at most `config_cache_ttl` seconds. Changes to the browse-relevant part (`server_url`, `browse_root`, `browse_depth`, `filter`, `exclude_branches`, …) also trigger an immediate re-browse on the next config reload; other changes keep the cached structure * A failed connection drops the cached config, so a credential or endpoint fix is retried on the next call rather than after `config_cache_ttl` * `disable_config_cache = true` disables both caches, so config and address space are re-read on every call * Any unhandled error clears all caches, forcing a fresh reload on the next call ### Namespace URI resolution When using `nsu=` node IDs (either from `namespaces` aliases in CLI mode or directly in TOML), the URI-to-index resolution happens **once at connection time**. The resolved numeric indexes are stored in the config object. If the server assigns different namespace indexes after a restart, the plugin will re-resolve them on the next reconnect. This applies to `nodes`, `tag_nodes`, and `browse_root` — use `browse_root = "nsu=;s=..."` in browse mode for the same restart stability. If the URI is not found on the server, connection fails with a configuration error rather than browsing into an empty result. ### Namespace aliases in TOML vs CLI In **CLI arguments** mode, you can define short aliases (for example, `siemens`) via the `namespaces` parameter and use them in `nodes` and `tag_nodes` definitions. In **TOML** mode, aliases are not supported — use the full `nsu=;...` format directly in node IDs. The `[opcua.namespaces]` section in TOML is validated but not used for alias substitution. ## Troubleshooting ### Check Plugin Logs ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'opcua_ingestion' ORDER BY time DESC LIMIT 20" ``` ### Common Issues #### “asyncua library not installed” or “No module named ‘asyncua’” ```bash influxdb3 install package asyncua ``` #### “Configuration file not found” * For relative paths, ensure `PLUGIN_DIR` environment variable is set * For absolute paths, verify the file exists at the specified location ```bash # For relative paths export PLUGIN_DIR=~/.plugins ls $PLUGIN_DIR/my_opcua_config.toml # Or use absolute path ls /etc/opcua/my_opcua_config.toml ``` #### “Failed to connect to OPC UA server” * Verify server URL and port (`opc.tcp://host:port`) * Check network connectivity to the OPC UA server * Ensure the OPC UA server is running and accepting connections * For secure connections, verify that the server trusts the client certificate #### “Certificate and private\_key required when security\_policy is set” When using a security policy, both files are required: * `certificate`: Client certificate in DER format (`.der`) * `private_key`: Client private key in PEM format (`.pem`) #### “Invalid ‘server\_url’ scheme” * `server_url` must use the `opc.tcp://` or `opc.tls://` scheme * Other schemes (`file://`, `http://`, etc.) are rejected #### “Refusing to send username/password over an unencrypted connection” * `username`/`password` are set but no `security_policy` is configured * Configure a `security_policy` for an encrypted connection (recommended) * Or, only on a trusted network, set `allow_insecure_auth = true` to permit cleartext credentials #### “Namespace URI not found on server” * The `nsu=` URI does not match any namespace registered on the server * Use an OPC UA browser (UaExpert) to see the server’s namespace array * Namespace URIs are case-sensitive #### “Bad status” errors for specific nodes * Verify the node ID exists on the server (use an OPC UA browser like UaExpert) * Check that the configured user has read access to the node * The node may be temporarily unavailable — check the server’s diagnostic info #### “Null value” for a node * The node exists but has no value assigned * The server may not have initialized the variable yet * Check the node’s status in an OPC UA client tool #### All nodes failing * Verify the namespace index is correct (namespaces may differ between server restarts — consider using `nsu=` URIs instead) * Use an OPC UA browser to confirm the exact node IDs * Check server logs for access control issues #### “Browse discovered no variables” (browse mode) * Verify `browse_root` points to a valid node that contains Object or Variable children * If `browse_root` uses a numeric `ns=` index, confirm the index is still current — it can shift after a server restart. Use `nsu=;s=...` instead so the plugin resolves the index at connection time * Check that `browse_depth` is sufficient to reach Variable nodes * The root node may contain nodes of other types (for example, Methods) — only Objects and Variables are traversed * If using `filter`, verify the regex pattern matches your variable names * If using `exclude_branches`, verify the pattern is not excluding all Object nodes at a required depth * Use an OPC UA browser (UaExpert) to inspect the address space structure #### Config changes not taking effect * Configuration is cached for `config_cache_ttl` seconds (default 1 hour) — either wait for expiry, lower `config_cache_ttl`, or set `disable_config_cache = true` * Any plugin error automatically clears the cache #### “Previous opcua call still running, skipping this tick” * The plugin reuses a single cached connection and event loop, which cannot be used by two calls at once. With an asynchronous trigger (`--run-asynchronous`), scheduled invocations may overlap; if a call is still running when the next tick fires, that tick is skipped instead of failing. * Occasional skips are harmless. Frequent skips mean each read takes longer than the interval — reduce the per-call runtime or give it more time: * Read fewer nodes per trigger, or narrow browse mode (smaller `browse_depth`, tighter `filter`/`exclude_branches`) * Increase the trigger interval (e.g. `--trigger-spec "every:30s"` instead of `every:10s`) * Check for a slow or unreachable server adding connection latency ## Limitations * **Polling only**: The plugin reads current values on each scheduled call. It does not use OPC UA subscriptions for change-based monitoring. Fast-changing signals may be missed between polling intervals. * **No Historical Data Access (HDA)**: Only current values are read, not historical data from the server. * **No write support**: The plugin only reads from OPC UA nodes. Writing values or calling methods is not supported. * **Scalar values only**: Arrays, structures, ExtensionObjects, and other complex data types are converted to string representation. * **Single table**: All nodes (or all browsed devices) are written to one measurement. For multiple tables, create separate plugin triggers. * **Browse mode requires hierarchy**: Browse discovers nodes based on the OPC UA address space hierarchy. If the server uses a flat namespace with dotted identifiers (for example, `API15_Histo1.zone_5130.Signal`) rather than a folder tree, browse mode will not discover child signals — use `name_tags` with `name_separator` to extract hierarchy from flat names, or use explicit nodes mode instead. * **No OPC-DA support**: Only OPC UA (Unified Architecture) is supported. Legacy OPC-DA (COM/DCOM-based) connections require a separate gateway. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Opcua plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/opcua) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | TOML Section | Description | | --- | --- | --- | --- | | Parameter | Type | TOML Section | Description | | server_url | string | [opcua] | OPC UA server endpoint URL (for example, opc.tcp://localhost:4840 ). Must use the opc.tcp:// or opc.tls:// scheme; other schemes ( file:// , http:// , etc.) are rejected. | | table_name | string | [opcua] | InfluxDB measurement name for storing data | | Parameter | Type | TOML Section | Description | | --- | --- | --- | --- | | Parameter | Type | TOML Section | Description | | nodes | string | [opcua.nodes] | Space-separated node mappings. Format: field_name:namespace:identifier[:type] . See Node mapping . | | browse_root | string | [opcua.browse] | Root node ID for auto-discovery mode (for example, ns=2;s=Devices or nsu=;s=Devices ). See Browse mode . | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | namespaces | string | none | [opcua.namespaces] | Space-separated namespace alias mappings. Format: alias=namespace_uri . Aliases can be used instead of numeric namespace indexes in nodes and tag_nodes for stable configuration that survives server restarts. The plugin resolves URIs to numeric indexes at connection time. | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | browse_root | string | none | [opcua.browse] | Root node ID for auto-discovery. Accepts a numeric namespace ( ns=2;s=Devices ) or a namespace URI ( nsu=;s=Devices ) resolved at connection time for restart stability. See Namespace URI resolution . | | browse_depth | int | 2 | [opcua.browse] | Maximum browse depth. 1 = direct children, 2 = children of children, etc. | | browse_cache_ttl | int | 3600 | [opcua.browse] | Seconds to cache the discovered node set before re-browsing. Discovery runs on the first call and every browse_cache_ttl seconds thereafter; cached node IDs are read in between. Set low for a fast-changing address space, high to run discovery infrequently while collecting on a short trigger interval. See Discovery caching . | | path_tags | list | [] | [opcua.browse] | Tag names mapping Object hierarchy levels to InfluxDB tags. First entry maps to depth-1 Objects, second to depth-2, etc. Length must be strictly less than browse_depth . Objects beyond path_tags length become field name prefixes. Required in TOML (specify [] explicitly for no hierarchy tags); defaults to [] in CLI. | | filter | string | none | [opcua.browse] | Regex pattern to filter discovered variable names. Only matching variables are included. Matches the original Variable browse name, not the prefixed field name (see filter in nested hierarchies ). | | exclude_branches | string | none | [opcua.browse] | Regex pattern to exclude Object nodes (branches) by browse name. Matched Objects and their entire subtree are skipped. Works at all browse depths. See Exclude branches . | | browse_tags | list | none | [opcua.browse] | Variable names that should be stored as InfluxDB tags instead of fields. Values are converted to strings. See Browse tags . | | name_separator | string | none | [opcua.browse] | Separator for splitting Variable browse names into segments for tag extraction. Required when name_tags is set. See Name tags . | | name_tags | list | none | [opcua.browse] | Tag names extracted from leading segments of Variable browse names split by name_separator . Remaining segments form the field name (joined with _ ). See Name tags . | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | default_tags | string | none | [opcua.default_tags] | Space-separated static tags added to every point. Format: key=value . | | tag_nodes | string | none | [opcua.tag_nodes] | Space-separated tag node mappings. Format: tag_name:namespace:identifier . Values are read from OPC UA nodes at each execution and used as string tags. See Static and dynamic tags . | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | quality_filter | string | “good” | [opcua] | Space-separated list of OPC UA quality categories to accept: good , uncertain , bad . Values with non-matching quality are skipped and logged to opcua_exceptions . | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | username | string | none | [opcua.auth] | Username for UserToken authentication | | password | string | none | [opcua.auth] | Password for UserToken authentication | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | security_policy | string | none | [opcua.security] | OPC UA security policy: Basic128Rsa15 , Basic256 , Basic256Sha256 , Aes128Sha256RsaOaep , Aes256Sha256RsaPss | | security_mode | string | “SignAndEncrypt” | [opcua.security] | OPC UA security mode: Sign , SignAndEncrypt . Used when security_policy is set. | | certificate | string | none | [opcua.security] | Path to client certificate (DER format). Required when security_policy is set. | | private_key | string | none | [opcua.security] | Path to client private key (PEM format). Required when security_policy is set. | | Parameter | Type | Default | TOML Section | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | TOML Section | Description | | config_file_path | string | none | CLI only | Path to TOML config file (absolute or relative to PLUGIN_DIR ) | | config_cache_ttl | int | 3600 | [opcua] | Seconds to cache the parsed configuration (max 2592000). Controls how quickly credential, endpoint, table, or filter changes take effect. Independent of browse_cache_ttl . | | disable_config_cache | bool | false | [opcua] | Reload configuration on every call instead of caching for config_cache_ttl seconds. Also disables caching of the discovered browse structure, so in browse mode the address space is re-walked on every call. Useful during development. | | allow_insecure_auth | bool | false | [opcua] | Permit sending username / password when security_policy is not set (credentials sent in cleartext over an unencrypted connection). Only enable on a trusted network. | | enable_full_logging | bool | false | [opcua] | When true , full exception messages are written to logs. When false (default), only the exception type is logged, to avoid leaking sensitive values (credentials, payloads, paths). Enable temporarily for debugging. | | TOML section | Description | | --- | --- | | TOML section | Description | | [opcua] | Required. server_url , table_name , quality_filter , config_cache_ttl , disable_config_cache , allow_insecure_auth | | [opcua.default_tags] | Static tags as key = "value" pairs | | [opcua.namespaces] | Namespace alias mappings as alias = "uri" pairs. Validated but not used for alias substitution in TOML — use nsu=;... directly in node IDs | | [opcua.tag_nodes] | Dynamic tags from OPC UA nodes | | [opcua.nodes] | Explicit node mappings (mutually exclusive with [opcua.browse] ) | | [opcua.browse] | Browse mode settings: browse_root , browse_depth , browse_cache_ttl , path_tags , filter , exclude_branches , browse_tags , name_separator , name_tags | | [opcua.security] | security_policy , security_mode , certificate , private_key | | [opcua.auth] | username , password | | Component | Required | Description | | --- | --- | --- | | Component | Required | Description | | field_name | yes | InfluxDB field name for this node value | | namespace | yes | OPC UA namespace index (non-negative integer) or alias defined in namespaces parameter | | identifier | yes | OPC UA node identifier with type prefix ( s= , i= , g= , b= ) | | type | no | Explicit type override: int , uint , float , string , bool | | OPC UA VariantType | InfluxDB type | LineBuilder method | | --- | --- | --- | | OPC UA VariantType | InfluxDB type | LineBuilder method | | Boolean | bool | bool_field() | | SByte, Int16, Int32, Int64 | int | int64_field() | | Byte, UInt16, UInt32, UInt64 | uint | uint64_field() | | Float, Double | float | float64_field() | | String, DateTime, and all other types | string | string_field() | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | server_url (tag) | tag | OPC UA server URL | | table_name (tag) | tag | Target InfluxDB measurement | | nodes_read | int | Nodes successfully read in current period | | nodes_failed | int | Nodes failed to read in current period | | points_written | int | Data points written in current period | | success_rate | float | Success rate for current period (%) | | total_nodes_read | int | Total nodes successfully read (all time) | | total_nodes_failed | int | Total nodes failed (all time) | | total_points_written | int | Total data points written (all time) | | total_success_rate | float | Total success rate (all time, %) | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | node_id (tag) | tag | OPC UA node ID that failed | | field_name (tag) | tag | Configured field name for this node | | error_type (tag) | tag | Error type: NodeReadError , TypeConversionError , InvalidFieldError , QualityFilterError | | error_message | string | Detailed error message | --- # Prophet forecasting plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/prophet-forecasting/ ⚡ scheduled, http 🏷️ forecasting, machine-learning, time-series, predictive-analytics 🔧 InfluxDB 3 Enterprise > [!Note] > This plugin requires InfluxDB 3 Enterprise 3.8.2 or later. The Prophet Forecasting Plugin forecasts time series stored in InfluxDB 3 Enterprise with Meta’s Prophet library, fitting a model to each series inside InfluxDB. A scheduled trigger forecasts on every run and writes the forecast back to InfluxDB; an HTTP trigger answers each request with the forecast. * **A field or a query**: forecast a numeric field of a measurement, or the result of any SQL query, such as a chart’s aggregating query. * **Per series**: `group_by` splits the data by tags, and each series is trained on its own. * **Prediction intervals**: any interval levels, as `lower_` / `upper_`. * **Seasonality and holidays**: seasonality mode, trend changepoints, custom holidays and built-in holiday calendars. * **Calendar grids**: forecast points on month starts, week ends or business days. * **Saved models**: train once and reuse the model of each series. * **Validation**: compare the forecast with the latest measured points, withhold a series that misses a threshold, and send one alert for the run. * **Backtest**: on request, check the forecast on the latest history: the share of points inside each interval and the mean absolute error. * **Shared forecasting interface**: requests, responses, errors and stored rows follow the contract shared by forecasting plugins, and [`forecast_interface.json`](https://github.com/influxdata/influxdb3_plugins/blob/main/influxdata/prophet_forecasting/forecast_interface.json) describes what this plugin supports. ## Software requirements * **InfluxDB 3 Enterprise**: 3.8.2 or later, with the Processing Engine enabled. * **Python packages**: * `prophet` (time series forecasting) * `pandas` (for data manipulation; 2.x and 3.x are both supported) * `requests` (validation alerts) * `influxdata-plugin-utils` (configuration loading and writing) * **Notification Sender Plugin** *(optional)*: needed for validation alerts; see the[influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/). ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package prophet influxdb3 install package pandas influxdb3 install package requests influxdb3 install package influxdata-plugin-utils ``` 3. *(Optional)* For alerts, install the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) and create an HTTP trigger for it. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ## Scheduled trigger A scheduled trigger forecasts on every run of its schedule and writes the forecast to `target_measurement`. It returns nothing; a failure goes to the plugin log. ### Scheduled trigger parameters Set these with `--trigger-arguments` (CLI) or `trigger_arguments` (HTTP API) when creating the trigger, or in the[TOML file](#scheduled-trigger-toml-configuration). A `string/list` parameter is a space-separated string in trigger arguments, through the CLI and the HTTP API alike, and a list in the TOML file; `string/table` is `tag=value` pairs in trigger arguments and a table in the TOML file. See [Value formats](#value-formats). | Parameter | Type | Default | Example | Description | |-------------------------|----------------|---------------------------------|-------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `source_measurement` | string | required¹ | `temperature` | Measurement to read | | `field` | string | required¹ | `value` | Numeric field to forecast | | `query` | string | required¹ |`SELECT * FROM temperature WHERE time > now() - INTERVAL '30 days'`|SQL whose result is forecast instead of `source_measurement` and `field`; see [Forecasting a query](#forecasting-a-query). A query with commas goes in the TOML file| | `source_database` | string | the trigger’s database | `demo` | Database to read from; see [Security](#security) | | `tag_filter` | string/table | none | `room=Kitchen site=eu` or `{ room = ["Kitchen", "Lounge"] }` | Tags the rows must match: one value per tag in a string, one value or a list per tag in a table. Not with `query` | | `sql_filter` | string | none | `value > 0` | SQL condition on tags and fields, without `WHERE`, AND-ed with `tag_filter` and inserted as is; see [Security](#security). Not with `query` | | `group_by` | string/list | none | `room site` or `["room", "site"]` | Tags to split by; each distinct combination is its own series with its own model; see [Series](#series). Not with `query` | | `series_limit` | integer ≥ 1 | none | `5` | Forecast only the first N forecasts (series × fields) in the order of [Series](#series) and skip the rest; at most `max_series` | | `window` | duration | required² | `30d` | History as a time window ending now, including `validation_window`; see [History](#history) | | `point_limit` | integer ≥ 3 | required² | `2000` | History as the most recent points of each series; see [History](#history) | | `sampling_interval` | duration | none | `1h` | Resample history to the mean per bucket, also the forecast step; see [Forecast step and horizon](#forecast-step-and-horizon) | | `forecast_freq` |pandas frequency| inferred | `MS` | Step of the forecast points over raw history; see [Forecast step and horizon](#forecast-step-and-horizon) | | `horizon_steps` | integer ≥ 1 | required³ | `24` | Forecast length in points | | `horizon_duration` | duration | required³ | `2d` | Forecast length as a duration from the last history point, rounded up to whole steps | | `levels` | string/list | `0.5 0.8` | `0.5 0.8 0.95` or `[0.5, 0.8, 0.95]` | Interval levels, each between 0 and 1 | | `model` | string | `prophet` | `prophet` | The only model; any other value is refused | | `target_measurement` | string |`_forecasts.`| `_forecasts.temperature` | Measurement the forecast is written to; required when `query` reads several measurements | | `target_database` | string | the trigger’s database | `forecasts` | Database the forecast is written to | | `forecast_id` | string | none | `ops-1h` | Tag `forecast_id` in every row, so forecasts of one series with different settings keep their own rows | | `model_mode` | string | `train` | `predict` | `train` fits a new model on every run; `predict` reuses the saved one; see [Models](#models) | | `unique_suffix` | string | none | `20250619_v1` | Model version, up to 64 characters from letters, digits, `.`, `_` and `-`; required with `model_mode` `predict` | | `seasonality_mode` | string | `additive` | `multiplicative` | `additive` or `multiplicative` | |`changepoint_prior_scale`| number \> 0 | `0.05` | `0.1` | Flexibility of trend changepoints | | `changepoints` | string/list | none | `2025-01-01 2025-06-01` or `["2025-01-01", "2025-06-01"]` | ISO dates of trend changepoints, in UTC unless they name a timezone; each must fall within every series’ training history | | `holiday_date_list` | string/list | none | `2024-12-25 2025-12-25` or `["2024-12-25", "2025-12-25"]` | ISO dates of custom holidays, in UTC unless they name a timezone, set together with `holiday_names` and as many | | `holiday_names` | string/list | none | `Christmas Christmas` or `["Christmas", "Christmas"]` | Names matching `holiday_date_list`; a name may repeat, and one with spaces needs a list; a name Prophet reserves, such as `trend`, is refused | | `holiday_country_names` | string/list | none | `US` or `["US"]` | Country code for built-in holidays, refused if Prophet does not know it; of several only the first is used, with a warning | | `validation_window` | duration | none | `3d` | Duration held back from training before now; see [Validation](#validation) | | `validation_alignment` | string | `position` | `nearest` | How measured and forecast points are paired: `position` or `nearest`; see [Validation](#validation) | | `msre_threshold` | number ≥ 0 | no limit | `0.05` | Highest Mean Squared Relative Error a validated series may have | | `max_series` | integer ≥ 1 | `100` | `50` | Most forecasts one run may produce (series × fields); more is refused unless `series_limit` keeps fewer | | `max_forecast_points` | integer ≥ 1 | `10000` | `1000` | Most forecast points per series, counting the validation window; a longer horizon is refused | | `config_file_path` | string | none | `prophet_forecasting_scheduler.toml` | TOML file with more arguments; see [Scheduled trigger TOML configuration](#scheduled-trigger-toml-configuration) | ¹ Either `source_measurement` with `field`, or `query`. ² At least one of `window` and `point_limit`, unless `query` is set. ³ Exactly one of `horizon_steps` and `horizon_duration`. A scheduled trigger always writes and returns no response, and its history moves with each run, so it refuses`write_results`, `backtest_points`, `start_time` and `end_time`. An unknown argument or an invalid value fails every run with `bad_request` in the log. ### Validation alert parameters `influxdb3_auth_token` and `twilio_token` are secrets: as trigger arguments they fail every run with `bad_request`, so set them in the [TOML file](#scheduled-trigger-toml-configuration) or the[environment](#scheduled-trigger-environment-variables). The other parameters are set like those above. | Parameter | Type | Default | Example | Description | |----------------------|-----------|----------------------|--------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | `is_sending_alert` | boolean | `false` | `true` | Send one alert per run listing the series that failed validation | | `senders` |string/list| none | `slack sms` or `["slack", "sms"]` | Sender types: `slack`, `discord`, `http`, `sms`, `whatsapp`; required with alerts | | `notification_text` | string | a built-in text | `Validation failed on $measurement: $series` | Template with the variables `$version`, `$measurement`, `$field`, `$series`, `$start_time`, `$end_time`, `$output_measurement` | | `notification_path` | string | `notify` | `notify` | URL path of the notifier plugin’s trigger | | `port_override` | integer | `8181` | `8182` | Port of the InfluxDB HTTP API, 1–65535 | |`influxdb3_auth_token`| string |`INFLUXDB3_AUTH_TOKEN`| `your_token` | Token for the notifier’s endpoint. TOML file or environment only | | `slack_webhook_url` | string | none | `https://hooks.slack.com/services/...` | Webhook URL; required with the `slack` sender | | `slack_headers` | string | none |`eyJDb250ZW50LVR5cGUiOiAiYXBwbGljYXRpb24vanNvbiJ9`| Base64-encoded headers for the Slack webhook | |`discord_webhook_url` | string | none | `https://discord.com/api/webhooks/...` | Webhook URL; required with the `discord` sender | | `discord_headers` | string | none |`eyJDb250ZW50LVR5cGUiOiAiYXBwbGljYXRpb24vanNvbiJ9`| Base64-encoded headers for the Discord webhook | | `http_webhook_url` | string | none | `https://example.com/webhook` | Webhook URL; required with the `http` sender | | `http_headers` | string | none |`eyJDb250ZW50LVR5cGUiOiAiYXBwbGljYXRpb24vanNvbiJ9`| Base64-encoded headers for the HTTP webhook | | `twilio_sid` | string | none | `ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` | Twilio Account SID, for the `sms` and `whatsapp` senders | | `twilio_token` | string | none | `your_auth_token` |Twilio Auth Token, for the `sms` and `whatsapp` senders. TOML file or environment only; unset, the notifier reads its own `TWILIO_TOKEN`| | `twilio_to_number` | string | none | `+1234567890` | Recipient number; required with the `sms` and `whatsapp` senders | | `twilio_from_number` | string | none | `+19876543210` | Twilio sender number; required with the `sms` and `whatsapp` senders | ### Scheduled trigger environment variables Every parameter can also come from an environment variable of the InfluxDB server named`INFLUXDB3_PROPHET_FORECASTING_` in upper case, for example `INFLUXDB3_PROPHET_FORECASTING_LEVELS` sets`levels` for every trigger of the plugin. * The environment is the lowest layer: a trigger argument overrides it, and the TOML file overrides both. * `INFLUXDB3_PROPHET_FORECASTING_CONFIG_FILE_PATH` names the TOML file when the trigger has no `config_file_path`argument. * `INFLUXDB3_AUTH_TOKEN` also gives `influxdb3_auth_token`, below `INFLUXDB3_PROPHET_FORECASTING_INFLUXDB3_AUTH_TOKEN`. * Variables of the HTTP-only keys (`..._WRITE_RESULTS`, `..._BACKTEST_POINTS`, `..._START_TIME`, `..._END_TIME`) are ignored by a scheduled trigger. ### Scheduled trigger TOML configuration `config_file_path`, a trigger argument or the environment variable above, names a `.toml` file with more arguments. * The path is absolute or relative to the plugin directory: the first set of `PLUGIN_DIR`, `INFLUXDB3_PLUGIN_DIR` and the parent of the engine’s virtual environment. * The file’s keys override the trigger arguments; an argument the file does not set still applies. * Lists and tables are native: `group_by = ["room"]`, `holiday_names = ["New Year"]`, and `[tag_filter]` with several values per tag (`room = ["Kitchen", "Lounge"]`). Keep the `[tag_filter]` table last: keys after a table header belong to the table. * A `query` or `sql_filter` with a comma goes here: the CLI splits `--trigger-arguments` at every comma. * An unknown key, `config_file_path` inside the file, `write_results`, `backtest_points`, `start_time` or `end_time`fails every run with `bad_request`. #### Example scheduled TOML configuration [prophet\_forecasting\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/prophet_forecasting/prophet_forecasting_scheduler.toml) lists every key a scheduled trigger accepts, with its default. A query, which has commas, set through a file: ```toml query = "SELECT date_bin(INTERVAL '1 hour', time) AS time, room, avg(value) AS value FROM temperature WHERE time > now() - INTERVAL '30 days' GROUP BY 1, room" horizon_steps = 24 series_limit = 5 ``` ### Create a scheduled trigger With trigger arguments: ```bash influxdb3 create trigger \ --database mydb \ --path prophet_forecasting.py \ --trigger-spec "every:1h" \ --trigger-arguments "source_measurement=temperature,field=value,group_by=room,window=30d,sampling_interval=1h,horizon_steps=24" \ prophet_forecast_scheduled ``` With the TOML file: ```bash influxdb3 create trigger \ --database mydb \ --path prophet_forecasting.py \ --trigger-spec "every:1h" \ --trigger-arguments config_file_path=prophet_forecasting_scheduler.toml \ prophet_forecast_scheduled ``` A `cron:` spec takes six fields, seconds first (`cron:20 0 * * * *`), which spreads the runs of several triggers. Tune the settings with on-demand requests to an [HTTP trigger](#http-trigger) first: the response shows `points_read`and `step_ms` for every series. ### Scheduled trigger example The trigger above writes 24 one-hour points per room to `_forecasts.temperature` on every run. Read the latest run of each series: ```sql SELECT time, room, forecast, lower_80, upper_80, step, run_time FROM ( SELECT *, max (run_time) OVER (PARTITION BY field, room) AS latest FROM "_forecasts.temperature" ) AS f WHERE run_time = latest ORDER BY room, time ``` Expected output (first rows): | time | room |forecast|lower\_80|upper\_80|step| run\_time | |-------------------|-------|--------|---------|---------|----|--------------------| |2026-10-29T13:00:00|Kitchen| 21.42 | 20.61 | 22.25 | 1 |2026-10-29T12:00:05Z| |2026-10-29T14:00:00|Kitchen| 21.58 | 20.74 | 22.39 | 2 |2026-10-29T12:00:05Z| The rows are described in [Output data structure](#output-data-structure). With `forecast_id`, add `WHERE forecast_id = ''` to the inner query. ### Scheduled trigger errors A run that fails as a whole writes nothing and logs `[] : ` at `ERROR`, with the codes of the[HTTP errors](#http-trigger-errors). A series that fails does not stop the others, which are written: * too few points logs `[] too_few_points for : ` at `WARN`; * a failed validation (`other`), an unreadable saved model (`model_unavailable`) or a failed training (`internal`) logs the same line at `ERROR`. These lines and the validation alert come before the write, so a failed write costs the run’s rows, not them. Read the log: ```sql SELECT event_time, log_level, log_text FROM system.processing_engine_logs WHERE trigger_name = 'prophet_forecast_scheduled' ORDER BY event_time DESC LIMIT 20 ``` ## HTTP trigger An HTTP trigger answers each request with the forecast and writes it only with `write_results`. Only an admin token can call it. ### HTTP request parameters Send these as a JSON object in the body of a POST request. The trigger’s arguments and TOML file set defaults for the same keys; see[HTTP trigger arguments](#http-trigger-arguments). A `string/list` parameter is a list in the body and the TOML file, and a space-separated string in trigger arguments, headers and the query string; the body takes the string too. `string/object` is the same for `tag_filter`, with`tag=value` pairs as the string. | Parameter | Type | Default | Example | Description | |-------------------------|----------------|---------------------------------|-------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| | `source_measurement` | string | required¹ | `"temperature"` | Measurement to read | | `field` | string | required¹ | `"value"` | Numeric field to forecast | | `query` | string | required¹ |`"SELECT time, room, value FROM temperature WHERE time > now() - INTERVAL '30 days'"`| SQL whose result is forecast instead of `source_measurement` and `field`; see [Forecasting a query](#forecasting-a-query) | | `source_database` | string | the trigger’s database | `"demo"` | Database to read from; an unknown one is refused; see [Security](#security) | | `tag_filter` | string/object | none | `{"room": ["Kitchen", "Lounge"], "site": "eu"}` or `"room=Kitchen site=eu"` | Tags the rows must match: one value or an array of several per tag in an object, one value per tag in a string. Not with `query` | | `sql_filter` | string | none | `"value > 0"` | SQL condition on tags and fields, without `WHERE`, AND-ed with `tag_filter` and inserted as is; see [Security](#security). Not with `query` | | `group_by` | string/list | none | `["room", "site"]` or `"room site"` | Tags to split by; each distinct combination is its own series with its own model; see [Series](#series). Not with `query` | | `series_limit` | integer ≥ 1 | none | `5` | Forecast only the first N forecasts (series × fields) in the order of [Series](#series) and skip the rest; at most `max_series` | | `window` | duration | required² | `"30d"` | History as a time window ending now, including `validation_window`; see [History](#history) | | `point_limit` | integer ≥ 3 | required² | `2000` | History as the most recent points of each series; see [History](#history) | | `start_time` | ISO 8601 | required² | `"2025-05-20T00:00:00Z"` | Start of an absolute history, with a timezone. Set with `end_time`. Not with `query` | | `end_time` | ISO 8601 | required² | `"2025-06-19T00:00:00Z"` | End of an absolute history, with a timezone; the forecast continues from the last point before it; see [History](#history) | | `sampling_interval` | duration | none | `"1h"` | Resample history to the mean per bucket, also the forecast step; see [Forecast step and horizon](#forecast-step-and-horizon) | | `forecast_freq` |pandas frequency| inferred | `"MS"` | Step of the forecast points over raw history; see [Forecast step and horizon](#forecast-step-and-horizon) | | `horizon_steps` | integer ≥ 1 | required³ | `24` | Forecast length in points | | `horizon_duration` | duration | required³ | `"2d"` | Forecast length as a duration from the last history point, rounded up to whole steps | | `levels` | string/list | `[0.5, 0.8]` | `[0.5, 0.8, 0.95]` or `"0.5 0.8 0.95"` | Interval levels, each between 0 and 1 | | `model` | string | `"prophet"` | `"prophet"` | The only model; any other value is refused | | `write_results` | boolean | `false` | `true` | Also write the forecast; see [Output data structure](#output-data-structure) | | `target_measurement` | string |`_forecasts.`| `"_forecasts.temperature"` | Measurement a written forecast goes to; required when `query` reads several measurements | | `target_database` | string | the trigger’s database | `"forecasts"` | Database a written forecast goes to | | `forecast_id` | string | none | `"ops-1h"` | Tag `forecast_id` in every written row, so forecasts of one series with different settings keep their own rows | | `model_mode` | string | `"train"` | `"predict"` | `train` fits a new model per request; `predict` reuses the saved one; see [Models](#models) | | `unique_suffix` | string | none | `"20250619_v1"` | Model version, up to 64 characters from letters, digits, `.`, `_` and `-`; required with `model_mode` `predict` | | `seasonality_mode` | string | `"additive"` | `"multiplicative"` | `additive` or `multiplicative` | |`changepoint_prior_scale`| number \> 0 | `0.05` | `0.1` | Flexibility of trend changepoints | | `changepoints` | string/list | none | `["2025-01-01", "2025-06-01"]` or `"2025-01-01 2025-06-01"` | ISO dates of trend changepoints, in UTC unless they name a timezone; each must fall within every series’ training history | | `holiday_date_list` | string/list | none | `["2024-12-25", "2025-12-25"]` or `"2024-12-25 2025-12-25"` | ISO dates of custom holidays, in UTC unless they name a timezone, set together with `holiday_names` and as many | | `holiday_names` | string/list | none | `["Christmas", "Christmas"]` or `"Christmas Christmas"` | Names matching `holiday_date_list`; a name may repeat, and one with spaces needs a list; a name Prophet reserves, such as `trend`, is refused | | `holiday_country_names` | string/list | none | `["US"]` or `"US"` | Country code for built-in holidays, refused if Prophet does not know it; of several only the first is used, with a warning | | `validation_window` | duration | none | `"3d"` | Duration held back from training before now or `end_time`; see [Validation](#validation). Not with `backtest_points` | | `validation_alignment` | string | `"position"` | `"nearest"` | How measured and forecast points are paired: `position` or `nearest`; see [Validation](#validation) | | `msre_threshold` | number ≥ 0 | no limit | `0.05` | Highest Mean Squared Relative Error a validated series may have | | `backtest_points` | integer ≥ 1 | none | `12` |Hold out the latest N history points and return each series’ `backtest`, at most `max_forecast_points`; see [Backtest](#backtest). Not with `validation_window`| | `max_series` | integer ≥ 1 | `100` | `50` | Most forecasts the request may produce (series × fields); more is refused unless `series_limit` keeps fewer | | `max_forecast_points` | integer ≥ 1 | `10000` | `1000` | Most forecast points per series, counting the validation window; a longer horizon is refused | ¹ Either `source_measurement` with `field`, or `query`. ² A measurement needs `window`, `point_limit`, or `start_time` with `end_time`; a query needs none of them. ³ Exactly one of `horizon_steps` and `horizon_duration`. An unknown key or an invalid value answers `bad_request`.`config_file_path` is never read from a request. ### HTTP headers and query string Any request parameter can also come from a header or the query string, as text: * A header is named `X-Influxdb3-Prophet-Forecasting-`, with hyphens for underscores, in any case:`X-Influxdb3-Prophet-Forecasting-horizon-steps: 24`. A header with this prefix that names no parameter answers `bad_request`; other headers are ignored. * The query string takes the key as is: `?horizon_steps=24&levels=0.5%200.8`. From lowest to highest, the sources are the environment, trigger arguments, the TOML file, the body, headers and the query string. ### HTTP trigger arguments Trigger arguments and the TOML file set defaults for every request to the trigger. A choice between keys is made as a whole by the highest source that sets either side, and the other side from lower sources is dropped: | Set by the request | Replaces, set on the trigger | |------------------------|---------------------------------------------------------------------| | `query` |`source_measurement`, `field`, `tag_filter`, `sql_filter`, `group_by`| | `horizon_steps` | `horizon_duration` | |`start_time`, `end_time`| `window`, `point_limit` | | `forecast_freq` | `sampling_interval` | Each works the other way round too, and both sides in one source answer `bad_request`. In this choice, empty text, lists and objects do not count as set. Elsewhere, empty text and `null` keep the value from a lower source, and an empty list or object replaces it:`group_by: []` means no split for this request. ### HTTP trigger environment variables As for the [scheduled trigger](#scheduled-trigger-environment-variables): every request parameter can come from`INFLUXDB3_PROPHET_FORECASTING_`, the lowest source.`INFLUXDB3_PROPHET_FORECASTING_CONFIG_FILE_PATH` names the TOML file when the trigger has no `config_file_path`argument. ### HTTP trigger TOML configuration `config_file_path`, a trigger argument or the environment variable above, names a `.toml` file of defaults; a request can never name it. Its path and format are those of the [scheduled trigger’s file](#scheduled-trigger-toml-configuration), and an unknown key fails every request with `bad_request`. #### Example HTTP TOML configuration [prophet\_forecasting\_http.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/prophet_forecasting/prophet_forecasting_http.toml) lists every key a request accepts, with its default. A trigger that fixes the source and leaves each request the horizon: ```toml source_measurement = "temperature" field = "value" group_by = ["room"] window = "30d" sampling_interval = "1h" ``` ### Create an HTTP trigger ```bash influxdb3 create trigger \ --database mydb \ --path prophet_forecasting.py \ --trigger-spec "request:forecast" \ --trigger-arguments config_file_path=prophet_forecasting_http.toml \ prophet_forecast_http ``` The trigger answers at `/api/v3/engine/forecast`. Without trigger arguments, each request carries its whole configuration. ### HTTP trigger examples #### Forecast a field, per room ```bash curl --request POST http://localhost:8181/api/v3/engine/forecast \ --header "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "source_measurement": "temperature", "field": "value", "group_by": ["room"], "window": "30d", "sampling_interval": "1h", "horizon_steps": 2, "levels": [0.8] }' ``` Expected response: ```json { "status": "ok", "model": "prophet", "levels": [ 0.8 ], "series_total": 2, "results": [ { "field": "value", "tags": { "room": "Kitchen" }, "points_read": 720, "step_ms": 3600000, "forecast": [ { "timestamp": 1761742800000, "forecast": 21.42, "lower_80": 20.61, "upper_80": 22.25 }, { "timestamp": 1761746400000, "forecast": 21.58, "lower_80": 20.74, "upper_80": 22.39 } ] }, { "field": "value", "tags": { "room": "Lounge" }, "status": "error", "code": "too_few_points", "message": "need >= 3 points, got 2 in the last 30d" } ] } ``` | Field | Description | |-------------------------------------|----------------------------------------------------------------------------------| | `model`, `levels` |What ran; every point holds `lower_` and `upper_` for each level| | `series_total` | Forecasts (series × fields) the request selects before `series_limit` | | `results[].field`, `tags` | The forecast field and the series’ tags; see [Series](#series) | | `results[].points_read` | History points the model was trained on | | `results[].step_ms` | Spacing between forecast points, in milliseconds | | `results[].forecast[].timestamp` | The point’s time, in epoch milliseconds | | `results[].forecast[].forecast` | The central forecast, Prophet’s `yhat` | | `results[].backtest` |With `backtest_points`: the check on the latest history; see [Backtest](#backtest)| |`results[].status`, `code`, `message`| Present only when this series failed, which leaves the request `200` | #### Forecast a query ```bash curl --request POST http://localhost:8181/api/v3/engine/forecast \ --header "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "query": "SELECT date_bin(INTERVAL '\''1 hour'\'', time) AS time, room, avg(value) AS value FROM temperature WHERE time > now() - INTERVAL '\''30 days'\'' GROUP BY 1, room", "horizon_duration": "1d" }' ``` #### An absolute history, a calendar grid and holidays ```bash curl --request POST http://localhost:8181/api/v3/engine/forecast \ --header "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "source_measurement": "sales", "field": "total", "start_time": "2023-01-01T00:00:00Z", "end_time": "2026-01-01T00:00:00Z", "forecast_freq": "MS", "horizon_steps": 6, "seasonality_mode": "multiplicative", "holiday_country_names": ["US"], "write_results": true, "forecast_id": "sales-monthly" }' ``` ### Backtest With `backtest_points: N`, the plugin also checks the forecast on the latest history. It holds out the last N points of each series, trains a second model on the points before them with the same settings, levels and history length, forecasts the held-out times, and compares. Each successful result then carries `backtest`: ```json { "backtest": { "points": 12, "coverage": { "50": 0.5, "80": 0.833 }, "mae": 1.43 } } ``` * `points`: the measured points compared. * `coverage`: per level, the share of those points inside its interval. A share close to the level means the intervals are calibrated. * `mae`: the mean absolute difference between the central forecast and the measured points, in the field’s units. `backtest` is `null` when fewer than 3 points precede the held-out ones, or when training on them fails, for example with a date in `changepoints` after the cut, which the plugin logs as a warning; the forecast itself is the same as without the check. The check reads N more points per series and trains each series a second time, so it about doubles the training time. With `model_mode: predict` the forecast still comes from the saved model, as that model has seen the held-out points the check trains a fresh one, which is not saved. With N equal to the horizon, it measures accuracy over the same distance as the forecast; a small N gives a noisy estimate. N above `max_forecast_points`, or `backtest_points` with `validation_window`, answers `bad_request`. ### HTTP trigger errors A request that fails as a whole answers with an HTTP error status and a JSON body: ```json { "status": "error", "code": "bad_request", "message": "set exactly one of horizon_steps and horizon_duration" } ``` | Code |HTTP | When | |-------------------|-----|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `bad_request` |`400`|Unknown or invalid parameter, unknown measurement, field or database, a query that does not plan, too many forecasts or forecast points, a series mixing tag sets without `sampling_interval`| | `internal` |`500`| A query or the write failed, or a forecast row cannot be built | | `too_few_points` | — | Per series, in `results`, with the request still `200` | | `other` | — | Per series, in `results`: the series failed validation | |`model_unavailable`| — | Per series, in `results`: its saved model cannot be read | | `internal` | — | Per series, in `results`: training or predicting this series failed | The plugin logs a `bad_request` at `WARN` and the other request-level codes at `ERROR`. ## Forecasting details ### Value formats * In trigger arguments, headers and the query string every value is a string: lists are space-separated (`group_by=room site`, `levels=0.5 0.8`, `senders=slack sms`), and `tag_filter` is space-separated `tag=value` pairs with one value per tag. * The JSON body and the TOML file take native lists and objects, also for a name with spaces (`["New Year"]`). * **Durations** are a whole number and a unit: `us`, `ms`, `s`, `min`, `h`, `d`, `w`, `m` (a month, 30.42 days), `q`(91.25 days) or `y` (365 days). `m` is never minutes. * The CLI splits `--trigger-arguments` at every comma, so a `query` or `sql_filter` with a comma (`room IN ('a', 'b')`) goes in the TOML file or the request. ### Series * Without `group_by` all rows matching the filters form one series; with it, each distinct combination of the `group_by`tags is its own series. With `query`, each distinct combination of the key columns is. * A series’ `tags` hold the `group_by` tags and every other tag that all its rows carry with one value; a tag missing from some rows is left out. When the data changes which tags a series carries, for example a second host appears, the forecast goes to a new series. * `results` are ordered by the series’ `group_by` tags, or the query’s key columns, compared as text with a missing value last, then by field name.`series_limit: N` keeps the first N forecasts in this order and skips the rest without an error; `series_total` in the response counts them all, so a chart can say “the first 5 of 12”. In measurement mode skipped series are neither read nor checked for mixed tag sets; with `query` the result is read whole, then cut. The scheduled trigger logs how many it kept. * Rows of a series sharing a timestamp are averaged. * Without `sampling_interval`, a series whose rows come from several tag sets writing at the same time within the history the model trains on is refused with `bad_request`, for example several hosts without `group_by`: interleaved, they would form a saw-tooth. That history is the series’ last `point_limit` points, or all of `window` when only it is set, before the`validation_window` if there is one. Add the tags the message names to `group_by`, pin them with `tag_filter`, or set `sampling_interval` to forecast their mean per bucket. A tag that changes value over time, such as a version after an upgrade, does not make a series mixed; one that flips back and forth, such as a status, does, so set `sampling_interval` for it. With `query` the plugin cannot see tag sets, so the query has to keep series apart. * When the forecast is written, a `group_by` tag or a query key column named like a column of the[stored rows](#output-data-structure) is refused with `bad_request`: as a tag it would break the row or merge series. ### History * A measurement needs a history: `window`, `point_limit`, or, on demand, `start_time` with `end_time`. * `window` reads every point of the window; with `point_limit`, its last points per series. There is no default cap on the points: `window: 7d` on one-second data sends about 600,000 points per series to Prophet, and training time grows with them, so bound them with `point_limit` or `sampling_interval`. * `point_limit` alone reads its buckets of `sampling_interval` back from now, at most 365 days, or 30 days of raw points, so a read never scans the whole table. * `window` and `point_limit` read history that ends now, so a row with a later timestamp is not read until its time comes: with a device whose clock runs an hour fast, its latest hour stays out of the forecast. * `start_time` and `end_time` read that range, including rows later than now, and exclude `window` and `point_limit`. * With `sampling_interval` the history always ends at the last full bucket before now or `end_time`, whichever comes first. A `window` shorter than one bucket, or a `start_time` not before that end, answers `bad_request`. * With `query` there is no default window; see [Forecasting a query](#forecasting-a-query). * A series needs at least 3 points; fewer answers `too_few_points` for that series, naming what was read. ### Forecast step and horizon The forecast starts one step after the last history point and the horizon counts from there, so the forecast continues the chart’s line. With `end_time` that is the last point before it. | Set | History the model reads | Step of the forecast | |-------------------|------------------------------------------------------------------------------|---------------------------------------------------------------------------------------| |`sampling_interval`|The mean per bucket, aligned like `date_bin`, without the bucket still filling| The interval | | `forecast_freq` | The raw points |The pandas frequency: fixed (`30min`, `1h`) or calendar (`D`, `W-SUN`, `MS`, `QS`, `B`)| | neither | The raw points | The frequency pandas infers from the data, or else their median spacing | * Prophet takes unevenly spaced points as they are, so an empty bucket is simply no point. * The response gives times in milliseconds, so with a step below 1 ms `step_ms` is `0` and points can share a`timestamp`; stored rows keep nanoseconds. * The median spacing is rounded to a round unit about 1% of it, and forecast points keep the rhythm most history points follow, so a later run overwrites the same timestamps despite jitter. * `horizon_steps` counts points of the step; `horizon_duration` is rounded up to whole steps, at least one, so `1h` on daily data gives one point a day ahead. ### Models The only model is `prophet`. * `model_mode: train`, the default, fits a new model on every run or request. * `model_mode: predict` loads the model saved for `unique_suffix` and the series, or trains and saves it when there is no file. A loaded model is not refreshed: to retrain, change `unique_suffix`. `points_read` is then the number of points it was trained on. A loaded model still needs at least 3 history points, which place its forecast in time; `point_limit: 3` is the smallest read. * Saved models live in `/prophet_models/`, as `prophet_model__.json`. The digest stands for the data the model trains on: `source_database`, the source measurement or `query`, the field, the series’ `group_by` tags or key columns, `tag_filter`, `sql_filter` and `sampling_interval`. Changing any of them trains a new model once, and the earlier file stays on disk. * `window`, `point_limit`, `start_time`, `end_time` and the model’s settings, such as `seasonality_mode`, are not part of the digest: to apply a change to them, change `unique_suffix`. A tag outside `group_by` does not change the model either; add it to `group_by` to get a model per value. * The plugin cannot see which database its trigger belongs to, so two triggers in different databases with one`unique_suffix` share a file unless they set `source_database`. * A `query` whose text differs on every call, such as one with absolute times, trains and saves a new model each time; use `model_mode: train` for it. * A file that cannot be read answers `model_unavailable` for its series, and the other series go on. * A file that cannot be written, as in a read-only plugin directory, logs a warning: the forecast still answers, and the series trains again on every run. * Each series is trained on its own, one after another, about 0.1–0.5 s per series on CPU, so `max_series` also bounds how long a request takes. **Intervals** come from Prophet’s own samples (1000 per point), so any level works. Level `L` gives the percentiles `(1 − L) / 2` and `(1 + L) / 2` of the samples, widened if needed to hold the central forecast. ### Validation `validation_window` holds back the latest measured points to check the forecast on them. * The model trains on the history before the cut, which is `validation_window` before the end of the history: now or`end_time`, or with `sampling_interval` the last full bucket.`points_read` and `point_limit` count that training history. * With `sampling_interval` the window is rounded out to whole buckets, at least one: `30min` with `1h` buckets checks one bucket, and `90min` checks two. A window of few points gives a noisy estimate. * It forecasts through the window and on past the last measured point; only the part after that point is answered and written, and the horizon counts from it. * Measured and forecast points are compared by the Mean Squared Relative Error: `position` pairs them in time order, and`nearest` pairs each measured point with the closest forecast point within half a step. Measured zeros are skipped, as a relative error divides by them. * A series fails when its MSRE exceeds `msre_threshold`, cannot be computed, or the window holds no measured point. It answers `other` without a forecast, and nothing is written for it; the other series go on. * A scheduled trigger with `is_sending_alert` sends one alert for the run, listing the failed series in `$series`. * It does not combine with `backtest_points`, which answers `bad_request`: both hold back the latest points. ### Forecasting a query With `query`, the plugin runs the SQL against `source_database` and forecasts what it returns: * **time**: the column named `time`, so the query aliases its timestamp `AS time`. No `time` column answers `bad_request`. * **series keys**: the text and boolean columns; each distinct combination is one series. * **values**: the numeric columns; each is forecast on its own, per series. A result without a numeric column answers `bad_request`. A timestamp column other than `time` (`max(time) AS last_seen`) reaches the plugin as an integer and is forecast as a number, so leave such columns out of the query. * **column types**: the Processing Engine passes `BIGINT`, `BIGINT UNSIGNED`, `DOUBLE`, `BOOLEAN`, text and timestamp columns to plugins. A query that returns another type (`INT`, `REAL`, `DECIMAL`, an array or a struct) fails with `internal`, so cast such a column, for example `CAST(x AS DOUBLE)`. * **source measurement**: taken from the query’s plan (`EXPLAIN`). When the query reads one measurement, it is the `source_measurement` tag and names the default `target_measurement`; with several, writing needs `target_measurement`. `window`, `point_limit`, `sampling_interval` and `validation_window` apply to the query’s result; `start_time` and`end_time` do not. The plugin reads the whole result into memory before `point_limit` cuts it, so the query must bound its own time range (`WHERE time > now() - INTERVAL '30 days'`), above all in a scheduled trigger, which runs it on every run. With `window`, the server filters the result by its `time` column, as`SELECT * FROM () WHERE time > now() - `; a query that cannot be wrapped this way, such as one with an integer `time`, is read whole and filtered by the plugin, which logs a warning. The query defines its own history, so rows with later timestamps stay, and the forecast continues after them, as with day-ahead prices. To leave them out, add `WHERE time < now()` to the query. Only `sampling_interval` ends the result at the last full bucket. ### Output data structure The scheduled trigger, and an HTTP request with `write_results`, write one row per forecast point to`target_measurement`. Tags: * `field`: the forecast field * `model`: `prophet` * `source_measurement`: the source measurement; with `query`, only when it reads one * `forecast_id`: when set * the series tags, as in the response `tags`, except one named like a tag or field of the row, which is left out Fields: * `forecast` (float): the central forecast, Prophet’s `yhat` * `lower_`, `upper_` (float): interval bounds, for example `lower_80`, `upper_97.5` * `step` (integer): the forecast step, `1` for the first point after the history * `points_read` (integer): history points the model was trained on * `run_time` (string): when the run happened, UTC to the second, for example `2026-10-26T12:00:00Z` Timestamp: `time`, the forecast point’s time in nanoseconds. A later run overwrites the points it repeats, and a longer horizon from an earlier run remains, so read the latest`run_time`. Two triggers that forecast one series with different settings, for example different horizons, write to the same rows; give each its own `forecast_id` and read by it. ### Security * `sql_filter` is inserted into the query as is. It can reference any table of the source database, for example in a subquery. * The plugin reads `source_database` and writes `target_database` with the Processing Engine’s rights, not the caller’s. Only an admin token can call an HTTP trigger, so whoever calls it can already read every database. * Trigger arguments can be read from `system.processing_engine_trigger_arguments`, so `influxdb3_auth_token` and`twilio_token` are refused there. Webhook URLs and `*_headers` are accepted; when they hold a token, keep them in the TOML file or the environment. ### Forecast interface manifest [`forecast_interface.json`](https://github.com/influxdata/influxdb3_plugins/blob/main/influxdata/prophet_forecasting/forecast_interface.json) describes this plugin under the[shared forecasting interface](https://github.com/influxdata/influxdb3_plugins/blob/main/docs/FORECASTING_INTERFACE.md), contract version 1: the engine’s name and description, its modes, filters, history and horizon controls, its model and limits, and advanced parameters. An app can read it from the plugin’s registry package to configure its UI. The plugin never reads it: it validates every request on its own. ## Code overview ### Key functions * **`process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None)`**: HTTP entry point; returns the response body and its HTTP status. * **`process_scheduled_call(influxdb3_local, call_time, args=None)`**: scheduled entry point; writes every run, logs errors with their code, and sends the validation alert. * **`_load_settings(args, names, request_layers)`** / **`_resolve(cfg, scheduled, now_ns)`**: merge every settings source with `influxdata-plugin-utils`, resolve the choices between keys by the highest source, validate each value, and check the parameters against each other. * **`_measurement_plan(influxdb3_local, req)`** / **`_read_measurement(influxdb3_local, req, plan)`**: check the names against the schema, find the series and their single-valued tags with one aggregate query (which also enforces`max_series` and `series_limit`, keeping only the chosen series with `_chosen_series`), and refuse a series mixing tag sets that write at the same time; then read the history of every series, and the validation window’s points in a second query. * **`_query_measurements(influxdb3_local, req)`** / **`_read_query(influxdb3_local, req, task_id)`**: plan the query with `EXPLAIN` for its source measurement, run it (with `window` as a filter the server applies when the query can be wrapped), split the result into series by column type, sort them, and apply `series_limit`. * **`_forecast_step(series, req)`** / **`_forecast_grid(series, req)`**: find the step and the first point after the history, and lay out the points through the validation window and the horizon; a horizon past `max_forecast_points`refuses the request before any model trains. * **`_model_for(influxdb3_local, req, series, task_id)`** / **`_train(req, series)`**: train a model, or load and save the one for this version and series. * **`_predict(model, times, levels)`**: the central forecast from `predict`, and the bounds of each level from one`predictive_samples` draw. * **`_validation_problem(influxdb3_local, series, grid, central, req, task_id)`**: compare the forecast with the measured points by MSRE and say why a series fails. * **`_split_backtest(series, req)`** / **`_backtest(influxdb3_local, req, split, task_id)`**: hold out the last`backtest_points` points, train a model on a history as long as the forecast’s, and score it against them: the share inside each level and the MAE. * **`_send_validation_alert(influxdb3_local, cfg, req, failed, task_id)`**: send one alert for the run through the notifier plugin. ## Troubleshooting ### Common issues **`too_few_points`** * A series needs at least 3 points in the window the message names, and before the cut with `validation_window`. Widen `window`, or shorten `sampling_interval`. **`other: validation failed`** * The forecast missed the measured points by more than `msre_threshold`, or the validation window holds no measured point. Check the `MSRE` line in the log, raise `msre_threshold`, tune `seasonality_mode` or `changepoint_prior_scale`, or retrain with a new `unique_suffix`. **`model_unavailable`** * A saved model file cannot be read, or the plugin directory cannot be found: set `PLUGIN_DIR`. Delete the file named in `prophet_models/` or change `unique_suffix`. **`internal: forecast failed: Changepoints must fall within training data`** * Every date in `changepoints` must lie within each series’ training history. **`bad_request: invalid holidays: ...`** * `holiday_names` holds a name Prophet reserves, such as `trend`, or `holiday_country_names` a country it does not know. **`bad_request: rows of the series ... come from several tag sets writing at the same time`** * Without `sampling_interval`, rows of several tag sets of one series would interleave. Add the tags the message names to `group_by`, pin them with `tag_filter`, or set `sampling_interval` to forecast their mean per bucket. **`bad_request: ... more forecasts than max_series`** * `group_by` or a query produced more series × fields than `max_series`. Set `series_limit` to forecast the first ones, narrow the request with `tag_filter` or `sql_filter`, or raise`max_series`. **`bad_request: ... more than max_forecast_points`** * The horizon in points, with the validation window, is longer than `max_forecast_points`; `horizon_duration` is converted by the data spacing, so a fine spacing makes many points. Shorten the horizon, set `sampling_interval`, or raise `max_forecast_points`. **`bad_request: ... clashes with a column of the stored forecast rows`** * A `group_by` tag or a query key column is named like a column the plugin writes, such as `step` or `model`. Forecast with a query that renames it, for example `SELECT time, step AS process_step, usage FROM process`, or forecast without writing. **`internal: the query failed: unsupported Arrow type for Python conversion`** * The query returns a column type the Processing Engine does not pass to plugins, such as `INT`, `REAL`, `DECIMAL`, an array or a struct. Cast it to `BIGINT`, `DOUBLE` or `VARCHAR` in the query. **`bad_request: no time column in query result`** * Alias the query’s timestamp `AS time`, for example `date_bin(INTERVAL '1 hour', time) AS time`. **`internal: forecast computed but its rows cannot be built`** * The engine’s `LineBuilder` refuses some names, such as a measurement, tag or column name with a space. Rename it in the query with an alias, or forecast without writing. **A scheduled trigger fails at the first comma** * `--trigger-arguments` splits at every comma, so a `query` or `sql_filter` with one is cut or refused; set it in the TOML file. **Slow forecasts** * Training time grows with the points of a series: bound them with `point_limit` or `sampling_interval`. * Each series trains on its own, so a request takes about as long as its series together; narrow it with `series_limit`, or reuse saved models with `model_mode: predict`. * `backtest_points` trains each series a second time. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Prophet forecasting plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/prophet_forecasting) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[forecasting](/influxdb3/enterprise/tags/forecasting/)[prophet](/influxdb3/enterprise/tags/prophet/)[machine-learning](/influxdb3/enterprise/tags/machine-learning/) | Parameter | Type | Default | Example | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | Example | Description | | source_measurement | string | required¹ | temperature | Measurement to read | | field | string | required¹ | value | Numeric field to forecast | | query | string | required¹ | SELECT * FROM temperature WHERE time > now() - INTERVAL '30 days' | SQL whose result is forecast instead of source_measurement and field ; see Forecasting a query . A query with commas goes in the TOML file | | source_database | string | the trigger’s database | demo | Database to read from; see Security | | tag_filter | string/table | none | room=Kitchen site=eu or { room = ["Kitchen", "Lounge"] } | Tags the rows must match: one value per tag in a string, one value or a list per tag in a table. Not with query | | sql_filter | string | none | value > 0 | SQL condition on tags and fields, without WHERE , AND-ed with tag_filter and inserted as is; see Security . Not with query | | group_by | string/list | none | room site or ["room", "site"] | Tags to split by; each distinct combination is its own series with its own model; see Series . Not with query | | series_limit | integer ≥ 1 | none | 5 | Forecast only the first N forecasts (series × fields) in the order of Series and skip the rest; at most max_series | | window | duration | required² | 30d | History as a time window ending now, including validation_window ; see History | | point_limit | integer ≥ 3 | required² | 2000 | History as the most recent points of each series; see History | | sampling_interval | duration | none | 1h | Resample history to the mean per bucket, also the forecast step; see Forecast step and horizon | | forecast_freq | pandas frequency | inferred | MS | Step of the forecast points over raw history; see Forecast step and horizon | | horizon_steps | integer ≥ 1 | required³ | 24 | Forecast length in points | | horizon_duration | duration | required³ | 2d | Forecast length as a duration from the last history point, rounded up to whole steps | | levels | string/list | 0.5 0.8 | 0.5 0.8 0.95 or [0.5, 0.8, 0.95] | Interval levels, each between 0 and 1 | | model | string | prophet | prophet | The only model; any other value is refused | | target_measurement | string | _forecasts. | _forecasts.temperature | Measurement the forecast is written to; required when query reads several measurements | | target_database | string | the trigger’s database | forecasts | Database the forecast is written to | | forecast_id | string | none | ops-1h | Tag forecast_id in every row, so forecasts of one series with different settings keep their own rows | | model_mode | string | train | predict | train fits a new model on every run; predict reuses the saved one; see Models | | unique_suffix | string | none | 20250619_v1 | Model version, up to 64 characters from letters, digits, . , _ and - ; required with model_mode predict | | seasonality_mode | string | additive | multiplicative | additive or multiplicative | | changepoint_prior_scale | number > 0 | 0.05 | 0.1 | Flexibility of trend changepoints | | changepoints | string/list | none | 2025-01-01 2025-06-01 or ["2025-01-01", "2025-06-01"] | ISO dates of trend changepoints, in UTC unless they name a timezone; each must fall within every series’ training history | | holiday_date_list | string/list | none | 2024-12-25 2025-12-25 or ["2024-12-25", "2025-12-25"] | ISO dates of custom holidays, in UTC unless they name a timezone, set together with holiday_names and as many | | holiday_names | string/list | none | Christmas Christmas or ["Christmas", "Christmas"] | Names matching holiday_date_list ; a name may repeat, and one with spaces needs a list; a name Prophet reserves, such as trend , is refused | | holiday_country_names | string/list | none | US or ["US"] | Country code for built-in holidays, refused if Prophet does not know it; of several only the first is used, with a warning | | validation_window | duration | none | 3d | Duration held back from training before now; see Validation | | validation_alignment | string | position | nearest | How measured and forecast points are paired: position or nearest ; see Validation | | msre_threshold | number ≥ 0 | no limit | 0.05 | Highest Mean Squared Relative Error a validated series may have | | max_series | integer ≥ 1 | 100 | 50 | Most forecasts one run may produce (series × fields); more is refused unless series_limit keeps fewer | | max_forecast_points | integer ≥ 1 | 10000 | 1000 | Most forecast points per series, counting the validation window; a longer horizon is refused | | config_file_path | string | none | prophet_forecasting_scheduler.toml | TOML file with more arguments; see Scheduled trigger TOML configuration | | Parameter | Type | Default | Example | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | Example | Description | | is_sending_alert | boolean | false | true | Send one alert per run listing the series that failed validation | | senders | string/list | none | slack sms or ["slack", "sms"] | Sender types: slack , discord , http , sms , whatsapp ; required with alerts | | notification_text | string | a built-in text | Validation failed on $measurement: $series | Template with the variables $version , $measurement , $field , $series , $start_time , $end_time , $output_measurement | | notification_path | string | notify | notify | URL path of the notifier plugin’s trigger | | port_override | integer | 8181 | 8182 | Port of the InfluxDB HTTP API, 1–65535 | | influxdb3_auth_token | string | INFLUXDB3_AUTH_TOKEN | your_token | Token for the notifier’s endpoint. TOML file or environment only | | slack_webhook_url | string | none | https://hooks.slack.com/services/... | Webhook URL; required with the slack sender | | slack_headers | string | none | eyJDb250ZW50LVR5cGUiOiAiYXBwbGljYXRpb24vanNvbiJ9 | Base64-encoded headers for the Slack webhook | | discord_webhook_url | string | none | https://discord.com/api/webhooks/... | Webhook URL; required with the discord sender | | discord_headers | string | none | eyJDb250ZW50LVR5cGUiOiAiYXBwbGljYXRpb24vanNvbiJ9 | Base64-encoded headers for the Discord webhook | | http_webhook_url | string | none | https://example.com/webhook | Webhook URL; required with the http sender | | http_headers | string | none | eyJDb250ZW50LVR5cGUiOiAiYXBwbGljYXRpb24vanNvbiJ9 | Base64-encoded headers for the HTTP webhook | | twilio_sid | string | none | ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxx | Twilio Account SID, for the sms and whatsapp senders | | twilio_token | string | none | your_auth_token | Twilio Auth Token, for the sms and whatsapp senders. TOML file or environment only; unset, the notifier reads its own TWILIO_TOKEN | | twilio_to_number | string | none | +1234567890 | Recipient number; required with the sms and whatsapp senders | | twilio_from_number | string | none | +19876543210 | Twilio sender number; required with the sms and whatsapp senders | | time | room | forecast | lower_80 | upper_80 | step | run_time | | --- | --- | --- | --- | --- | --- | --- | | time | room | forecast | lower_80 | upper_80 | step | run_time | | 2026-10-29T13:00:00 | Kitchen | 21.42 | 20.61 | 22.25 | 1 | 2026-10-29T12:00:05Z | | 2026-10-29T14:00:00 | Kitchen | 21.58 | 20.74 | 22.39 | 2 | 2026-10-29T12:00:05Z | | Parameter | Type | Default | Example | Description | | --- | --- | --- | --- | --- | | Parameter | Type | Default | Example | Description | | source_measurement | string | required¹ | "temperature" | Measurement to read | | field | string | required¹ | "value" | Numeric field to forecast | | query | string | required¹ | "SELECT time, room, value FROM temperature WHERE time > now() - INTERVAL '30 days'" | SQL whose result is forecast instead of source_measurement and field ; see Forecasting a query | | source_database | string | the trigger’s database | "demo" | Database to read from; an unknown one is refused; see Security | | tag_filter | string/object | none | {"room": ["Kitchen", "Lounge"], "site": "eu"} or "room=Kitchen site=eu" | Tags the rows must match: one value or an array of several per tag in an object, one value per tag in a string. Not with query | | sql_filter | string | none | "value > 0" | SQL condition on tags and fields, without WHERE , AND-ed with tag_filter and inserted as is; see Security . Not with query | | group_by | string/list | none | ["room", "site"] or "room site" | Tags to split by; each distinct combination is its own series with its own model; see Series . Not with query | | series_limit | integer ≥ 1 | none | 5 | Forecast only the first N forecasts (series × fields) in the order of Series and skip the rest; at most max_series | | window | duration | required² | "30d" | History as a time window ending now, including validation_window ; see History | | point_limit | integer ≥ 3 | required² | 2000 | History as the most recent points of each series; see History | | start_time | ISO 8601 | required² | "2025-05-20T00:00:00Z" | Start of an absolute history, with a timezone. Set with end_time . Not with query | | end_time | ISO 8601 | required² | "2025-06-19T00:00:00Z" | End of an absolute history, with a timezone; the forecast continues from the last point before it; see History | | sampling_interval | duration | none | "1h" | Resample history to the mean per bucket, also the forecast step; see Forecast step and horizon | | forecast_freq | pandas frequency | inferred | "MS" | Step of the forecast points over raw history; see Forecast step and horizon | | horizon_steps | integer ≥ 1 | required³ | 24 | Forecast length in points | | horizon_duration | duration | required³ | "2d" | Forecast length as a duration from the last history point, rounded up to whole steps | | levels | string/list | [0.5, 0.8] | [0.5, 0.8, 0.95] or "0.5 0.8 0.95" | Interval levels, each between 0 and 1 | | model | string | "prophet" | "prophet" | The only model; any other value is refused | | write_results | boolean | false | true | Also write the forecast; see Output data structure | | target_measurement | string | _forecasts. | "_forecasts.temperature" | Measurement a written forecast goes to; required when query reads several measurements | | target_database | string | the trigger’s database | "forecasts" | Database a written forecast goes to | | forecast_id | string | none | "ops-1h" | Tag forecast_id in every written row, so forecasts of one series with different settings keep their own rows | | model_mode | string | "train" | "predict" | train fits a new model per request; predict reuses the saved one; see Models | | unique_suffix | string | none | "20250619_v1" | Model version, up to 64 characters from letters, digits, . , _ and - ; required with model_mode predict | | seasonality_mode | string | "additive" | "multiplicative" | additive or multiplicative | | changepoint_prior_scale | number > 0 | 0.05 | 0.1 | Flexibility of trend changepoints | | changepoints | string/list | none | ["2025-01-01", "2025-06-01"] or "2025-01-01 2025-06-01" | ISO dates of trend changepoints, in UTC unless they name a timezone; each must fall within every series’ training history | | holiday_date_list | string/list | none | ["2024-12-25", "2025-12-25"] or "2024-12-25 2025-12-25" | ISO dates of custom holidays, in UTC unless they name a timezone, set together with holiday_names and as many | | holiday_names | string/list | none | ["Christmas", "Christmas"] or "Christmas Christmas" | Names matching holiday_date_list ; a name may repeat, and one with spaces needs a list; a name Prophet reserves, such as trend , is refused | | holiday_country_names | string/list | none | ["US"] or "US" | Country code for built-in holidays, refused if Prophet does not know it; of several only the first is used, with a warning | | validation_window | duration | none | "3d" | Duration held back from training before now or end_time ; see Validation . Not with backtest_points | | validation_alignment | string | "position" | "nearest" | How measured and forecast points are paired: position or nearest ; see Validation | | msre_threshold | number ≥ 0 | no limit | 0.05 | Highest Mean Squared Relative Error a validated series may have | | backtest_points | integer ≥ 1 | none | 12 | Hold out the latest N history points and return each series’ backtest , at most max_forecast_points ; see Backtest . Not with validation_window | | max_series | integer ≥ 1 | 100 | 50 | Most forecasts the request may produce (series × fields); more is refused unless series_limit keeps fewer | | max_forecast_points | integer ≥ 1 | 10000 | 1000 | Most forecast points per series, counting the validation window; a longer horizon is refused | | Set by the request | Replaces, set on the trigger | | --- | --- | | Set by the request | Replaces, set on the trigger | | query | source_measurement , field , tag_filter , sql_filter , group_by | | horizon_steps | horizon_duration | | start_time , end_time | window , point_limit | | forecast_freq | sampling_interval | | Field | Description | | --- | --- | | Field | Description | | model , levels | What ran; every point holds lower_ and upper_ for each level | | series_total | Forecasts (series × fields) the request selects before series_limit | | results[].field , tags | The forecast field and the series’ tags; see Series | | results[].points_read | History points the model was trained on | | results[].step_ms | Spacing between forecast points, in milliseconds | | results[].forecast[].timestamp | The point’s time, in epoch milliseconds | | results[].forecast[].forecast | The central forecast, Prophet’s yhat | | results[].backtest | With backtest_points : the check on the latest history; see Backtest | | results[].status , code , message | Present only when this series failed, which leaves the request 200 | | Code | HTTP | When | | --- | --- | --- | | Code | HTTP | When | | bad_request | 400 | Unknown or invalid parameter, unknown measurement, field or database, a query that does not plan, too many forecasts or forecast points, a series mixing tag sets without sampling_interval | | internal | 500 | A query or the write failed, or a forecast row cannot be built | | too_few_points | — | Per series, in results , with the request still 200 | | other | — | Per series, in results : the series failed validation | | model_unavailable | — | Per series, in results : its saved model cannot be read | | internal | — | Per series, in results : training or predicting this series failed | | Set | History the model reads | Step of the forecast | | --- | --- | --- | | Set | History the model reads | Step of the forecast | | sampling_interval | The mean per bucket, aligned like date_bin , without the bucket still filling | The interval | | forecast_freq | The raw points | The pandas frequency: fixed ( 30min , 1h ) or calendar ( D , W-SUN , MS , QS , B ) | | neither | The raw points | The frequency pandas infers from the data, or else their median spacing | --- # Resampler plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/resampler/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later (uses the synchronous write API). > > The Resampler Plugin interpolates time series with non-uniform timestamps (sensor timestamp jitter) onto a uniform time grid and writes the result to a separate measurement. Uniform sampling is required by most signal-processing and forecasting algorithms (FIR/IIR filters, FFT, ML models). Numeric fields are interpolated; all other fields (strings, booleans) are carried onto the same grid points by last known value, so the full data set is preserved. A separate `snap` mode skips interpolation entirely: each point’s timestamp is rounded to the nearest grid node and all values and types stay unchanged (on node collisions the latest point wins). Each run processes a sliding window of history. Output timestamps are exact multiples of the grid interval, so overlapping runs overwrite the same points idempotently and late-arriving data is picked up automatically. Every unique tag combination is resampled as an independent series, with all tags preserved. The plugin does not fill large data gaps: when two neighboring source points are more than `max_gap` apart, grid points between them are not written, so sensor outages stay visible in the output. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. This plugin supports TOML configuration files, which can be specified using the `config_file_path` parameter. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |--------------------|------|--------|-----------------------------------------------------------| | `measurement` |string|required| Source measurement with non-uniform timestamps | |`target_measurement`|string|required|Output measurement; must differ from the source measurement| ### Grid parameters |Parameter | Type |Default| Description | |----------|------|-------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`interval`|string| `1s` |Uniform grid step, at least `1ms`. Units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`. (`window` + `max_gap`)/`interval` is capped at 1,000,000 grid points per run, so sub-second grids need a proportionally small `window`| | `window` |string|`10min`| How much history each run processes; at least one `interval` | | `offset` |string| `0s` | Processing delay for late-arriving data | ### Processing parameters | Parameter | Type | Default | Description | |----------------------|------|--------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `mode` |string|`interpolate` | `interpolate` recalculates numeric fields at grid nodes; `snap` only rounds timestamps to the nearest node, values and types unchanged (`interpolation_method` and `max_gap` are ignored) | | `fields` |string| all numeric | Space-separated numeric fields to interpolate; numeric fields not listed are dropped. In snap mode: fields of any type to carry (default all) | | `excluded_fields` |string| none | Space-separated fields of any type to exclude from the output; listing a field in both `fields` and `excluded_fields` is an error | |`interpolation_method`|string| `linear` |`linear`, `nearest`, `cubic`, `previous`, or `next`. Use `previous` (last known value) for step-like signals such as states or counters. `nearest`/`previous`/`next` return source values unchanged (integer fields keep their type and precision); `linear`/`cubic` always produce float values| | `max_gap` |string|2 × `interval`| Source-point spacing above which grid points in between are not written; at least one `interval` | ### Output parameters | Parameter | Type | Default | Description | |-----------------|-------|----------|---------------------------------------| |`target_database`|string |trigger DB|Optional target database for the output| | `max_retries` |integer| `5` | Maximum number of write attempts | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|---------------------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (replaces trigger arguments entirely when set)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. #### Example TOML configuration [resampler\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/resampler/resampler_config_scheduler.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Data requirements Interpolation needs at least 2 source points per series inside the window; a grid node is written only when it has a source point on each side at most`max_gap` apart (or an exact source match). Snap mode has no such requirement. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages** (declared in `manifest.toml`): * `influxdata-plugin-utils>=0.3.0` * `scipy` (installs `numpy`) ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.3.0" influxdb3 install package scipy ``` ## Trigger setup ### Scheduled resampling ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/resampler/resampler.py \ --trigger-spec "every:10s" \ --trigger-arguments 'measurement=signal,target_measurement=signal_resampled,interval=1s,window=1min' \ resampler_trigger ``` ### Scheduled resampling with TOML configuration ```bash # Copy and edit the configuration file cp resampler_config_scheduler.toml $PLUGIN_DIR/resampler_config.toml influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/resampler/resampler.py \ --trigger-spec "every:10s" \ --trigger-arguments config_file_path=resampler_config.toml \ resampler_trigger ``` ## Example usage ### Example 1: Linear interpolation onto a 1-second grid ```bash # Create the trigger influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/resampler/resampler.py \ --trigger-spec "every:10s" \ --trigger-arguments 'measurement=signal,target_measurement=signal_resampled,interval=1s,window=10min' \ resampler_trigger # Query resampled data (after trigger runs) influxdb3 query --database mydb "SELECT * FROM signal_resampled" ``` Source data with timestamp jitter: ``` signal value=39.1 1750000000083000000 # 00.083 signal value=40.2 1750000000947000000 # 00.947 signal value=41.0 1750000002114000000 # 02.114 ``` ### Expected output ``` signal_resampled value=40.24 1750000001000000000 # 01.000 signal_resampled value=40.92 1750000002000000000 # 02.000 ``` Grid points inside gaps larger than `max_gap` are not written and can be filled by a downstream fill plugin. Non-numeric fields follow the same gap rules but are carried by last known value instead of being interpolated. ### Example 2: Snap mode ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/resampler/resampler.py \ --trigger-spec "every:10s" \ --trigger-arguments 'measurement=signal,target_measurement=signal_resampled,interval=1s,mode=snap' \ resampler_snap ``` ### Expected output Timestamps are aligned to the grid; values and types are unchanged: ``` signal_resampled value=39.1 1750000000000000000 # 00.083 -> 00.000 signal_resampled value=40.2 1750000001000000000 # 00.947 -> 01.000 signal_resampled value=41.0 1750000002000000000 # 02.114 -> 02.000 ``` ## Code overview ### Files * `resampler.py`: The main plugin code containing the handler for scheduled resampling * `resampler_config_scheduler.toml`: Example TOML configuration file ### Logging Logs are stored in the `_internal` database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'resampler_trigger' ORDER BY event_time DESC LIMIT 20" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Entry point for the scheduled trigger. Parses and validates the configuration, queries the source window, groups rows by tag set, resamples or snaps each series, and writes the result with retries. #### `resample_series(...)` Resamples all fields of one series: numeric fields selected for interpolation are recalculated at grid nodes (`resample_field`), all other carried fields use last known value (`carry_field_previous`), with the same `max_gap` rules. #### `snap_series(...)` Rounds each source point’s timestamp to the nearest grid node, keeping values and types unchanged; on node collisions the latest point wins per field. ## Troubleshooting ### Common issues #### Issue: No output data **Solution**: Check that the window contains at least 2 source points per series and that `interval` is not larger than `window`. #### Issue: Missing points near “now” **Solution**: The freshest grid points need a source neighbor on the right; they are written by the next run once that neighbor arrives. Increase `offset`to delay processing instead. #### Issue: Holes in the output **Solution**: Source gaps larger than `max_gap` are preserved by design. Increase `max_gap` to interpolate across larger gaps. #### Issue: `cubic` falls back to linear **Solution**: Cubic interpolation needs at least 4 points per series in the window; add more data or use `linear`. #### Issue: A numeric field is missing from the output **Solution**: `fields` is set and does not list it — numeric fields not listed there are dropped by design. #### Issue: Duplicate-looking values in snap mode **Solution**: Two source points rounded to the same grid node — the later one wins per field. Use a smaller `interval` to keep them apart. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Resampler plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/resampler) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Source measurement with non-uniform timestamps | | target_measurement | string | required | Output measurement; must differ from the source measurement | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | interval | string | 1s | Uniform grid step, at least 1ms . Units: us , ms , s , min , h , d , w . ( window + max_gap )/ interval is capped at 1,000,000 grid points per run, so sub-second grids need a proportionally small window | | window | string | 10min | How much history each run processes; at least one interval | | offset | string | 0s | Processing delay for late-arriving data | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | mode | string | interpolate | interpolate recalculates numeric fields at grid nodes; snap only rounds timestamps to the nearest node, values and types unchanged ( interpolation_method and max_gap are ignored) | | fields | string | all numeric | Space-separated numeric fields to interpolate; numeric fields not listed are dropped. In snap mode: fields of any type to carry (default all) | | excluded_fields | string | none | Space-separated fields of any type to exclude from the output; listing a field in both fields and excluded_fields is an error | | interpolation_method | string | linear | linear , nearest , cubic , previous , or next . Use previous (last known value) for step-like signals such as states or counters. nearest / previous / next return source values unchanged (integer fields keep their type and precision); linear / cubic always produce float values | | max_gap | string | 2 × interval | Source-point spacing above which grid points in between are not written; at least one interval | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | target_database | string | trigger DB | Optional target database for the output | | max_retries | integer | 5 | Maximum number of write attempts | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (replaces trigger arguments entirely when set) | --- # River anomaly detector plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/river-anomaly-detector/ An adaptive InfluxDB 3 Enterprise Processing Engine plugin that detects anomalies in incoming time-series data using the [River ML](https://riverml.xyz) library. The detector automatically adapts its detection strategy based on data characteristics learned by the companion auto-profiler plugin. ## Features * **Adaptive detection:** Automatically selects detectors (Z-Score, Seasonal, ADWIN) based on profiler’s pattern classification * **Zero-config:** Works out of the box with sensible defaults — just create a trigger and go * **Auto-tune by default:** Reads profiler recommendations when available, falls back to conservative defaults otherwise * **Always-learn detectors:** Rolling, seasonal, and ADWIN detectors all learn from every observation; only the ones active in the current mode contribute votes to the anomaly decision * **Seasonal awareness:** Learns hour-of-day (24 buckets) or hour-of-week (168 buckets) patterns — activated only when the profiler detects seasonality * **Drift detection:** Uses ADWIN algorithm for trending data — activated only when profiler detects trends * **Hysteresis on mode changes:** Detector mode only switches after 3 consecutive same-mode recommendations to prevent flip-flopping * **Configurable combination:** Combine detector votes with “any” (OR), “majority”, or “all” (AND) logic * **Per-series models:** Each unique combination of table + tags + field gets its own detector set * **Online learning:** Models update incrementally with each observation — no batch retraining needed * **LRU eviction:** Configurable limit on tracked series to control memory usage * **Model persistence:** Models are pickled, compressed, chunked, and checkpointed to the database so they survive server restarts ## Prerequisites * InfluxDB 3 Enterprise with Processing Engine enabled * River ML library (`river>=0.23.0`) ## Quick Start ### 1. Install River ```bash influxdb3 install package river ``` ### 2. Create a trigger (zero-config) ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_anomaly_detector.py \ --trigger-spec "all_tables" \ anomaly_detector ``` That’s it! The plugin will start monitoring all numeric fields in all tables. With `auto_tune=true` (default), it reads profiler recommendations when available. ### 3. Query anomalies ```sql SELECT * FROM "_anomalies.cpu" ORDER BY time DESC LIMIT 10 ``` ## Customized Example Monitor specific fields with explicit detector overrides: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_anomaly_detector.py \ --trigger-spec "table:cpu" \ --trigger-arguments 'include_fields=temperature humidity' 'rolling_std_threshold=3.0' 'enable_seasonal=true' \ anomaly_detector_cpu ``` ## Trigger Arguments Arguments can be passed inline (space-separated lists) or via TOML config file (native TOML lists). Parameter names are the same in both modes. | Argument |Required|Default| Description | |-----------------------------|--------|-------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| | `include_fields` | No | `""` | Fields to monitor (space-separated). If set, only these numeric fields are processed | | `exclude_fields` | No | `""` | Fields to exclude from detection (space-separated) | | `exclude_tables` | No | `""` | Tables to skip when using `all_tables` trigger (space-separated) | | `string_fields` | No | `""` | String columns that are fields, not tags (space-separated). All other strings become tags | | `rolling_std_threshold` | No | `5.0` | Standard deviations from the mean to flag as anomaly | | `ew_fading_factor` | No | `0.3` | Fading factor for exponentially weighted stats (lower = longer memory) | | `combination_mode` | No | `any` | How to combine detector votes: `any` (OR), `majority`, or `all` (AND) | | `enable_seasonal` | No | `""` | Force seasonal detection: `true`/`false` to override, empty = auto from profiler | | `enable_adwin` | No | `""` | Force ADWIN drift detection: `true`/`false` to override, empty = auto from profiler | | `detector_mode` | No | `""` |Explicitly set detector mode components (space-separated or TOML list). Overrides profiler’s mode recommendation. See [Adaptive Detection](#adaptive-detection)| | `adwin_delta` | No |`0.002`| ADWIN sensitivity parameter. Lower values = more sensitive to drift | | `seasonal_fading_factor` | No | `0.1` | Fading factor for seasonal bucket stats (lower = longer memory) | | `seasonal_period` | No | `""` | Seasonal bucket period: `hourly` (24 buckets, hour-of-day) or `weekly` (168 buckets, hour-of-week). Empty = auto from profiler, falls back to `hourly` | | `seasonal_threshold` | No | `3.0` | Std devs from seasonal bucket mean to flag as anomaly | | `min_seasonal_observations` | No | `5` | Min observations per bucket before contributing to anomaly detection | | `min_rolling_observations` | No | `10` | Min total observations per series before the rolling Z-score starts scoring (cold-start guard) | | `max_series` | No |`1000` | Max unique series to track (LRU eviction) | | `auto_tune` | No |`true` | Read per-series parameters from `_meta.series_profiles` (requires auto-profiler plugin) | | `tune_refresh_interval` | No | `100` | Observations per series between auto-tune refreshes from `_meta.series_profiles` | | `log_anomalies` | No |`true` | Log detected anomalies to server log | |`checkpoint_interval_seconds`| No |`1800` | Seconds between model checkpoints to the database | | `max_checkpoint_age_hours` | No | `24` | Ignore checkpoints older than this when restoring | |`min_checkpoint_observations`| No | `10` | Skip checkpointing models with fewer observations than this (avoids persisting cold-start state) | | `config_file_path` | No | — | Path to TOML config file. Supports absolute paths or relative paths (resolved via PLUGIN\_DIR) | ## Column Classification The plugin classifies each column in incoming data: 1. `time` — skipped 2. String columns listed in `string_fields` — skipped from anomaly detection and not used as tags in the series key 3. Other string columns — treated as **tags** (used in series key) 4. Numeric columns (`int`, `float`, not `bool`) — treated as **fields** for anomaly detection * Filtered by `include_fields` (if set, only listed fields are monitored) * Filtered by `exclude_fields` (excluded fields are skipped) NaN and infinity values are skipped with a warning. An anomaly whose tag value cannot be written as line protocol (empty, containing a newline or tab, or ending with a backslash) is skipped with a warning. Column types are inferred from the first row of each batch. ## Table Filtering All tables starting with `_` are automatically skipped (for example, `_anomalies.*`, `_meta.*`, `_system.*`, `_forecasts.*`). Additional tables can be excluded via `exclude_tables`. ## Adaptive Detection `detector_mode` is a space-separated list of components. Exactly one **base** component selects the Z-score behavior; optional **add-ons** enable seasonal and ADWIN voting: **Base components** (pick one, reflects profiler’s pattern classification): | Base Mode | Used For | |---------------------|--------------------------------| |`zscore_conservative`| No profile yet (default) | | `zscore_low` | Stable data (lower threshold) | | `zscore_high` | Noisy data (higher threshold) | | `zscore_adaptive` |Bursty data (adaptive threshold)| **Add-on components** (zero or more): |Component | Effect | |----------|---------------------------------------------------------------| |`seasonal`| Enables seasonal-bucket voting on top of the base Z-score | | `adwin` |Enables ADWIN drift-detection voting on top of the base Z-score| Examples: `zscore_conservative`, `zscore_low seasonal`, `zscore_high adwin`, `zscore_conservative seasonal adwin`. > > > **Note:** All detectors always *learn* from every observation regardless of mode. The mode only controls which detectors *vote* on the anomaly decision. > > ### How Adaptation Works Mode selection: 1. **Explicit `detector_mode`** — if set, overrides the base mode from the profiler 2. **Auto-tune** (`auto_tune=true`) — reads `recommended_detector_mode` from `_meta.series_profiles` 3. **Model default** — `zscore_conservative` After the base mode is determined, `enable_seasonal` and `enable_adwin` overrides are applied on top (can add or remove the `seasonal` / `adwin` components from any base mode). Even when `detector_mode` is explicitly set, the profiler is still consulted (with `auto_tune=true`) for per-series `threshold`, `ew_fading_factor`, `seasonal_fading_factor`, and `seasonal_period` — only the mode itself is overridden. **Hysteresis:** `detector_mode` only switches after **3 consecutive fresh profiler recommendations** of the same new mode (one recommendation = one `tune_refresh_interval` query). Non-mode fields (`threshold`, `fading_factor`, `seasonal_fading`, `seasonal_period`) update on every refresh without hysteresis. This prevents flip-flopping on noisy pattern classifications while still letting thresholds and fading factors track the data. **Seasonality readiness gate:** If the profiler recommends `seasonal` but hasn’t marked `seasonality_ready=true` yet, the `seasonal` component is stripped before the model is configured (so the model doesn’t score against an immature seasonal grid). ### Detector Details **Rolling Z-Score (always active for scoring):** Exponentially weighted mean and variance track the “normal” range. When a value exceeds `mean ± N*std` (configurable threshold), it’s flagged. Starts scoring after `min_rolling_observations` observations (default 10) to reduce false positives during cold start. When the fading factor changes via auto-tune, the EW stats are re-seeded from the prior mean/std to avoid a cold restart. **Seasonal:** Time-based buckets, each maintaining its own EW mean and variance. Values are compared against their bucket’s learned pattern. The bucketing scheme is controlled by `seasonal_period`: * `hourly` — 24 buckets (H00–H23), one per hour-of-day * `weekly` — 168 buckets (Mon-H00 … Sun-H23), one per hour-of-week Each bucket matures independently after `min_seasonal_observations` (default 5) data points. Buckets always learn; they only *vote* when `seasonal` is part of the detector mode. Changing `seasonal_period` at runtime resets the seasonal grid. **ADWIN:** River ML’s Adaptive Windowing algorithm on raw values. Detects when the statistical properties of the data stream change (concept drift). Sensitivity is controlled by `adwin_delta` (default 0.002; lower = more sensitive). ADWIN always learns but only votes when `adwin` is part of the detector mode. ### Combination Modes The `combination_mode` parameter controls how active detector votes are combined: * `any` (default): Anomaly if **any** active detector flags it (OR logic) * `majority`: Anomaly if **at least half** of active detectors flag it (⌈n/2⌉ votes). With 2 active detectors, 1 flag is enough; with 3, at least 2 are required * `all`: Anomaly if **all** active detectors flag it (AND logic) Only detectors that are active in the current mode participate in voting. ## Output Schema Anomalies are written to `_anomalies.{source_table}`. **Tags:** All original tags preserved + `field_name` **Fields:** | Field | Type | Description | |-----------------------|-------|-----------------------------------------------------------------------------------| | `original_value` | float | The value that triggered detection | | `is_anomaly` |boolean| True if detector combination flagged | | `observations` |integer| Total observations for this series | | `detector_mode` |string | Active detector mode (for example, `zscore_conservative seasonal`) | | `rolling_anomaly` |boolean| Rolling stats detector flagged | | `rolling_mean` | float | Current exponentially weighted mean | | `rolling_std` | float | Current exponentially weighted std deviation | | `rolling_deviation` | float | How many std devs from mean | | `rolling_threshold` | float | Configured std dev threshold | | `seasonal_anomaly` |boolean| Seasonal detector flagged | | `seasonal_mature` |boolean| Whether bucket has enough observations | | `seasonal_mean` | float | Exponentially weighted mean for this bucket | | `seasonal_std` | float | Exponentially weighted std dev for this bucket | | `seasonal_deviation` | float | How many std devs from seasonal bucket mean | | `seasonal_bucket` |string |Bucket identifier, for example, `H14` (2pm UTC) for hourly, or `Mon-H14` for weekly| |`seasonal_observations`|integer| Observation count in this bucket | | `adwin_anomaly` |boolean| ADWIN drift detector flagged | | `drift_detected` |boolean| Whether ADWIN detected a concept drift | Only rows where `is_anomaly=true` are written. Seasonal bucket fields (`seasonal_mean`, `seasonal_std`, `seasonal_deviation`, `seasonal_bucket`, `seasonal_observations`) appear once the seasonal grid has accumulated enough data for the current bucket, regardless of whether the `seasonal` component is active — because seasonal stats always learn. `seasonal_anomaly` and `adwin_anomaly` only flip to `true` when their respective components are part of the active detector mode. ## Model Persistence The plugin automatically checkpoints and restores models so they survive server restarts. ### Checkpoint Every `checkpoint_interval_seconds` (default: 1800 = 30 minutes), the plugin pickles each per-series model, compresses with zlib, base64-encodes, and writes to `_system.model_checkpoints`. Large models are automatically chunked across multiple rows (60KB chunks). Models with fewer than `min_checkpoint_observations` observations (default: 10) are skipped — there is no point persisting cold-start state that would be rebuilt from scratch on restore anyway. ### Restore On the first invocation after a server restart (detected by an empty model cache), the plugin queries `_system.model_checkpoints` for the latest checkpoint per series. Checkpoints older than `max_checkpoint_age_hours` (default: 24) are ignored. Restored models resume scoring immediately with their full learned history. ### Checkpoint Schema (`_system.model_checkpoints`) | Column | Type | Description | |-----------------------|-------|---------------------------------------------------| | `plugin` | tag | `"river_anomaly_detector"` | | `series_key` | tag | Unique series identifier | | `chunk_index` | tag | 0-based chunk index (for multi-row models) | | `chunk_total` | tag | Total chunks for this model | | `model_data` |string |Base64-encoded, zlib-compressed pickle of the model| | `model_type` |string | `"SeriesModel"` | | `observation_count` |integer| Observations the model has processed | |`checkpoint_size_bytes`|integer| Size of the compressed data | ## TOML Configuration ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_anomaly_detector.py \ --trigger-spec "all_tables" \ --trigger-arguments config_file_path=river_anomaly_detector_config.toml \ anomaly_detector ``` Or with an absolute path (no PLUGIN\_DIR needed): ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_anomaly_detector.py \ --trigger-spec "all_tables" \ --trigger-arguments config_file_path=/etc/influxdb3/river_anomaly_detector_config.toml \ anomaly_detector ``` See `river_anomaly_detector_config.toml` for an example configuration. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [River anomaly detector plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/river_anomaly_detector) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Argument | Required | Default | Description | | --- | --- | --- | --- | | Argument | Required | Default | Description | | include_fields | No | "" | Fields to monitor (space-separated). If set, only these numeric fields are processed | | exclude_fields | No | "" | Fields to exclude from detection (space-separated) | | exclude_tables | No | "" | Tables to skip when using all_tables trigger (space-separated) | | string_fields | No | "" | String columns that are fields, not tags (space-separated). All other strings become tags | | rolling_std_threshold | No | 5.0 | Standard deviations from the mean to flag as anomaly | | ew_fading_factor | No | 0.3 | Fading factor for exponentially weighted stats (lower = longer memory) | | combination_mode | No | any | How to combine detector votes: any (OR), majority , or all (AND) | | enable_seasonal | No | "" | Force seasonal detection: true / false to override, empty = auto from profiler | | enable_adwin | No | "" | Force ADWIN drift detection: true / false to override, empty = auto from profiler | | detector_mode | No | "" | Explicitly set detector mode components (space-separated or TOML list). Overrides profiler’s mode recommendation. See Adaptive Detection | | adwin_delta | No | 0.002 | ADWIN sensitivity parameter. Lower values = more sensitive to drift | | seasonal_fading_factor | No | 0.1 | Fading factor for seasonal bucket stats (lower = longer memory) | | seasonal_period | No | "" | Seasonal bucket period: hourly (24 buckets, hour-of-day) or weekly (168 buckets, hour-of-week). Empty = auto from profiler, falls back to hourly | | seasonal_threshold | No | 3.0 | Std devs from seasonal bucket mean to flag as anomaly | | min_seasonal_observations | No | 5 | Min observations per bucket before contributing to anomaly detection | | min_rolling_observations | No | 10 | Min total observations per series before the rolling Z-score starts scoring (cold-start guard) | | max_series | No | 1000 | Max unique series to track (LRU eviction) | | auto_tune | No | true | Read per-series parameters from _meta.series_profiles (requires auto-profiler plugin) | | tune_refresh_interval | No | 100 | Observations per series between auto-tune refreshes from _meta.series_profiles | | log_anomalies | No | true | Log detected anomalies to server log | | checkpoint_interval_seconds | No | 1800 | Seconds between model checkpoints to the database | | max_checkpoint_age_hours | No | 24 | Ignore checkpoints older than this when restoring | | min_checkpoint_observations | No | 10 | Skip checkpointing models with fewer observations than this (avoids persisting cold-start state) | | config_file_path | No | — | Path to TOML config file. Supports absolute paths or relative paths (resolved via PLUGIN_DIR) | | Base Mode | Used For | | --- | --- | | Base Mode | Used For | | zscore_conservative | No profile yet (default) | | zscore_low | Stable data (lower threshold) | | zscore_high | Noisy data (higher threshold) | | zscore_adaptive | Bursty data (adaptive threshold) | | Component | Effect | | --- | --- | | Component | Effect | | seasonal | Enables seasonal-bucket voting on top of the base Z-score | | adwin | Enables ADWIN drift-detection voting on top of the base Z-score | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | original_value | float | The value that triggered detection | | is_anomaly | boolean | True if detector combination flagged | | observations | integer | Total observations for this series | | detector_mode | string | Active detector mode (for example, zscore_conservative seasonal ) | | rolling_anomaly | boolean | Rolling stats detector flagged | | rolling_mean | float | Current exponentially weighted mean | | rolling_std | float | Current exponentially weighted std deviation | | rolling_deviation | float | How many std devs from mean | | rolling_threshold | float | Configured std dev threshold | | seasonal_anomaly | boolean | Seasonal detector flagged | | seasonal_mature | boolean | Whether bucket has enough observations | | seasonal_mean | float | Exponentially weighted mean for this bucket | | seasonal_std | float | Exponentially weighted std dev for this bucket | | seasonal_deviation | float | How many std devs from seasonal bucket mean | | seasonal_bucket | string | Bucket identifier, for example, H14 (2pm UTC) for hourly, or Mon-H14 for weekly | | seasonal_observations | integer | Observation count in this bucket | | adwin_anomaly | boolean | ADWIN drift detector flagged | | drift_detected | boolean | Whether ADWIN detected a concept drift | | Column | Type | Description | | --- | --- | --- | | Column | Type | Description | | plugin | tag | "river_anomaly_detector" | | series_key | tag | Unique series identifier | | chunk_index | tag | 0-based chunk index (for multi-row models) | | chunk_total | tag | Total chunks for this model | | model_data | string | Base64-encoded, zlib-compressed pickle of the model | | model_type | string | "SeriesModel" | | observation_count | integer | Observations the model has processed | | checkpoint_size_bytes | integer | Size of the compressed data | --- # River auto profiler plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/river-auto-profiler/ A zero-config InfluxDB 3 Enterprise Processing Engine plugin that incrementally profiles incoming time-series data, classifies data patterns, and writes recommended anomaly detection parameters per series. This is the “zero-config” enabler — it learns about your data so the anomaly detector can auto-tune. ## Features * **Incremental profiling:** Uses River ML’s streaming statistics to build profiles one observation at a time * **Pattern classification:** Automatically classifies data as stable, noisy, trending, seasonal, or bursty * **Adaptive recommendations:** Recommends detector mode, threshold, and fading factors based on data characteristics * **Calibrated EW stats:** Collects initial observations to determine optimal fading factor before creating EW statistics, then replays calibration data * **Exceedance-calibrated thresholds:** Automatically adjusts anomaly thresholds based on observed data distribution (targets \~1% anomaly rate) * **Per-series profiles:** Each unique combination of table + tags + field gets its own profile * **Low overhead:** Only writes profiles every N observations (default 50) or every 5 minutes for slow data * **LRU eviction:** Configurable limit on tracked series to control memory usage ## How It Works On every write, the profiler iterates over each numeric field in each row and updates streaming statistics for a unique series, keyed by `table + tag values + field name`. Tracked stats include EW mean/variance, skewness, kurtosis, write interval, and seasonal variance buckets (hourly or weekly). Each series starts in a short **calibration phase** (30 observations). During calibration values are buffered; once it ends, the profiler uses the observed write interval to pick an appropriate fading factor and replays the buffered values through the EW stats so the profile starts off already informed by early history. After calibration, every `profile_write_interval` observations (or every 5 minutes for slow streams) the plugin writes a profile snapshot to `_meta.series_profiles` — pattern label, recommended detector mode, threshold, and fading factors. Profiles with fewer than `min_observations` are flagged `profile_mature=false`, so downstream consumers can fall back to safe defaults until the profile has seen enough data. ## Prerequisites * InfluxDB 3 Enterprise with Processing Engine enabled * River ML library (`river>=0.23.0`) ## Quick Start ### 1. Install River (if not already installed) ```bash influxdb3 install package river ``` ### 2. Create the profiler trigger ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_auto_profiler.py \ --trigger-spec "all_tables" \ auto_profiler ``` ### 3. Query profiles ```sql SELECT * FROM "_meta.series_profiles" ORDER BY time DESC LIMIT 10 ``` ## Trigger Arguments Arguments can be passed inline (space-separated lists) or via TOML config file (native TOML lists). Parameter names are the same in both modes. | Argument |Required| Default | Description | |-----------------------------|--------|----------|--------------------------------------------------------------------------------------------------| | `include_fields` | No | `""` | Fields to profile (space-separated). If set, only these numeric fields are processed | | `exclude_fields` | No | `""` | Fields to exclude from profiling (space-separated) | | `exclude_tables` | No | `""` | Tables to skip when using `all_tables` trigger (space-separated) | | `string_fields` | No | `""` | String columns that are fields, not tags (space-separated). All other strings become tags | | `max_series` | No | `1000` | Max unique series to track (LRU eviction) | | `initial_fading_factor` | No | `0.3` | Initial fading factor for EW stats. Auto-adapted based on write frequency after calibration | | `seasonal_period` | No |`"hourly"`| Seasonal period for variance buckets: `"hourly"` (24 buckets) or `"weekly"` (168 buckets) | | `profile_write_interval` | No | `50` | Write profile every N observations per series | | `min_observations` | No | `50` | Observations needed before profile is considered mature | | `log_profiles` | No | `false` | Log profile updates to server log | |`checkpoint_interval_seconds`| No | `1800` | Seconds between model checkpoints to the database | | `max_checkpoint_age_hours` | No | `24` | Ignore checkpoints older than this when restoring | |`min_checkpoint_observations`| No | `10` |Skip checkpointing profiles with fewer observations than this (avoids persisting cold-start state)| | `config_file_path` | No | — | Path to TOML config file. Supports absolute paths or relative paths (resolved via PLUGIN\_DIR) | ### Inline example ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_auto_profiler.py \ --trigger-spec "all_tables" \ --trigger-arguments 'include_fields=temperature humidity' 'exclude_tables=debug_info system_logs' \ auto_profiler ``` ### TOML config example ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_auto_profiler.py \ --trigger-spec "all_tables" \ --trigger-arguments config_file_path=river_auto_profiler_config.toml \ auto_profiler ``` Or with an absolute path (no PLUGIN\_DIR needed): ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_auto_profiler.py \ --trigger-spec "all_tables" \ --trigger-arguments config_file_path=/etc/influxdb3/river_auto_profiler_config.toml \ auto_profiler ``` See `river_auto_profiler_config.toml` for an example configuration. ## Column Classification The plugin classifies each column in incoming data: 1. `time` — skipped 2. String columns listed in `string_fields` — skipped (not a tag, not numeric) 3. Other string columns — treated as **tags** (used in series key) 4. Numeric columns (`int`, `float`, not `bool`) — treated as **fields** to profile * Filtered by `include_fields` (if set, only listed fields are profiled) * Filtered by `exclude_fields` (excluded fields are skipped) ## Table Filtering All tables starting with `_` are automatically skipped (for example, `_meta.*`, `_anomalies.*`, `_system.*`, `_forecasts.*`). Additional tables can be excluded via `exclude_tables`. ## Output Schema Profiles are written to `_meta.series_profiles`. **Tags:** `source_table`, `field_name`, plus all original tags from the source data. **Fields:** | Field | Type | Description | |-----------------------------|-------|------------------------------------------------------------------| | `observations` |integer| Total observation count | | `write_interval_seconds` | float | Average seconds between writes | | `value_mean` | float | Exponentially weighted mean | | `value_std` | float | Exponentially weighted standard deviation | | `value_min` | float | Minimum observed value | | `value_max` | float | Maximum observed value | | `coefficient_of_variation` | float | std / mean (key metric for tuning) | | `pattern_label` |string | Data pattern: stable, noisy, trending, seasonal, bursty | | `recommended_detector_mode` |string | Recommended detector mode for anomaly detector | | `seasonality_strength` | float | Seasonality strength (0.0-1.0) | | `trend_strength` | float | Trend strength (0.0-1.0) | | `data_skewness` | float | Data asymmetry (skewness) | | `data_kurtosis` | float | Heavy-tailedness (kurtosis) | | `recommended_threshold` | float |Exceedance-calibrated `rolling_std_threshold` for anomaly detector| | `recommended_fading_factor` | float | Recommended `ew_fading_factor` for anomaly detector | |`recommended_seasonal_fading`| float | Recommended seasonal fading factor | | `profile_mature` |boolean| True if observations ≥ min\_observations | | `seasonality_ready` |boolean|True if enough seasonal buckets are filled for reliable detection | | `seasonal_buckets_filled` |integer| Number of seasonal buckets with 2+ observations | ## Integration with Anomaly Detector The recommended parameters written to `_meta.series_profiles` are consumed by the companion `river_anomaly_detector` plugin when it runs with `auto_tune=true` (default). Deploying both triggers on the same database lets the detector auto-tune per series: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_auto_profiler.py \ --trigger-spec "all_tables" \ auto_profiler influxdb3 create trigger \ --database mydb \ --plugin-filename river_anomaly_detector.py \ --trigger-spec "all_tables" \ anomaly_detector ``` Until a profile matures (`profile_mature=true`, i.e. at least `min_observations` observations), the anomaly detector falls back to conservative defaults. ## Pattern Classification The profiler classifies each series into one of five patterns based on streaming statistics: | Condition |Pattern | Recommended Detector Mode | |---------------------------|--------|-----------------------------| |Seasonality strength \> 0.4|seasonal|zscore\_conservative seasonal| | Trend strength \> 0.7 |trending| zscore\_conservative adwin | | CV \< 0.05 | stable | zscore\_low | |Kurtosis \> 5 or CV \> 1.0 | bursty | zscore\_adaptive | | CV \> 0.2 | noisy | zscore\_high | | Otherwise | stable | zscore\_low | ### Seasonality Detection Uses hourly (24 buckets) or weekly (168 buckets) variance buckets. If the coefficient of variation across bucket variances is high, the data is seasonal. Requires at least 25% of buckets with 2+ observations each. The `seasonality_ready` field in the output indicates whether enough data has been collected for reliable seasonality detection. ### Trend Detection Uses EW mean drift — compares a fast EW mean (fading factor 0.3) against a slow EW mean (fading factor 0.05). Large divergence relative to the standard deviation indicates trending data. Requires at least 20 observations. ## Tuning Rules ### Threshold Recommendation (exceedance-calibrated) The threshold is calibrated automatically using exceedance rate tracking. The profiler monitors what fraction of observations fall outside `mean ± threshold × std` and adjusts the threshold to target a \~1% anomaly rate. The threshold is bounded to [2.5, 10.0] and calibrated every 200 observations. Initial value is set from pattern classification: | Data Pattern |Initial Threshold| |------------------|-----------------| | stable | 3.5 | |noisy (CV 0.2-0.5)| 6.0 | |noisy (CV \> 0.5) | 8.0 | | trending | 5.0 | | seasonal | 5.0 | | bursty | 7.0 | ### Fading Factor Recommendation (based on write frequency) |Write Interval|Recommended Factor| |--------------|------------------| |\< 10 seconds | 0.1 | | 10s - 60s | 0.2 | | 1min - 5min | 0.3 | | \> 5 minutes | 0.5 | ### Seasonal Fading Factor Recommendation |Write Interval|Recommended Factor| |--------------|------------------| |\< 60 seconds | 0.05 | | 1min - 5min | 0.1 | | \> 5 minutes | 0.2 | ## Model Persistence The plugin automatically checkpoints and restores models so they survive server restarts. ### Checkpoint Every `checkpoint_interval_seconds` (default: 1800 = 30 minutes), the plugin pickles each per-series profile, compresses with zlib, base64-encodes, and writes to `_system.model_checkpoints`. Large models are automatically chunked across multiple rows (60KB chunks) to stay within InfluxDB’s string field limit. Profiles with fewer than `min_checkpoint_observations` observations (default: 10) are skipped — there is no point persisting cold-start state that would be rebuilt from scratch on restore anyway. ### Restore On the first invocation after a server restart (detected by an empty profile cache), the plugin queries `_system.model_checkpoints` for the latest checkpoint per series. Checkpoints older than `max_checkpoint_age_hours` (default: 24) are ignored. Restored profiles resume profiling immediately with their full statistical history. ### Checkpoint Schema (`_system.model_checkpoints`) | Column | Type | Description | |-----------------------|-------|------------------------------------------| | `plugin` | tag | `"river_auto_profiler"` | | `series_key` | tag | Unique series identifier | | `chunk_index` | tag |0-based chunk index (for multi-row models)| | `chunk_total` | tag | Total chunks for this model | | `model_data` |string | Base64-encoded, zlib-compressed pickle | | `model_type` |string | `"SeriesProfile"` | | `observation_count` |integer| Observations the profile has processed | |`checkpoint_size_bytes`|integer| Size of the compressed data | ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [River auto profiler plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/river_auto_profiler) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Argument | Required | Default | Description | | --- | --- | --- | --- | | Argument | Required | Default | Description | | include_fields | No | "" | Fields to profile (space-separated). If set, only these numeric fields are processed | | exclude_fields | No | "" | Fields to exclude from profiling (space-separated) | | exclude_tables | No | "" | Tables to skip when using all_tables trigger (space-separated) | | string_fields | No | "" | String columns that are fields, not tags (space-separated). All other strings become tags | | max_series | No | 1000 | Max unique series to track (LRU eviction) | | initial_fading_factor | No | 0.3 | Initial fading factor for EW stats. Auto-adapted based on write frequency after calibration | | seasonal_period | No | "hourly" | Seasonal period for variance buckets: "hourly" (24 buckets) or "weekly" (168 buckets) | | profile_write_interval | No | 50 | Write profile every N observations per series | | min_observations | No | 50 | Observations needed before profile is considered mature | | log_profiles | No | false | Log profile updates to server log | | checkpoint_interval_seconds | No | 1800 | Seconds between model checkpoints to the database | | max_checkpoint_age_hours | No | 24 | Ignore checkpoints older than this when restoring | | min_checkpoint_observations | No | 10 | Skip checkpointing profiles with fewer observations than this (avoids persisting cold-start state) | | config_file_path | No | — | Path to TOML config file. Supports absolute paths or relative paths (resolved via PLUGIN_DIR) | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | observations | integer | Total observation count | | write_interval_seconds | float | Average seconds between writes | | value_mean | float | Exponentially weighted mean | | value_std | float | Exponentially weighted standard deviation | | value_min | float | Minimum observed value | | value_max | float | Maximum observed value | | coefficient_of_variation | float | std / mean (key metric for tuning) | | pattern_label | string | Data pattern: stable, noisy, trending, seasonal, bursty | | recommended_detector_mode | string | Recommended detector mode for anomaly detector | | seasonality_strength | float | Seasonality strength (0.0-1.0) | | trend_strength | float | Trend strength (0.0-1.0) | | data_skewness | float | Data asymmetry (skewness) | | data_kurtosis | float | Heavy-tailedness (kurtosis) | | recommended_threshold | float | Exceedance-calibrated rolling_std_threshold for anomaly detector | | recommended_fading_factor | float | Recommended ew_fading_factor for anomaly detector | | recommended_seasonal_fading | float | Recommended seasonal fading factor | | profile_mature | boolean | True if observations ≥ min_observations | | seasonality_ready | boolean | True if enough seasonal buckets are filled for reliable detection | | seasonal_buckets_filled | integer | Number of seasonal buckets with 2+ observations | | Condition | Pattern | Recommended Detector Mode | | --- | --- | --- | | Condition | Pattern | Recommended Detector Mode | | Seasonality strength > 0.4 | seasonal | zscore_conservative seasonal | | Trend strength > 0.7 | trending | zscore_conservative adwin | | CV < 0.05 | stable | zscore_low | | Kurtosis > 5 or CV > 1.0 | bursty | zscore_adaptive | | CV > 0.2 | noisy | zscore_high | | Otherwise | stable | zscore_low | | Data Pattern | Initial Threshold | | --- | --- | | Data Pattern | Initial Threshold | | stable | 3.5 | | noisy (CV 0.2-0.5) | 6.0 | | noisy (CV > 0.5) | 8.0 | | trending | 5.0 | | seasonal | 5.0 | | bursty | 7.0 | | Write Interval | Recommended Factor | | --- | --- | | Write Interval | Recommended Factor | | < 10 seconds | 0.1 | | 10s - 60s | 0.2 | | 1min - 5min | 0.3 | | > 5 minutes | 0.5 | | Write Interval | Recommended Factor | | --- | --- | | Write Interval | Recommended Factor | | < 60 seconds | 0.05 | | 1min - 5min | 0.1 | | > 5 minutes | 0.2 | | Column | Type | Description | | --- | --- | --- | | Column | Type | Description | | plugin | tag | "river_auto_profiler" | | series_key | tag | Unique series identifier | | chunk_index | tag | 0-based chunk index (for multi-row models) | | chunk_total | tag | Total chunks for this model | | model_data | string | Base64-encoded, zlib-compressed pickle | | model_type | string | "SeriesProfile" | | observation_count | integer | Observations the profile has processed | | checkpoint_size_bytes | integer | Size of the compressed data | --- # River forecaster plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/river-forecaster/ An InfluxDB 3 Enterprise Processing Engine plugin that provides online time-series forecasting using [River ML](https://riverml.xyz)’s SNARIMAX model. The plugin learns incrementally from incoming data and periodically produces multi-step-ahead forecasts, all within a single WAL trigger. ## Features * **Online learning:** SNARIMAX model updates incrementally with each observation — no batch retraining needed * **Single-trigger design:** Learns from incoming data and produces forecasts when previous predictions are fully consumed — self-throttling per model * **Auto-horizon:** Automatically determines forecast horizon from the model’s own observed write frequency * **Per-series models:** Each unique combination of table + tags + field gets its own forecast model * **Explicit table selection:** You specify which tables and fields to forecast — no surprise cardinality explosions * **LRU eviction:** Configurable limit on tracked series to control memory usage ## Prerequisites * InfluxDB 3 Enterprise with Processing Engine enabled * River ML library (`river>=0.23.0`) ## Quick Start ### 1. Install River ```bash influxdb3 install package river ``` ### 2. Create the trigger Specify which tables to forecast with `include_tables`: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_forecaster.py \ --trigger-spec "all_tables" \ --trigger-arguments "include_tables=system_cpu system_memory" \ forecaster ``` The plugin will start learning from all numeric fields in the specified tables and produce forecasts once models warm up (default: 30 observations). Each model produces a new forecast automatically once its previous predictions have been fully consumed by incoming actuals. ### 3. Query forecasts ```sql SELECT * FROM "_forecasts.system_cpu" WHERE field_name = 'idle' ORDER BY time DESC LIMIT 12 ``` ## Customized Example Forecast specific fields with tuned parameters: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_forecaster.py \ --trigger-spec "all_tables" \ --trigger-arguments "include_tables=system_cpu system_memory" "include_fields=idle used available" "max_series=100" "default_horizon=24" "log_forecasts=true" \ forecaster ``` ## Trigger Arguments | Argument |Required| Default | Description | |-----------------------------|--------|----------|------------------------------------------------------------------------------------------------------------------------------------------------| | `include_tables` |**Yes** | — | Space-separated tables to forecast (for example, `system_cpu system_memory`). In TOML config use a list. | | `include_fields` | No | *(all)* | Space-separated fields to forecast (for example, `idle used`). Omit to use all numeric fields. In TOML config use a list. | | `string_fields` | No | *(none)* |Space-separated string column names that should be treated as fields, not tags. All other string columns become tags. In TOML config use a list.| | `max_series` | No | `50` | Max unique series to track (LRU eviction) | | `min_observations` | No | `30` | Minimum observations before forecasting | | `default_horizon` | No | `12` | Fallback horizon if the model’s own write\_interval estimate is unavailable | | `forecast_target_seconds` | No | `3600` | Target forecast window for auto-horizon | | `snarimax_p` | No |`0` (auto)| SNARIMAX autoregressive order. 0 = auto-tune from write frequency (\~1h lookback, clamped 4-30) | | `snarimax_d` | No | `1` | SNARIMAX differencing order | | `snarimax_q` | No | `1` | SNARIMAX moving average order | | `calibration_count` | No | `30` |Observations buffered before finalizing SNARIMAX `p` (only used when `snarimax_p=0`). Larger = more reliable interval estimate, longer warm-up. | | `log_forecasts` | No | `false` | Log forecast details | |`checkpoint_interval_seconds`| No | `1800` | Seconds between model checkpoints to the database | | `max_checkpoint_age_hours` | No | `24` | Ignore checkpoints older than this when restoring | |`min_checkpoint_observations`| No | `10` | Skip checkpointing models with fewer observations than this (avoids persisting cold-start models) | | `config_file_path` | No | — | Path to TOML config file | ## Output Schema Forecasts are written to `_forecasts.{source_table}`. **Tags:** All original tags from the source row, plus `field_name` (name of the forecasted field) and `model` (always `"snarimax"`). **Fields:** | Field | Type | Description | |----------------|-------|------------------------------------------------| | `horizon_step` |integer| 1-based index of this step in the forecast run | |`forecast_value`| float | Predicted value for this future step | |`horizon_total` |integer| Total steps in this forecast | | `observations` |integer|How many observations the model has learned from| **Timestamps:** Each forecast row gets a future timestamp: ``` forecast_time = last_observed_time + (step * write_interval_seconds) ``` ## How It Works ### SNARIMAX Model River’s `time_series.SNARIMAX` is an online ARIMA variant. Default parameters: | Parameter | Default | Rationale | |------------------|----------|--------------------------------------------------------------------------------------------------------------| | `p` (AR order) |`0` (auto)|Auto-tuned from write frequency to capture \~1 hour of lag history (clamped 4-30). Set explicitly to override.| |`d` (differencing)| `1` | Handle non-stationary trends | | `q` (MA order) | `1` | Basic error correction | ### Calibration Phase When `snarimax_p=0` (auto — the default), the model cannot be created immediately because it does not yet know the write frequency. Instead the plugin enters a calibration phase: 1. The first `calibration_count` observations (default: 30) are buffered in memory. A smoothed write-interval estimate is tracked in parallel. 2. Once the buffer is full, the plugin chooses `p = round(3600 / write_interval_s)` (clamped to `[4, 30]`), creates the SNARIMAX model, and replays the buffered observations through it. 3. From that point on the model learns online on every incoming observation — calibration is a one-time bootstrap. Calibration happens only once per series. Increasing `calibration_count` gives a more reliable write-interval estimate at the cost of a longer warm-up. When `snarimax_p` is set to a non-zero value, calibration is skipped entirely and the model is created immediately. ### Warm-up Behavior Models need at least `min_observations` (default: 30) before producing forecasts. During warm-up, the plugin continues learning but skips the series when producing forecasts. When `snarimax_p=0`, the calibration phase counts toward the observation total: with the defaults (`calibration_count=30`, `min_observations=30`) the first forecast is produced right after calibration finishes. If `min_observations > calibration_count`, the model still needs extra observations after calibration before it starts forecasting. ### Auto-Horizon The forecast horizon — how many future steps each run predicts — is derived from each model’s own EW-smoothed write interval (tracked from incoming timestamps). The goal is to cover roughly `forecast_target_seconds` of future data per run: 1. **Primary:** `horizon = max(1, round(forecast_target_seconds / write_interval))`. With defaults (1 hour target, 10 s write interval) this gives 360 future points per forecast. 2. **Fallback:** `default_horizon` (12) — used only when the interval estimate is not yet available (e.g. immediately after a checkpoint restore, before the second observation has arrived). ### LRU Eviction When the number of tracked series exceeds `max_series`, the least recently used series is evicted. If an evicted series reappears, it starts fresh with a new warm-up period. ## Model Persistence Models are stored in `influxdb3_local.cache` (in-memory) and persist across WAL flushes. To survive server restarts, the plugin automatically checkpoints and restores them. ### Checkpoint Every `checkpoint_interval_seconds` (default: 1800 = 30 minutes), the plugin pickles each per-series model, zlib-compresses it, base64-encodes the result, and writes it to `_system.model_checkpoints`. Large checkpoints are split into chunks that fit within InfluxDB’s 64 KB string field limit. This is time-gated so it only runs once per interval regardless of how often WAL flushes trigger the plugin. Models with fewer than `min_checkpoint_observations` observations (default: 10) are skipped — there is no point persisting cold-start state that would be rebuilt from scratch on restore anyway. ### Restore On the first invocation after a server restart (detected by an empty model cache), the plugin queries `_system.model_checkpoints` for the latest checkpoint per series. Checkpoints older than `max_checkpoint_age_hours` (default: 24) are ignored. Chunks are grouped by their timestamp so that chunks from different checkpoint runs are never mixed together. Restored models resume learning and forecasting immediately. ### Checkpoint Schema (`_system.model_checkpoints`) | Column | Type | Description | |-----------------------|-------|-------------------------------------------| | `plugin` | tag | `"river_forecaster"` | | `series_key` | tag | Unique series identifier | | `chunk_index` | tag | 0-based index of this chunk | | `chunk_total` | tag |Total number of chunks for this checkpoint | | `model_data` |string |Base64-encoded zlib-compressed pickle chunk| | `model_type` |string | `"SeriesForecaster"` | | `observation_count` |integer| Observations the model has processed | |`checkpoint_size_bytes`|integer| Size of the compressed pickle | ## Forecast Evaluation After each forecast run, the plugin stores predictions in memory and compares them step-by-step against incoming actual values. A new forecast is only produced once all steps from the previous forecast have been evaluated, so MAE always covers the full forecast horizon. MAE is tracked per model in memory (and persisted via checkpoints). When `log_forecasts=true`, it is written to the plugin log after each forecast run that accumulated new evaluation points, in the form `EVAL: MAE= ( points)`. ## TOML Configuration Instead of passing trigger arguments inline, you can use a TOML config file: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename river_forecaster.py \ --trigger-spec "all_tables" \ --trigger-arguments config_file_path=river_forecaster_config.toml \ forecaster ``` See `river_forecaster_config.toml` for an example configuration. ## Query Examples ```sql -- All forecasts for a specific host and field SELECT time, horizon_step, forecast_value FROM "_forecasts.system_cpu" WHERE host = 'server01' AND field_name = 'idle' ORDER BY time DESC LIMIT 12 -- Latest forecast run for one series (all horizon steps) SELECT time, horizon_step, forecast_value, horizon_total FROM "_forecasts.system_cpu" WHERE host = 'server01' AND field_name = 'idle' AND time > now() - interval '2 hours' ORDER BY horizon_step ASC -- Recent actuals for the same series (for manual comparison against forecasts) SELECT time, idle FROM system_cpu WHERE host = 'server01' ORDER BY time DESC LIMIT 12 ``` ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [River forecaster plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/river_forecaster) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Argument | Required | Default | Description | | --- | --- | --- | --- | | Argument | Required | Default | Description | | include_tables | Yes | — | Space-separated tables to forecast (for example, system_cpu system_memory ). In TOML config use a list. | | include_fields | No | (all) | Space-separated fields to forecast (for example, idle used ). Omit to use all numeric fields. In TOML config use a list. | | string_fields | No | (none) | Space-separated string column names that should be treated as fields, not tags. All other string columns become tags. In TOML config use a list. | | max_series | No | 50 | Max unique series to track (LRU eviction) | | min_observations | No | 30 | Minimum observations before forecasting | | default_horizon | No | 12 | Fallback horizon if the model’s own write_interval estimate is unavailable | | forecast_target_seconds | No | 3600 | Target forecast window for auto-horizon | | snarimax_p | No | 0 (auto) | SNARIMAX autoregressive order. 0 = auto-tune from write frequency (~1h lookback, clamped 4-30) | | snarimax_d | No | 1 | SNARIMAX differencing order | | snarimax_q | No | 1 | SNARIMAX moving average order | | calibration_count | No | 30 | Observations buffered before finalizing SNARIMAX p (only used when snarimax_p=0 ). Larger = more reliable interval estimate, longer warm-up. | | log_forecasts | No | false | Log forecast details | | checkpoint_interval_seconds | No | 1800 | Seconds between model checkpoints to the database | | max_checkpoint_age_hours | No | 24 | Ignore checkpoints older than this when restoring | | min_checkpoint_observations | No | 10 | Skip checkpointing models with fewer observations than this (avoids persisting cold-start models) | | config_file_path | No | — | Path to TOML config file | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | horizon_step | integer | 1-based index of this step in the forecast run | | forecast_value | float | Predicted value for this future step | | horizon_total | integer | Total steps in this forecast | | observations | integer | How many observations the model has learned from | | Parameter | Default | Rationale | | --- | --- | --- | | Parameter | Default | Rationale | | p (AR order) | 0 (auto) | Auto-tuned from write frequency to capture ~1 hour of lag history (clamped 4-30). Set explicitly to override. | | d (differencing) | 1 | Handle non-stationary trends | | q (MA order) | 1 | Basic error correction | | Column | Type | Description | | --- | --- | --- | | Column | Type | Description | | plugin | tag | "river_forecaster" | | series_key | tag | Unique series identifier | | chunk_index | tag | 0-based index of this chunk | | chunk_total | tag | Total number of chunks for this checkpoint | | model_data | string | Base64-encoded zlib-compressed pickle chunk | | model_type | string | "SeriesForecaster" | | observation_count | integer | Observations the model has processed | | checkpoint_size_bytes | integer | Size of the compressed pickle | --- # Sagemaker plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/sagemaker/ The SageMaker Inference Plugin enables periodic, real-time scoring of time-series data in InfluxDB 3 Enterprise against an Amazon SageMaker endpoint. On every scheduled tick the plugin queries recent rows from a measurement, builds a request body matching the shape your SageMaker model expects, calls `InvokeEndpoint`, parses the response, and writes the predictions back into InfluxDB as a new measurement. It supports both batch and per-row inference, single- and multi-output models, custom containers, AWS built-in algorithms, TensorFlow Serving, Hugging Face / JumpStart LLMs, SageMaker Canvas tabular models, multi-model endpoints, and time-series forecasting models such as Amazon Chronos-Bolt. Key features: * Pulls rows from any measurement with a configurable lookback window and row limit * Builds CSV or one of eight JSON body shapes per row or in batch * Time-series forecasting mode (`inputs_timeseries` + `forecast_output=true`): collects N historical rows into one request and expands each array output field into N separate prediction rows * Optional Hugging Face / LLM-style nested parameters via `extra_body`, including array values (`[v1;v2;v3]`) * Multi-model endpoints via `target_model` * Type-aware response extraction (JSON, JSON Lines, CSV, plain text) * Per-row prediction extraction with [JMESPath](https://jmespath.org/) for JSON/JSONLines or column index for CSV * Multi-output predictions (e.g. score + class label + confidence) via `output_fields` * Optional response-driven timestamps for forecasting models * Tag-based filtering of source data, with auto-tags identifying the SageMaker endpoint, source measurement, region, and model ## Configuration Plugin parameters can be provided in two ways: as **trigger arguments** (inline key-value pairs) or via a **TOML configuration file**. Both approaches accept the same parameter names. The TOML approach is recommended when you have complex configs such as many `feature_order` tokens, `extra_body` entries, or multiple `tag_values` filters, because it is easier to read and maintain. ### Option 1: Trigger arguments (inline) Pass parameters as a comma-separated list in `--trigger-arguments`. Use `|` as a separator inside list-valued parameters (`feature_order`, `output_fields`, `extra_body`, `tag_values`): ```bash influxdb3 create trigger \ --database mydb \ --path "sagemaker.py" \ --trigger-spec "every:1m" \ --trigger-arguments 'endpoint_name=my-endpoint,source_measurement=sensor_data,feature_order={motor_speed}|{ambient_temperature},output_fields=score=predictions[*].score' \ sagemaker_score ``` ### Option 2: TOML configuration file Put all parameters into a `.toml` file, then pass only `config_file_path` in the trigger arguments: ```bash influxdb3 create trigger \ --database mydb \ --path "sagemaker.py" \ --trigger-spec "every:1m" \ --trigger-arguments 'config_file_path=sagemaker_config.toml' \ sagemaker_score ``` The `config_file_path` value is resolved as follows: * **Absolute path** (e.g. `/etc/plugins/sagemaker_config.toml`) — used as-is. * **Relative path** (e.g. `sagemaker_config.toml`) — resolved relative to the plugin directory taken from the `INFLUXDB3_PLUGIN_DIR` environment variable (set by the Processing Engine), falling back to `PLUGIN_DIR`. When `config_file_path` is provided all other trigger arguments are ignored; all parameters must be in the file. #### TOML format The TOML file uses the same parameter names as trigger arguments. The key format differences are: | Parameter | Trigger args format | TOML format | |-----------------|-------------------------------------------|----------------------------------------------------------------| | `feature_order` |`{col1}|{col2}|0.0` (pipe-separated string)| `feature_order = ["{col1}", "{col2}", "0.0"]` (array) | | `output_fields` | `score=pred[*].score|label=pred[*].label` |`output_fields = ["score=pred[*].score", "label=pred[*].label"]`| | `extra_body` | `parameters.top_p=0.9|parameters.n=3` | `extra_body = ["parameters.top_p=0.9", "parameters.n=3"]` | |`batch_inference`| `"true"` or `"false"` (string) | `batch_inference = true` (native bool) | | `limit` | `"10"` (string) | `limit = 10` (integer) | | `tag_values` |`sensor_id:A1@A2.env:prod` (encoded string)| `[tag_values]` section (see below) | Tag filters in TOML use a dedicated section where each key maps to a list of allowed values: ```toml [tag_values] sensor_id = ["A1", "A2"] env = ["prod"] ``` #### Minimal TOML example ```toml endpoint_name = "my-endpoint-2025-01-15" source_measurement = "motor_data" region = "eu-central-1" interval = "5min" limit = 10 content_type = "application/json" accept = "application/json" json_shape = "instances_array" batch_inference = true feature_order = ["{motor_speed}", "{ambient_temperature}", "0.0"] output_fields = ["score=predictions[*].score"] target_measurement = "motor_predictions" [tag_values] sensor_id = ["A1", "A2"] ``` A complete reference with examples for all supported body shapes is provided in `sagemaker_config_example.toml`. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines the supported trigger type and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |--------------------|--------------|--------|------------------------------------------------------------------------------------------------------------------------------| | `endpoint_name` | string |required| Name of the deployed SageMaker real-time endpoint | |`source_measurement`| string |required| InfluxDB measurement (table) the plugin reads rows from | | `feature_order` |string / array|required|Tokens describing how to build each row’s request body. Pipe-separated string in args; TOML array. See *feature\_order syntax*| | `output_fields` |string / array|required| `name=path` pairs describing how to extract predictions from the response. Pipe-separated in args; TOML array | ### Request format parameters | Parameter | Type | Default | Description | |--------------|--------------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------| |`content_type`| string | `application/json` | Request body Content-Type. One of `application/json`, `text/csv` | | `accept` | string |same as `content_type`| Expected response Content-Type. One of `application/json`, `application/jsonlines`, `text/csv`, `text/plain` | | `json_shape` | string | `instances_array` | JSON body shape (only when `content_type=application/json`). See *json\_shape values* below | | `extra_body` |string / array| none |`path=value` pairs merged into the JSON body (e.g. nested LLM `parameters`). Pipe-separated in args; TOML array. Only with JSON content types| ### Inference parameters | Parameter | Type | Default | Description | |-----------------|-------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `interval` |string | `60min` | Lookback window for the source query. Format: `` where unit is `s`, `min`, `h`, `d`, `w`, `m` (months≈30d), `q` (quarters≈91d), `y` (years=365d). Examples: `5min`, `1h`, `30d` | | `limit` |integer| `1` | Maximum number of rows to read per scheduled call. String or integer accepted | |`batch_inference`|boolean| `true` | If `true`, send all selected rows in one request and parse a batch response. If `false`, one request per row. String `"true"`/`"false"` (args) or native bool (TOML) | |`forecast_output`|boolean| `false` |If `true`, `output_fields` paths may return arrays; each position becomes a separate output row. Scalar values are broadcast to all rows. Only supported with `accept=application/json`. Use with `json_shape=inputs_timeseries` for time-series forecasting models| | `region` |string |`eu-central-1`| AWS region of the SageMaker endpoint | | `target_model` |string | none | Optional model identifier for **multi-model endpoints** — sets `X-Amzn-SageMaker-Target-Model` header | ### Output parameters | Parameter | Type | Default | Description | |--------------------|------|----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`target_measurement`|string|`_predictions`| Measurement to write predictions into | | `target_database` |string| trigger’s database | Database to write predictions into | | `timestamp_path` |string| none |Optional path to extract per-row timestamps from the response. JMESPath for JSON/JSONLines, integer column index for CSV. If empty, the plugin uses `time.time_ns()`. If set but a row’s timestamp is missing or unparseable, that prediction is skipped with an error (no wall-clock fallback)| ### Filtering parameters | Parameter | Type |Default| Description | |------------|--------------|-------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`tag_values`|string / table| none |Tag filter applied to the source query. In args: `tag1:val1@val2.tag2:val3` — dot separates tag pairs, colon separates tag name from values, `@` separates multiple values. In TOML: `[tag_values]` section with `tag = ["v1", "v2"]`. Tags with a single value are also written to the output line| ## `feature_order` syntax `feature_order` is a list of tokens. In trigger arguments it is pipe-separated; in TOML it is a native array. Each token is either a column reference or a literal value. Aliases (`:alias`) are mainly used in object-shape JSON bodies as JSON keys. | Token | Meaning | |---------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------| | `{col}` | Take the value of column `col` from the row. In object shapes, the JSON key equals `col` | | `{col:alias}` |Take the value of column `col`. In object shapes, the JSON key equals `alias`; in array/CSV shapes the alias is ignored (one warning at init)| | `0`, `0.0`, `-3.14`, `1e3` | Numeric literal (int or float). In CSV the literal is written as text | | `true`, `false` | Boolean literal (in JSON bodies) | | `null` | JSON `null` | | `"text"`, `'text'` | String literal | |`"text":alias`, `0.0:alias`, etc.| Literal **with key alias**. Required for literals in object shapes (`instances_object`, `raw_object`); ignored elsewhere with a warning | To use a literal containing `:` or `|`, wrap it in quotes: `"1:30":time`. ## `output_fields` syntax `output_fields = name=path|name=path|...` (args) or `output_fields = ["name=path", ...]` (TOML) Each entry produces one field on the output line. The path syntax depends on `accept`: | `accept` | Path syntax | Example | |-----------------------|----------------------------------------------------------------|---------------------------------------------------------------------------| | `application/json` |JMESPath — yields an array in batch mode, scalar in per-row mode|`forecasting=predictions[*].score` (batch) or `forecasting=score` (per-row)| |`application/jsonlines`| JMESPath, applied per line | `forecasting=score` | | `text/csv` | integer column index (0-based) | `forecasting=0` | | `text/plain` | empty (response body becomes value); exactly one entry | `result=` | In batch mode, `M` (the number of written rows) equals `min(length)` across all `output_fields` paths and `timestamp_path`. If lengths differ, the plugin truncates and logs a warning. In **forecast mode** (`forecast_output=true`), each `output_fields` path may return an array — every array element becomes a separate output row. Paths that return a scalar are broadcast to all rows. All arrays must have equal length; if they differ, the min-length rule applies with a warning. The Python type of each extracted value determines the InfluxDB field type: * `int` → `int64_field` * `float` → `float64_field` * `bool` → `bool_field` * `str` → `string_field` * `list`/`dict` → row error (use a more specific JMESPath; in forecast mode, nested arrays inside a forecast array are not supported) ## `json_shape` values | Shape | Body produced | Used by | |--------------------|-------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------| | `instances_array` | `{"instances": [[v1,v2,...], ...]}` | TF Serving REST (positional) | | `instances_object` | `{"instances": [{"col": v, ...}, ...]}` | TF Serving REST (named columns) | |`instances_features`| `{"instances": [{"features": [v1,v2,...]}, ...]}` |Built-in AWS algorithms: KMeans, k-NN, RCF, Linear Learner, NTM, PCA| | `inputs` | `{"inputs": [v1,v2,...]}` — single row | TF Serving simplified columnar (single tensor) | | `inputs_array` | `{"inputs": [[v1,v2], ...]}` | TF Serving / PyTorch batch | | `inputs_flat` | `{"inputs": [v1,v2,...]}` — one value per row, requires exactly one `{col}` token | Hugging Face NLP batch (`{"inputs": ["text1", "text2"]}`) | |`inputs_timeseries` |`{"inputs": [{"target": [v1,v2,...,vN]}]}` — all rows collected into one target array; requires exactly one `{col}` token| Time-series forecasting models (Amazon Chronos-Bolt, etc.) | | `raw_array` | `[[v1,v2,...], ...]` | Custom containers, simplified TF Serving | | `raw_object` | `{"col": v, ...}` — single row | Hugging Face simple, custom containers (e.g. `{"inputs": "text"}`) | **Batch compatibility:** `inputs` and `raw_object` produce a single-row body and require either `batch_inference=false` or `limit=1`. All other shapes are batch-friendly. **Time-series forecasting:** use `inputs_timeseries` together with `forecast_output=true`. With `limit=N`, all N rows from the source query are packed into a single `target` array and sent in one request. The response’s array fields are then expanded back into N output rows — one per forecast step. **Object shapes** (`instances_object`, `raw_object`) accept literal tokens **only with an alias** (`literal:alias`); the alias becomes the JSON key. ## `extra_body` syntax `extra_body = path=value|path=value|...` (args) or `extra_body = ["path=value", ...]` (TOML) Each entry adds a static value at a JSON path. Dotted paths produce nested objects; values are coerced (numbers, booleans, `null`, quoted strings). **Array values** use semicolons as element separators inside square brackets: `[v1;v2;v3]`. ```bash # args — scalar values extra_body = parameters.max_new_tokens=50|parameters.top_p=0.95|parameters.do_sample=true # args — array value (semicolons, no spaces inside brackets) extra_body = parameters.quantile_levels=[0.1;0.5;0.9]|parameters.prediction_length=10 # TOML extra_body = [ "parameters.max_new_tokens=50", "parameters.top_p=0.95", "parameters.do_sample=true", "parameters.quantile_levels=[0.1;0.5;0.9]", ] ``` → merged into the body as: ```json {"parameters":{"max_new_tokens":50,"top_p":0.95,"do_sample":true,"quantile_levels":[0.1,0.5,0.9]}} ``` Restrictions: * Cannot be used with `content_type=text/csv`. * Cannot be used with `json_shape=raw_array` (top-level list — nothing to merge into). * Conflicts with feature\_order keys at any depth fail at run time. ## Auto-tags Every output line carries the following tags: | Tag | Value | |------------------------------|--------------------------------| | `sagemaker_endpoint` | `endpoint_name` | |`sagemaker_source_measurement`| `source_measurement` | | `sagemaker_region` | `region` | | `sagemaker_model` |`target_model` *(only when set)*| Single-valued tags from `tag_values` are also written to the output line. ## Software Requirements * **InfluxDB 3 Enterprise** with the Processing Engine enabled * **AWS credentials** available to the InfluxDB 3 Enterprise host process (via environment variables, IAM role, or AWS credentials file) with permission to call `sagemaker:InvokeEndpoint` * **Python packages**: * `boto3` (AWS SDK) * `pandas` (timestamp parsing) * `jmespath` is included as a transitive dependency of `boto3`/`botocore` ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled: ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package boto3 influxdb3 install package pandas ``` 3. Make AWS credentials available to InfluxDB 3 Enterprise (one of): * Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optionally `AWS_SESSION_TOKEN` in the host environment * Run InfluxDB 3 Enterprise on an EC2 instance / EKS pod with an IAM role that allows `sagemaker:InvokeEndpoint` * Mount `~/.aws/credentials` and set `AWS_PROFILE` ## Trigger setup ### With trigger arguments ```bash influxdb3 create trigger \ --database mydb \ --path "sagemaker.py" \ --trigger-spec "every:1m" \ --trigger-arguments 'endpoint_name=my-endpoint,source_measurement=sensor_data,feature_order={motor_speed}|{ambient_temperature},output_fields=score=predictions[*].score' \ sagemaker_score ``` ### With a TOML configuration file ```bash influxdb3 create trigger \ --database mydb \ --path "sagemaker.py" \ --trigger-spec "every:1m" \ --trigger-arguments 'config_file_path=sagemaker_config.toml' \ sagemaker_score ``` Place `sagemaker_config.toml` in your plugin directory — the one set via `--plugin-dir` and exposed as `INFLUXDB3_PLUGIN_DIR` (or `PLUGIN_DIR`) — or provide an absolute path. See `sagemaker_config_example.toml` for ready-to-use templates covering all supported endpoint types. ## Example usage ### Example 1: SageMaker Canvas tabular model (CSV) Score the latest row of sensor data against a Canvas model that expects CSV input and returns a single float per row. ```bash influxdb3 create trigger \ --database factory \ --path "sagemaker.py" \ --trigger-spec "every:1m" \ --trigger-arguments 'endpoint_name=canvas-deployment-2026-01-10,source_measurement=press,feature_order={motor_speed}|{ambient_temperature}|{vibration},content_type=text/csv,accept=text/csv,output_fields=quality_score=0,limit=1,batch_inference=false,region=us-east-1' \ press_quality ``` Request body sent to SageMaker: ``` 1500,25.5,0.3 ``` Response: ``` 0.87 ``` Result: a row in measurement `press_predictions` with field `quality_score=0.87` (float). ### Example 2: AWS built-in algorithm — Random Cut Forest anomaly score RCF expects `{"instances":[{"features":[...]}]}`. ```bash influxdb3 create trigger \ --database iot \ --path "sagemaker.py" \ --trigger-spec "every:30s" \ --trigger-arguments 'endpoint_name=rcf-anomaly,source_measurement=metrics,feature_order={cpu}|{mem}|{rps}|{p99_ms},json_shape=instances_features,output_fields=anomaly_score=scores[*].score,interval=2min,limit=20' \ rcf_score ``` Request body for 20 rows (batch): ```json {"instances":[{"features":[42.1,71.0,1834,8.4]}, ..., {"features":[39.9,72.2,1903,9.0]}]} ``` Response: ```json {"scores":[{"score":0.83},{"score":0.71}, ..., {"score":0.92}]} ``` Each score is written as `anomaly_score` field. Use `timestamp_path` if your model returns timestamps. ### Example 3: TensorFlow Serving with renamed JSON keys Rename InfluxDB columns to keys the model expects (`Speed`, `Temp`, `Vib`). ```bash influxdb3 create trigger \ --database factory \ --path "sagemaker.py" \ --trigger-spec "every:30s" \ --trigger-arguments 'endpoint_name=tf-serving,source_measurement=press,feature_order={motor_speed:Speed}|{ambient_temperature:Temp}|{vibration:Vib},json_shape=instances_object,output_fields=p_fail=predictions[*].p_fail|p_ok=predictions[*].p_ok,limit=10' \ press_failure ``` Request body: ```json {"instances":[{"Speed":1500,"Temp":25.5,"Vib":0.3}, ...]} ``` Two prediction fields per row are written: `p_fail` and `p_ok`. ### Example 4: Hugging Face LLM with parameters (TOML config) Score a free-text sensor log against a Hugging Face text classification endpoint, with generation parameters. Using a TOML file makes the `extra_body` list easy to maintain. `hf_classifier.toml`: ```toml endpoint_name = "hf-classifier" source_measurement = "tickets" region = "eu-west-1" interval = "5min" limit = 1 content_type = "application/json" accept = "application/json" json_shape = "raw_object" batch_inference = false feature_order = ["{text:inputs}"] output_fields = ["label=label", "score=score"] extra_body = [ "parameters.top_k=5", "parameters.temperature=0.3", ] target_measurement = "ticket_classifications" ``` ```bash influxdb3 create trigger \ --database support \ --path "sagemaker.py" \ --trigger-spec "every:5min" \ --trigger-arguments 'config_file_path=hf_classifier.toml' \ ticket_classifier ``` Request body: ```json {"inputs":"Bearing 7 reports rising temperature for 20 minutes...","parameters":{"top_k":5,"temperature":0.3}} ``` ### Example 5: Forecasting with response timestamps DeepAR-style forecast where the model returns predicted timestamps along with values. The plugin writes each prediction at its own future timestamp. ```bash influxdb3 create trigger \ --database forecasts \ --path "sagemaker.py" \ --trigger-spec "every:15min" \ --trigger-arguments 'endpoint_name=deepar-fcast,source_measurement=demand,feature_order={value},json_shape=instances_features,output_fields=forecasting=predictions[*].mean,timestamp_path=predictions[*].t,limit=24,interval=24h,target_measurement=demand_forecasts' \ demand_forecast ``` Each forecasted point is written with timestamp from `predictions[*].t` — autodetected as ISO string, Unix seconds, or nanoseconds. ### Example 6: Multi-model endpoint with tag filtering Score only rows from a specific sensor against a specific model deployed on a multi-model endpoint, with the `region` tag carried through to the output. ```bash influxdb3 create trigger \ --database iot \ --path "sagemaker.py" \ --trigger-spec "every:1min" \ --trigger-arguments 'endpoint_name=mme-models,target_model=motor-v3.tar.gz,source_measurement=sensors,feature_order={rpm}|{torque},output_fields=health=predictions[*].health,tag_values=region:eu-central-1.sensor_id:motor-7,limit=5' \ motor_health ``` Output lines carry tags `sagemaker_endpoint=mme-models`, `sagemaker_source_measurement=sensors`, `sagemaker_region=eu-central-1` (default region), `sagemaker_model=motor-v3.tar.gz`, `region=eu-central-1`, `sensor_id=motor-7`. ### Example 7: Time-series forecasting with Amazon Chronos-Bolt (TOML config) Collect 20 historical sensor readings and send them to a Chronos-Bolt endpoint as one time series. The model returns mean and quantile forecasts for 10 future steps — each step is written as a separate row. `chronos.toml`: ```toml endpoint_name = "chronos-bolt-small-endpoint" source_measurement = "sensor_readings" region = "eu-central-1" interval = "1h" limit = 20 content_type = "application/json" accept = "application/json" json_shape = "inputs_timeseries" batch_inference = true forecast_output = true # Single {col} token required for inputs_timeseries feature_order = ["{value}"] # Each path returns a 10-element array → 10 output rows # Quantile key names start with a digit, so quote them in JMESPath output_fields = [ "mean=predictions[0].mean", "q10=predictions[0].\"0.1\"", "q90=predictions[0].\"0.9\"", ] # Array values use semicolons as element separator extra_body = [ "parameters.prediction_length=10", "parameters.quantile_levels=[0.1;0.5;0.9]", ] target_measurement = "sensor_forecasts" ``` ```bash influxdb3 create trigger \ --database iot \ --path "sagemaker.py" \ --trigger-spec "every:1h" \ --trigger-arguments 'config_file_path=chronos.toml' \ chronos_forecast ``` Request body sent to SageMaker (20 historical values packed into `target`): ```json {"inputs": [{"target": [1.1, 2.3, 1.8, ...]}], "parameters": {"prediction_length": 10, "quantile_levels": [0.1, 0.5, 0.9]}} ``` Response: ```json {"predictions": [{"mean": [-0.0, 3.5, ...], "0.1": [-2.4, 1.5, ...], "0.9": [1.7, 5.6, ...]}]} ``` Result: 10 rows written to `sensor_forecasts`, each with fields `mean`, `q10`, `q90`. ## Code overview ### Files * `sagemaker.py` — the main plugin file. Contains the JSON metadata header, configuration parsing, request body builders, response extractors, and the `process_scheduled_call` entry point. * `sagemaker_config_example.toml` — annotated TOML templates for all supported endpoint types (instances\_array, instances\_object, CSV, LLM/HuggingFace, multi-model, Chronos-Bolt time-series forecasting). ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name' ORDER BY event_time DESC LIMIT 50" ``` Each log line includes a unique `task_id` (UUID) so messages from a single scheduled call can be correlated. Levels: * **INFO** — initialization, source query, row count, write completion * **WARN** — Content-Type mismatch, length mismatch in batch response, missing path values, ignored aliases * **ERROR** — config validation failure, batch inference failure, write failure ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Entry point invoked by the Processing Engine on each tick. 1. Loads cached config or builds it from `args` (validates schema, parses `feature_order`, `output_fields`, `tag_values`, `extra_body`) 2. Builds and runs the source SQL query (with optional tag filter) 3. Either sends one batched `InvokeEndpoint` call or iterates per row, depending on `batch_inference` 4. Parses the response according to `accept` and `output_fields` 5. Builds typed `LineBuilder` per prediction row with auto-tags, single-valued filter tags, and timestamp 6. Writes all lines in one batched call to InfluxDB #### `build_request_body(rows, tokens, content_type, json_shape, extra_body=None)` Builds the request body as a Python object first, optionally deep-merges `extra_body`, then serialises to JSON. Returns a CSV string for `content_type=text/csv`. #### `extract_outputs(response_body, response_ct, accept, output_fields, timestamp_compiled, ..., forecast_output)` Returns a list of `(timestamp_or_None, {field: value})` tuples. Dispatches to one of: * `_extract_from_json_forecast` — when `forecast_output=True`: fields may be arrays (indexed per row) or scalars (broadcast) * `_extract_from_json` — single JSON document, JMESPath projections for batch or scalar paths for per-row * `_extract_from_jsonlines` — one record per line * `_extract_from_csv` — `csv.reader` per line, integer column index * `_extract_from_plain` — single scalar from the body If the actual response Content-Type differs from the requested `accept`, the plugin warns and tries to parse with the actual one (if supported). ## Troubleshooting ### Common issues #### Issue: `feature_order references columns that don't exist in ''` **Solution:** Check the column names in `feature_order` and confirm the source measurement has data. The plugin validates feature\_order columns against `information_schema.columns` at start-up. The error message lists missing columns and all available columns — fix the typo in `feature_order` or write some data into the measurement first. #### Issue: `Got M prediction(s) for N input row(s)` **Solution:** Verify that the SageMaker model returns one prediction per submitted input row, or lower the batch size while debugging. The number of predictions returned by the model differs from the number of source rows sent in the batch. The plugin writes whatever predictions came back and warns about the delta. If consistent, check whether your model auto-aggregates or rejects malformed rows. #### Issue: `text/plain accepts exactly one output field` **Solution:** Use a single output field with an empty path. Plain-text responses are scalar — define one entry in `output_fields` with an empty path: `output_fields=result=`. #### Issue: `json_shape 'inputs' is not batch-compatible` **Solution:** Disable batch inference, limit the query to one row, or choose a batch-compatible JSON shape. `inputs` and `raw_object` produce single-row bodies. Either set `batch_inference=false`, set `limit=1`, or switch to a batch-compatible shape (`inputs_array`, `instances_array`, etc.). #### Issue: `extra_body conflicts with main body at ''` **Solution:** Move the static value to a different path or remove the duplicate key. The static path you set in `extra_body` collides with a key produced by `feature_order`. Pick a different alias or remove the conflicting path. #### Issue: `Invalid interval format` **Solution:** Use a duration in `` form with a supported unit. Interval must be ``. Unit must be one of `s`, `min`, `h`, `d`, `w`, `m`, `q`, `y`. Note that `m` means **months** (≈30 days), not minutes — use `min` for minutes. #### Issue: AWS authentication errors (`UnrecognizedClientException`, `AccessDeniedException`) **Solution:** Provide AWS credentials or an IAM role that can invoke the target SageMaker endpoint. The InfluxDB 3 Enterprise host process needs AWS credentials with permission to call `sagemaker:InvokeEndpoint` on your endpoint. Set `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, configure an IAM role on the instance, or set `AWS_PROFILE` to a profile in `~/.aws/credentials`. #### Issue: `Empty response body from SageMaker` **Solution:** Check the SageMaker endpoint and model logs in CloudWatch. The model returned an empty body. Check the model’s CloudWatch logs in AWS — usually it indicates a model exception or an endpoint that is `OutOfService`. ### Debugging tips 1. **Inspect the source query** — every scheduled call logs `Source query: SELECT ...`. Run it manually: ``` influxdb3 query --database mydb "SELECT time, motor_speed, ambient_temperature FROM sensors WHERE time > now() - INTERVAL '5 minutes' ORDER BY time DESC LIMIT 1" ``` 2. **Test the endpoint outside the plugin** — confirm Content-Type and Accept on a known-good payload: ``` aws sagemaker-runtime invoke-endpoint \ --endpoint-name my-endpoint \ --content-type application/json \ --accept application/json \ --body '{"instances":[[1500,25.5,0.3]]}' \ /tmp/out.json && cat /tmp/out.json ``` 3. **Switch to per-row mode** — set `batch_inference=false` to isolate which row fails when batch parsing is unclear. 4. **Force a config rebuild** — config is cached for 1 hour. After updating trigger arguments or the TOML file, delete and recreate the trigger or wait for the cache to expire. ### Performance considerations * **Batch inference** is dramatically cheaper. With 60 rows at 200ms latency, batch sends one HTTP request (\~250ms total); per-row sends 60 requests (\~12s total). Use `batch_inference=false` only when the model truly does not support batched bodies. * **Limit your batch size** to stay under SageMaker’s 6 MB body limit. Roughly 50–500 rows per batch for tabular data is a safe range. * **Schema cache** — measurement columns are cached for 1 hour after the first lookup. * **Config cache** — parsed config is cached for 1 hour; the boto3 client is reused for the duration of one scheduled call. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Sagemaker plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/sagemaker) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Trigger args format | TOML format | | --- | --- | --- | | Parameter | Trigger args format | TOML format | | feature_order | {col1}|{col2}|0.0 (pipe-separated string) | feature_order = ["{col1}", "{col2}", "0.0"] (array) | | output_fields | score=pred[*].score|label=pred[*].label | output_fields = ["score=pred[*].score", "label=pred[*].label"] | | extra_body | parameters.top_p=0.9|parameters.n=3 | extra_body = ["parameters.top_p=0.9", "parameters.n=3"] | | batch_inference | "true" or "false" (string) | batch_inference = true (native bool) | | limit | "10" (string) | limit = 10 (integer) | | tag_values | sensor_id:A1@A2.env:prod (encoded string) | [tag_values] section (see below) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | endpoint_name | string | required | Name of the deployed SageMaker real-time endpoint | | source_measurement | string | required | InfluxDB measurement (table) the plugin reads rows from | | feature_order | string / array | required | Tokens describing how to build each row’s request body. Pipe-separated string in args; TOML array. See feature_order syntax | | output_fields | string / array | required | name=path pairs describing how to extract predictions from the response. Pipe-separated in args; TOML array | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | content_type | string | application/json | Request body Content-Type. One of application/json , text/csv | | accept | string | same as content_type | Expected response Content-Type. One of application/json , application/jsonlines , text/csv , text/plain | | json_shape | string | instances_array | JSON body shape (only when content_type=application/json ). See json_shape values below | | extra_body | string / array | none | path=value pairs merged into the JSON body (e.g. nested LLM parameters ). Pipe-separated in args; TOML array. Only with JSON content types | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | interval | string | 60min | Lookback window for the source query. Format: where unit is s , min , h , d , w , m (months≈30d), q (quarters≈91d), y (years=365d). Examples: 5min , 1h , 30d | | limit | integer | 1 | Maximum number of rows to read per scheduled call. String or integer accepted | | batch_inference | boolean | true | If true , send all selected rows in one request and parse a batch response. If false , one request per row. String "true" / "false" (args) or native bool (TOML) | | forecast_output | boolean | false | If true , output_fields paths may return arrays; each position becomes a separate output row. Scalar values are broadcast to all rows. Only supported with accept=application/json . Use with json_shape=inputs_timeseries for time-series forecasting models | | region | string | eu-central-1 | AWS region of the SageMaker endpoint | | target_model | string | none | Optional model identifier for multi-model endpoints — sets X-Amzn-SageMaker-Target-Model header | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | target_measurement | string | _predictions | Measurement to write predictions into | | target_database | string | trigger’s database | Database to write predictions into | | timestamp_path | string | none | Optional path to extract per-row timestamps from the response. JMESPath for JSON/JSONLines, integer column index for CSV. If empty, the plugin uses time.time_ns() . If set but a row’s timestamp is missing or unparseable, that prediction is skipped with an error (no wall-clock fallback) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | tag_values | string / table | none | Tag filter applied to the source query. In args: tag1:val1@val2.tag2:val3 — dot separates tag pairs, colon separates tag name from values, @ separates multiple values. In TOML: [tag_values] section with tag = ["v1", "v2"] . Tags with a single value are also written to the output line | | Token | Meaning | | --- | --- | | Token | Meaning | | {col} | Take the value of column col from the row. In object shapes, the JSON key equals col | | {col:alias} | Take the value of column col . In object shapes, the JSON key equals alias ; in array/CSV shapes the alias is ignored (one warning at init) | | 0 , 0.0 , -3.14 , 1e3 | Numeric literal (int or float). In CSV the literal is written as text | | true , false | Boolean literal (in JSON bodies) | | null | JSON null | | "text" , 'text' | String literal | | "text":alias , 0.0:alias , etc. | Literal with key alias . Required for literals in object shapes ( instances_object , raw_object ); ignored elsewhere with a warning | | accept | Path syntax | Example | | --- | --- | --- | | accept | Path syntax | Example | | application/json | JMESPath — yields an array in batch mode, scalar in per-row mode | forecasting=predictions[*].score (batch) or forecasting=score (per-row) | | application/jsonlines | JMESPath, applied per line | forecasting=score | | text/csv | integer column index (0-based) | forecasting=0 | | text/plain | empty (response body becomes value); exactly one entry | result= | | Shape | Body produced | Used by | | --- | --- | --- | | Shape | Body produced | Used by | | instances_array | {"instances": [[v1,v2,...], ...]} | TF Serving REST (positional) | | instances_object | {"instances": [{"col": v, ...}, ...]} | TF Serving REST (named columns) | | instances_features | {"instances": [{"features": [v1,v2,...]}, ...]} | Built-in AWS algorithms: KMeans, k-NN, RCF, Linear Learner, NTM, PCA | | inputs | {"inputs": [v1,v2,...]} — single row | TF Serving simplified columnar (single tensor) | | inputs_array | {"inputs": [[v1,v2], ...]} | TF Serving / PyTorch batch | | inputs_flat | {"inputs": [v1,v2,...]} — one value per row, requires exactly one {col} token | Hugging Face NLP batch ( {"inputs": ["text1", "text2"]} ) | | inputs_timeseries | {"inputs": [{"target": [v1,v2,...,vN]}]} — all rows collected into one target array; requires exactly one {col} token | Time-series forecasting models (Amazon Chronos-Bolt, etc.) | | raw_array | [[v1,v2,...], ...] | Custom containers, simplified TF Serving | | raw_object | {"col": v, ...} — single row | Hugging Face simple, custom containers (e.g. {"inputs": "text"} ) | | Tag | Value | | --- | --- | | Tag | Value | | sagemaker_endpoint | endpoint_name | | sagemaker_source_measurement | source_measurement | | sagemaker_region | region | | sagemaker_model | target_model (only when set) | --- # Schema validator plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/schema-validator/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later. > > The Schema Validator Plugin validates incoming line protocol against a user-defined JSON schema and forwards only conforming rows to a target database or table. It runs on every WAL flush of the tables named by its trigger specification: each row is checked against the measurement whitelist, the required tags and their allowed values, and the required fields, their types and their allowed values. Valid rows are stripped down to the schema-defined tags and fields and written to the target; rejected rows are optionally logged and recorded in a `_schema_rejections` measurement. Typical pipelines: * `raw_db` → validate → `clean_db` (cross-database) * `raw_table` → validate → `validated_table` (same database, different table) * `source_table` → validate → `source_table_clean` (same database, with a suffix) This is a single-file plugin (`schema_validator.py`) and can be loaded from GitHub via `gh:` trigger paths or created in InfluxDB 3 Explorer. > > > **Note:** The JSON schema file must be placed in the plugin directory on the server by hand — there is no API for uploading non-plugin files, so Explorer cannot upload it for you. Use `scp`, `rsync`, or any other file transfer method. > > ## Features * **Measurement whitelist**: `allowed_measurements` names the tables that are processed; a table outside the list is skipped * **Tag validation**: each tag is required or optional and may carry a list of allowed values * **Field validation**: each field is required or optional, is checked against its declared or inferred type, and may carry a list of allowed values * **Field stripping**: tags and fields the schema does not define are dropped from the written row * **Flexible targeting**: write to another database, to the name a table’s `target_table` gives, or to the source name with a prefix or suffix * **Per-table schemas**: every measurement carries its own tags, fields and target * **Rejection logging**: a rejected row is logged with its reason and, optionally, recorded in the `_schema_rejections` measurement * **Cached schema**: the JSON file is re-read at most once every five minutes ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |-------------|------|--------|-----------------------------------------------------------------------------------------------| |`schema_file`|string|required|Path to the JSON schema file, absolute or relative to the plugin directory. Must end in `.json`| ### Data write trigger parameters | Parameter | Type | Default | Description | |---------------------|-------|----------------|--------------------------------------------------------------------------------------------------------------------------------------------------| | `target_database` |string |trigger’s own DB| Database that validated rows and the rejection log are written to | |`target_table_prefix`|string | `""` | Prefix added to the source measurement name in the target. Ignored for tables that define `target_table` | |`target_table_suffix`|string | `""` | Suffix added to the source measurement name in the target. Ignored for tables that define `target_table` | | `log_rejected` |boolean| `true` | Log one warning per rejected row, plus one message per skipped table. Accepts `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off` | | `log_accepted` |boolean| `false` | Log one message per accepted row — noisy, use for debugging. Accepts `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off` | |`write_rejection_log`|boolean| `false` |Write rejected row details to the `_schema_rejections` measurement in the target database. Accepts `true`/`false`, `1`/`0`, `yes`/`no`, `on`/`off`| ### Environment variables Every parameter can also come from an environment variable named`INFLUXDB3_SCHEMA_VALIDATOR_` in upper case — for example,`INFLUXDB3_SCHEMA_VALIDATOR_SCHEMA_FILE` sets `schema_file`. The environment is the lowest layer: a trigger argument overrides it, and the TOML file overrides both. `INFLUXDB3_SCHEMA_VALIDATOR_CONFIG_FILE_PATH` names the TOML file when the trigger doesn’t carry a `config_file_path` argument. ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|------------------------------------------------------------------------| |`config_file_path`|string| none |Path to a TOML config file, absolute or relative to the plugin directory| To use a TOML configuration file, name it in `config_file_path` in the trigger arguments. A relative path — `config_file_path` and `schema_file` alike — is resolved against the plugin directory: `PLUGIN_DIR` when it is set, otherwise `INFLUXDB3_PLUGIN_DIR`, which the processing engine sets from `--plugin-dir`, otherwise the parent of `VIRTUAL_ENV`. An absolute path is used as written. The file accepts the same keys as inline arguments, and its values override them. A key outside the tables above is refused and named in the error, in the trigger arguments and in the TOML file alike, so a misspelling is reported rather than silently dropped. #### Example TOML configuration * [schema\_validator\_trigger\_config.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/schema_validator/schema_validator_trigger_config.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Schema configuration (JSON) ```json { "allowed_measurements": ["weather", "cpu"], "tables": { "weather": { "target_table": "weather_clean", "tags": { "location": { "required": true, "allowed_values": ["us-east", "us-west", "eu-west"] }, "station_id": { "required": true }, "region": { "required": false } }, "fields": { "temperature": { "required": true, "type": "float" }, "humidity": { "required": true, "type": "float" }, "condition": { "required": false, "type": "string", "allowed_values": ["sunny", "cloudy", "rain", "snow"] } } } } } ``` ### Top-level | Field | Type | Description | |----------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------| |`allowed_measurements`|`list[str]` (optional)|Whitelist of measurement names. When omitted or empty, no measurement filter is applied and all measurements fall through to the `tables` rules| | `tables` | `dict` (required) | Map of measurement name -\> table definition. Must contain at least one entry; measurements without an entry are skipped | ### Table definition | Field | Type | Description | |--------------|-----------------|--------------------------------------------------------------------------------------------| |`target_table`|`str` (optional) |Target measurement name for this table. Takes precedence over `target_table_prefix`/`suffix`| | `tags` |`dict` (optional)| Map of tag name -\> tag definition | | `fields` |`dict` (required)| Map of field name -\> field definition. Must contain at least one entry | ### Tag definition | Field | Type | Description | |----------------|-----------------|---------------------------------------------| | `required` | `bool` |When `true`, the tag must be present on a row| |`allowed_values`|`list` (optional)| Whitelist of values, compared as strings | ### Field definition | Field | Type | Description | |----------------|-----------------|----------------------------------------------------------------| | `required` | `bool` | When `true`, the field must be present on a row | | `type` |`str` (optional) |Expected type. When omitted, the type is inferred from the value| |`allowed_values`|`list` (optional)| Whitelist of values, compared by value and as strings | A tag or field written as a bare name instead of a definition object is treated as required with no value whitelist. ### Field types | `type` | Accepted values | |----------------------------|-----------------------------------------------------------------| |`float`, `float64`, `double`| Integers and floats, excluding booleans; must be finite | | `integer`, `int`, `int64` | Integers, excluding booleans; must fit into `int64` | |`uint64`, `unsigned`, `uint`|Non-negative integers, excluding booleans; must fit into `uint64`| | `string`, `str` | Strings | | `boolean`, `bool` | Booleans | An unknown type name in the schema is reported as an error and the plugin does not run. When a field definition has no `type`, the type is inferred from the value: booleans become `boolean`, integers `integer`, floats `float`, everything else `string`. ## Validation logic For each row, in order: 1. **Required tags** — every tag marked `required` must be present. 2. **Tag values** — a tag with `allowed_values` must carry one of them. 3. **Required fields** — every field marked `required` must be present. 4. **Field types** — a present field must match its declared or inferred type. 5. **Field values** — a field with `allowed_values` must carry one of them. If any check fails, the row is rejected and nothing is written for it. A row is also rejected when it carries no schema-defined field, when a value cannot be written as its type (a non-finite float, an integer outside the `int64`/`uint64` range), or when a name or tag value cannot be written as line protocol (an empty tag value, a value containing a newline or tab or ending with a backslash, a column named `time`). Tags and fields absent from the schema are stripped from the output. Tables absent from a non-empty `allowed_measurements` list are skipped, as are tables with no `tables` entry. ## Target resolution Validated rows of a table are written to: 1. the table’s `target_table`, when set; otherwise 2. ``. When `target_database` is not set, rows go back into the trigger’s own database, so a table must resolve to a measurement name other than its own — otherwise the trigger would feed itself. A flush carrying such a table is rejected, naming the offending table; tables that the trigger never receives are not checked. Set `target_database`, a prefix, a suffix, or a per-table `target_table`. The schema file is cached for 5 minutes; edits are picked up within that window. ## Rejection log With `write_rejection_log` enabled, every rejected row adds one point to `_schema_rejections` in the target database: | Column | Kind | Description | |--------------|---------|----------------------------------------------------------| |`source_table`| tag | Table the rejected row arrived on | | `reason` | field | Rejection reason | | `row_data` | field |The row rendered as a string, truncated to 1024 characters| | `time` |timestamp| Time of validation | Entries are batched per table and written in one call. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: `influxdata-plugin-utils>=0.4.0` ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.4.0" ``` 3. Copy the JSON schema file into the plugin directory: ``` scp schema_validator_config.json user@server:~/.plugins/ ``` ## Trigger setup Cross-database validation (`raw_db` -\> `clean_db`): ```bash influxdb3 create trigger \ --database raw_db \ --path "gh:influxdata/schema_validator/schema_validator.py" \ --trigger-spec "all_tables" \ --trigger-arguments "schema_file=schema_validator_config.json,target_database=clean_db" \ --error-behavior log \ schema_validator_trigger ``` Same database, different table: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/schema_validator/schema_validator.py" \ --trigger-spec "table:weather" \ --trigger-arguments "schema_file=schema_validator_config.json,target_table_suffix=_clean" \ --error-behavior log \ schema_validator_weather ``` Using a TOML config file: ```bash influxdb3 create trigger \ --database raw_db \ --path "gh:influxdata/schema_validator/schema_validator.py" \ --trigger-spec "all_tables" \ --trigger-arguments "config_file_path=schema_validator_trigger_config.toml" \ --error-behavior log \ schema_validator_trigger ``` ### Enable triggers ```bash influxdb3 enable trigger --database raw_db schema_validator_trigger ``` ## Example usage ### Example 1: Cross-database validation ```bash # This row has all required tags and fields -> written to clean_db influxdb3 write --database raw_db \ "weather,location=us-east,station_id=ST001 temperature=72.5,humidity=45.2" # This row is missing the required tag 'station_id' -> rejected influxdb3 write --database raw_db \ "weather,location=us-east temperature=72.5,humidity=45.2" # Query the validated data influxdb3 query --database clean_db "SELECT * FROM weather_clean ORDER BY time DESC LIMIT 10" ``` ### Example 2: Monitoring rejections ```bash influxdb3 create trigger \ --database raw_db \ --path "gh:influxdata/schema_validator/schema_validator.py" \ --trigger-spec "all_tables" \ --trigger-arguments "schema_file=schema_validator_config.json,target_database=clean_db,write_rejection_log=true" \ --error-behavior log \ schema_validator_all ``` ```sql SELECT source_table, reason, row_data FROM _schema_rejections WHERE time > now() - INTERVAL '1 hour' ORDER BY time DESC ``` ## Code overview ### Files * `schema_validator.py`: The main plugin code containing `process_writes` * `schema_validator_config.json`: Example JSON schema definition * `schema_validator_trigger_config.toml`: Example TOML trigger configuration * `test_schema_validator.py`: Pytest suite (55 tests, runs without a live InfluxDB 3 Enterprise server) * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.4.0`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Every log line is prefixed with a per-fire `task_id` (eight hex characters) so records from a single trigger fire can be correlated. ### Main functions #### `process_writes(influxdb3_local, table_batches, args)` Loads the configuration and the cached schema, then processes each table batch independently: rows are validated, valid ones are collected as line protocol and written with `write_sync` (or `write_sync_to_db` when `target_database` is set), and rejection log entries are written in a second batch. Writes are not retried, so a WAL flush is never held by backoff; a failing write is reported with the number of rows that did not land, those rows are counted as dropped in the closing summary instead of accepted, and the remaining tables are still processed. ## Troubleshooting ### Common issues #### Issue: No data appears in the target **Solution**: Only measurements with an entry in `tables` are forwarded, and a non-empty `allowed_measurements` list filters them further. Check `system.processing_engine_logs` for `No schema defined for table` and `not in allowed_measurements` messages. #### Issue: Trigger reports “would be written back into itself” **Solution**: With no `target_database`, the target measurement name must differ from the source. Set `target_database`, `target_table_prefix`, `target_table_suffix`, or the table’s `target_table`. #### Issue: Every row of a table is rejected **Solution**: Read the rejection reason in the logs or in `_schema_rejections`. Common causes are a `required` tag that arrives as a field in the source data (or the reverse), an `allowed_values` list that does not cover production values, and a `type` that does not match what is written (for example, `integer` against a value written as `72.5`). #### Issue: Schema edits do not take effect **Solution**: The schema file is cached for 5 minutes. Wait for the cache to expire or restart the trigger. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Schema validator plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/schema_validator) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | schema_file | string | required | Path to the JSON schema file, absolute or relative to the plugin directory. Must end in .json | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | target_database | string | trigger’s own DB | Database that validated rows and the rejection log are written to | | target_table_prefix | string | "" | Prefix added to the source measurement name in the target. Ignored for tables that define target_table | | target_table_suffix | string | "" | Suffix added to the source measurement name in the target. Ignored for tables that define target_table | | log_rejected | boolean | true | Log one warning per rejected row, plus one message per skipped table. Accepts true / false , 1 / 0 , yes / no , on / off | | log_accepted | boolean | false | Log one message per accepted row — noisy, use for debugging. Accepts true / false , 1 / 0 , yes / no , on / off | | write_rejection_log | boolean | false | Write rejected row details to the _schema_rejections measurement in the target database. Accepts true / false , 1 / 0 , yes / no , on / off | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | Path to a TOML config file, absolute or relative to the plugin directory | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | allowed_measurements | list[str] (optional) | Whitelist of measurement names. When omitted or empty, no measurement filter is applied and all measurements fall through to the tables rules | | tables | dict (required) | Map of measurement name -> table definition. Must contain at least one entry; measurements without an entry are skipped | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | target_table | str (optional) | Target measurement name for this table. Takes precedence over target_table_prefix / suffix | | tags | dict (optional) | Map of tag name -> tag definition | | fields | dict (required) | Map of field name -> field definition. Must contain at least one entry | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | required | bool | When true , the tag must be present on a row | | allowed_values | list (optional) | Whitelist of values, compared as strings | | Field | Type | Description | | --- | --- | --- | | Field | Type | Description | | required | bool | When true , the field must be present on a row | | type | str (optional) | Expected type. When omitted, the type is inferred from the value | | allowed_values | list (optional) | Whitelist of values, compared by value and as strings | | type | Accepted values | | --- | --- | | type | Accepted values | | float , float64 , double | Integers and floats, excluding booleans; must be finite | | integer , int , int64 | Integers, excluding booleans; must fit into int64 | | uint64 , unsigned , uint | Non-negative integers, excluding booleans; must fit into uint64 | | string , str | Strings | | boolean , bool | Booleans | | Column | Kind | Description | | --- | --- | --- | | Column | Kind | Description | | source_table | tag | Table the rejected row arrived on | | reason | field | Rejection reason | | row_data | field | The row rendered as a string, truncated to 1024 characters | | time | timestamp | Time of validation | --- # Signal filter plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/signal-filter/ The Signal Filter Plugin applies streaming digital IIR filters (Butterworth, Chebyshev I, Bessel — lowpass/highpass/bandpass/bandstop — or manually supplied second-order sections) to numeric time-series fields as they are written to InfluxDB 3 Enterprise. Filtering is causal and per series: each tag combination gets its own filter whose delay-line state persists in the trigger cache across WAL commits, so output over a stream of small writes is identical to filtering the whole signal at once. It pairs naturally with the[`signal_generator`](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/signal_generator)plugin but works with any measurement carrying numeric fields. * **Streaming, stateful**: per-series delay-line state persists in the trigger cache, so chunked writes filter identically to a single batch * **Preset or manual design**: SciPy-designed Butterworth, Chebyshev I, and Bessel prototypes, or raw second-order sections you supply * **Automatic sample-rate inference**: infers `fs` per series from the data when not configured explicitly * **Multi-field, multi-series**: filters each configured field and each tag combination independently * **Flexible output routing**: rename, prefix/suffix, and redirect the filtered field to another measurement or database ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments`flag (`influxdb3 create trigger`) or in the `trigger_arguments` field of the API. Values are strings; the plugin coerces them. Parameters may also come from a TOML file via `config_file_path` — see [TOML configuration](#toml-configuration). > > > **CLI limitation:** the `sos` argument is a JSON array containing commas, and`influxdb3 create trigger --trigger-arguments` splits on every comma, so the value > is fragmented before it reaches the plugin. Configure `sos` (i.e. `design_type=manual`) > through the InfluxDB 3 Explorer UI, the `/api/v3/configure/processing_engine_trigger`API, or a TOML file — not the CLI. Space-separated arguments such as `input_fields`and `tag_keys` are unaffected. > > ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the[InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Input parameters | Parameter | Type | Default | Description | |-------------------|------|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`input_measurement`|string|*(all tables)*| Table to filter. If omitted, every table the trigger fires for is filtered. | | `input_fields` |string| `value` | Space-separated numeric fields to filter; each is filtered independently. | | `tag_keys` |string| *(auto)* |Space-separated tag columns that define a series. Defaults to all string-valued columns except `time` and the input fields. Set explicitly if a string *field* would otherwise be mistaken for a tag.| ### Design parameters | Parameter | Type | Default | Description | |------------------------|-----------|------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `design_type` | string | `preset` | `preset` (SciPy-designed IIR) or `manual` (SOS coefficients supplied via `sos`). | | `prototype` | string | `butter` | Preset prototype: `butter`, `cheby1`, or `bessel`. | | `order` | integer | `4` | Filter order, 1–12. Band filters yield effective order 2N. | | `ripple` | float | — | Passband ripple in dB, 0.01–80 (0.1–3 typical). Required for `cheby1`; invalid otherwise. | | `filter_type` | string | `lowpass` | `lowpass`, `highpass`, `bandpass`, or `bandstop`. | | `fc` | float | — | Convenience alias for the single cutoff (Hz) of lowpass/highpass. Invalid for band types or together with the parameter it maps to. | | `fc1` | float | — | Lower cutoff (Hz). Required for highpass and band filters. | | `fc2` | float | — | Upper cutoff (Hz). Required for lowpass and band filters. Band filters require `fc1 < fc2`. | | `bessel_norm` | string | `phase` | Bessel normalization: `phase`, `delay`, or `mag`. Bessel only. | | `sos` |JSON string| — |Manual second-order sections as JSON `[[b0,b1,b2,a0,a1,a2], ...]`. Required for `design_type` `manual`; invalid otherwise. Rows are normalized by `a0`; unstable filters (any pole magnitude ≥ 1) are rejected.| | `sample_rate` | float |*(inferred)*| Sample rate in Hz for preset design. If omitted, inferred per series from the median inter-sample interval and frozen once enough samples are seen (see [Sample-rate inference](#sample-rate-inference)). | |`init_from_first_sample`| boolean | `true` | Initialize filter state from the first sample to suppress the startup transient. | ### Output parameters | Parameter | Type | Default | Description | |------------------------|------|----------------|-----------------------------------------------------------------------------------------------------------------------------| |`output_target_database`|string| *(trigger db)* | Database to write filtered output to. | | `output_measurement` |string|*(source table)*| Measurement to write filtered output to. | | `output_field` |string|*(source field)*| Base name override for the output field. Only valid when a single input field is configured. | | `field_prefix` |string| *(empty)* | Prefix for the output field name. `none` means no prefix; an empty value counts as unset. | | `field_suffix` |string| `_filtered` |Suffix for the output field name. `none` writes into the source field, replacing its samples; an empty value counts as unset.| | `config_file_path` |string| — |Path to a TOML file supplying parameters; its values override inline arguments. Relative paths resolve against `PLUGIN_DIR`. | The final output field name is `{field_prefix}{output_field or source_field}{field_suffix}`— by default, `value` becomes `value_filtered`. The raw input field is never copied to the output. ### Environment variables Every parameter can also come from an environment variable named`INFLUXDB3_SIGNAL_FILTER_` in upper case — for example,`INFLUXDB3_SIGNAL_FILTER_FC` sets `fc`. The environment is the lowest layer: a trigger argument overrides it, and the TOML file overrides both.`INFLUXDB3_SIGNAL_FILTER_CONFIG_FILE_PATH` names the TOML file when the trigger doesn’t carry a `config_file_path` argument. ### TOML configuration To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and reference the file with the `config_file_path` trigger argument (relative paths resolve against `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of`VIRTUAL_ENV`). The file and the inline trigger arguments are layered, and the file wins wherever both set a key, so a trigger can carry defaults that a file overrides. A blank value counts as unset and leaves the layer below it standing. See [`signal_filter_config_data_writes.toml`](signal_filter_config_data_writes.toml)for an annotated template. ## Data requirements * Input fields must be numeric (float or int). Null, boolean, and string values contribute no samples; NaN/±Inf samples are dropped (they would permanently poison IIR filter state). * Duplicate timestamps within a commit keep the last occurrence, matching the database’s last-write-wins semantics. * **Samples must arrive in time order across commits.** The plugin records the last filtered timestamp per series and drops late/backfilled samples with a warning — replaying old samples through a stateful causal filter would corrupt the output. To re-filter history, delete the trigger’s cache state (or recreate the trigger) and replay the data in order. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled (`--plugin-dir` configured). * **Python packages**: `numpy`, `scipy`, and `influxdata-plugin-utils`. ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install the Python dependencies into the plugin environment: ``` influxdb3 install package numpy scipy influxdata-plugin-utils ``` or via the HTTP API: ``` curl -X POST "http://localhost:8181/api/v3/configure/plugin_environment/install_packages" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{"packages": ["numpy", "scipy", "influxdata-plugin-utils"]}' ``` 3. Copy `signal_filter.py` into your plugin directory (or upload it with`POST /api/v3/plugins/files`). ## Trigger setup Create the trigger with `run_async: false` (the plugin’s per-series state assumes one invocation in flight at a time) and `error_behavior: log` or `retry` (state is saved only after all writes are buffered, so retries re-filter and re-emit the same points idempotently). Note the lowercase values — the live API rejects `Log`. ### Data write trigger ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename signal_filter.py \ --trigger-spec "table:signal" \ --trigger-arguments 'input_fields=value,filter_type=lowpass,fc=5.0,order=4' \ signal_lowpass ``` or via the HTTP API: ```bash curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "db": "mydb", "plugin_filename": "signal_filter.py", "trigger_name": "signal_lowpass", "trigger_specification": "table:signal", "trigger_settings": {"run_async": false, "error_behavior": "log"}, "trigger_arguments": { "input_fields": "value", "filter_type": "lowpass", "fc": "5.0", "order": "4" }, "disabled": false }' ``` ### Enable the trigger ```bash influxdb3 enable trigger --database mydb signal_lowpass ``` ### Testing without a trigger Use the WAL plugin test endpoint. Passing a fixed `cache_name` across calls exercises cross-commit state continuity: ```bash curl -X POST "http://localhost:8181/api/v3/plugin_test/wal" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "filename": "signal_filter.py", "database": "mydb", "input_lp": "signal,host=a value=1.0 1717000000000000000\nsignal,host=a value=2.0 1717000000100000000", "cache_name": "sigfilt_test", "input_arguments": {"filter_type": "lowpass", "fc": "5.0", "sample_rate": "10.0"} }' ``` Note: test caches expire after 30 minutes by default; production trigger caches persist indefinitely. ## Example usage ### Example 1: Lowpass smoothing (defaults) Attenuate high-frequency noise on the `value` field of the `signal` table with a 4th-order Butterworth lowpass at 5 Hz: ```bash # Create and enable the trigger influxdb3 create trigger \ --database mydb \ --plugin-filename signal_filter.py \ --trigger-spec "table:signal" \ --trigger-arguments 'input_fields=value,filter_type=lowpass,fc=5.0,order=4,sample_rate=100.0' \ signal_lowpass influxdb3 enable trigger --database mydb signal_lowpass # Write some data, then query the filtered field influxdb3 query \ --database mydb \ "SELECT time, value, value_filtered FROM signal ORDER BY time DESC LIMIT 10" ``` **Expected output**: for every written `value`, a `value_filtered` field appears at the same timestamp, with high-frequency content removed. The raw `value` is left untouched. ### Example 2: Bandpass with an explicit sample rate Isolate a 1–5 Hz band on a `vibration` field sampled at 50 Hz: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename signal_filter.py \ --trigger-spec "table:sensors" \ --trigger-arguments 'input_measurement=sensors,input_fields=vibration,filter_type=bandpass,fc1=1.0,fc2=5.0,order=4,sample_rate=50.0' \ vibration_bandpass influxdb3 enable trigger --database mydb vibration_bandpass ``` **Expected output**: a `vibration_filtered` field carrying only the 1–5 Hz band, per tag combination present in `sensors`. ### Example 3: Manual second-order sections Apply coefficients you designed elsewhere, routed to a separate measurement: ```bash curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Authorization: Bearer $INFLUXDB3_AUTH_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "db": "mydb", "plugin_filename": "signal_filter.py", "trigger_name": "manual_sos", "trigger_specification": "table:signal", "trigger_settings": {"run_async": false, "error_behavior": "log"}, "trigger_arguments": { "design_type": "manual", "sos": "[[0.2, 0.4, 0.2, 1.0, -0.4, 0.2]]", "output_measurement": "signal_filtered" }, "disabled": false }' ``` **Expected output**: filtered points written to `signal_filtered` (not the source`signal` table). Manual mode needs no sample rate — the coefficients are already digital. ## Sample-rate inference Preset design needs the sample rate. Resolution order per series: 1. Explicit `sample_rate` argument — always wins, so config corrections take effect immediately (a changed rate re-designs the filter and resets state). 2. Frozen per-series value from a previous successful inference. 3. Inference: `fs = 1e9 / median(inter-sample interval)` over accumulated timestamps. An estimate is frozen only after ≥ 8 inter-sample intervals have been observed; smaller commits accumulate timestamps across invocations and are skipped (with an info log, samples not emitted) until the threshold is met. At typical `signal_generator` rates a single commit clears it immediately. Manual mode needs no sample rate — the coefficients are already digital. ## Write-loop behavior Writing the output into the source measurement re-fires this trigger. This is safe by default: re-fired rows carry only the output field, the input field is null on them, and null values produce no samples. If your overrides resolve the output field to the *same name* as the input field in the same measurement and database — `field_suffix=none` with no`field_prefix`, or `input_fields=value_filtered` with `output_field=value` — the filtered values land on the source field at the same timestamps and**replace the source samples**, which cannot be undone. The re-fire that follows is dropped by the out-of-order guard (the samples are at or before the last processed timestamp), so it costs one extra empty invocation rather than running away. The plugin logs a prominent warning in that configuration; set`output_field`, `field_suffix`, `output_measurement`, or`output_target_database` to write elsewhere. ## Code overview ### Files * `signal_filter.py`: The main plugin code — config parsing/validation, filter design (preset + manual), the streaming runtime, per-series state helpers, and the `process_writes` WAL entry point. * `signal_filter_config_data_writes.toml`: Annotated example TOML configuration. * `test_signal_filter.py`: Unit and integration tests (mock-based; no running engine required). ### Logging Every log line is prefixed with a per-invocation task id: `[] signal_filter: ...`. Per invocation the plugin logs an info summary — tables and series processed, samples in, points written, and counts of dropped non-finite, out-of-order, and warm-up-skipped samples. Warnings cover out-of-order drops and the write-loop hazard; errors cover missing dependencies, invalid configuration, and filter design failures (for example a cutoff at or above the Nyquist frequency). Logs are available in the `system.processing_engine_logs` table. ### Main functions #### `process_writes(influxdb3_local, table_batches, args)` Entry point for data write triggers. Validates configuration, then for each table batch: extracts per-(field, series) samples, resolves the sample rate, designs (or reuses a memoized) filter, applies it with the cached delay-line state, writes the filtered points, and finally saves the advanced per-series state. Key operations: 1. Guards that `numpy`, `scipy`, and `influxdata-plugin-utils` are installed; logs an install command otherwise 2. Parses and validates the configuration (trigger arguments layered under a TOML file) 3. Warns on any write-loop hazard configuration 4. Groups rows into per-(field, series) samples, dropping null/non-numeric/non-finite values 5. Resolves the sample rate (explicit → frozen → inferred with warm-up) 6. Designs the IIR filter (memoized by parameters + `fs`) and checks stability 7. Applies the filter using cached `zi` state, writing each point with `write_sync(..., no_sync=True)` 8. Saves the advanced state (delay line, `fs`, coeff hash, last timestamp) to the cache ## Troubleshooting ### Common issues #### Issue: “required packages are not installed” **Cause**: The plugin environment lacks `numpy`, `scipy`, or `influxdata-plugin-utils`. **Solution**: Run `influxdb3 install package numpy scipy influxdata-plugin-utils` and re-enable the trigger. #### Issue: No output points **Cause**: Warm-up (sample-rate inference not yet frozen), a single-sample sparse series, or a non-numeric/null input field. **Solution**: * Check the info logs for warm-up skips: with no `sample_rate` argument the plugin waits for ≥ 8 inter-sample intervals per series before filtering. * A single-sample series with no `sample_rate` cannot infer a rate; set the argument explicitly for very sparse data. * Verify the input field is numeric and non-null in the written rows. #### Issue: “filter design failed: cutoff … must be within (0, fs/2)” **Cause**: The (possibly inferred) sample rate puts your cutoff at or beyond Nyquist. **Solution**: Set `sample_rate` explicitly or lower the cutoff. #### Issue: Output has a transient after a server restart **Cause**: The cache is cleared on restart, so filters re-initialize and inferred sample rates re-freeze (the cutoff may shift very slightly if the new estimate differs). **Solution**: This is expected; set `sample_rate` explicitly to pin the design. #### Issue: Series explosion / unexpected series **Cause**: The automatic tag heuristic treats every string-valued column as a tag, including string *fields*. **Solution**: Set `tag_keys` explicitly to bound the series set. ### Viewing logs ```bash influxdb3 query \ --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'signal_lowpass' ORDER BY event_time DESC LIMIT 20" ``` ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Signal filter plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/signal_filter) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | input_measurement | string | (all tables) | Table to filter. If omitted, every table the trigger fires for is filtered. | | input_fields | string | value | Space-separated numeric fields to filter; each is filtered independently. | | tag_keys | string | (auto) | Space-separated tag columns that define a series. Defaults to all string-valued columns except time and the input fields. Set explicitly if a string field would otherwise be mistaken for a tag. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | design_type | string | preset | preset (SciPy-designed IIR) or manual (SOS coefficients supplied via sos ). | | prototype | string | butter | Preset prototype: butter , cheby1 , or bessel . | | order | integer | 4 | Filter order, 1–12. Band filters yield effective order 2N. | | ripple | float | — | Passband ripple in dB, 0.01–80 (0.1–3 typical). Required for cheby1 ; invalid otherwise. | | filter_type | string | lowpass | lowpass , highpass , bandpass , or bandstop . | | fc | float | — | Convenience alias for the single cutoff (Hz) of lowpass/highpass. Invalid for band types or together with the parameter it maps to. | | fc1 | float | — | Lower cutoff (Hz). Required for highpass and band filters. | | fc2 | float | — | Upper cutoff (Hz). Required for lowpass and band filters. Band filters require fc1 < fc2 . | | bessel_norm | string | phase | Bessel normalization: phase , delay , or mag . Bessel only. | | sos | JSON string | — | Manual second-order sections as JSON [[b0,b1,b2,a0,a1,a2], ...] . Required for design_type manual ; invalid otherwise. Rows are normalized by a0 ; unstable filters (any pole magnitude ≥ 1) are rejected. | | sample_rate | float | (inferred) | Sample rate in Hz for preset design. If omitted, inferred per series from the median inter-sample interval and frozen once enough samples are seen (see Sample-rate inference ). | | init_from_first_sample | boolean | true | Initialize filter state from the first sample to suppress the startup transient. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | output_target_database | string | (trigger db) | Database to write filtered output to. | | output_measurement | string | (source table) | Measurement to write filtered output to. | | output_field | string | (source field) | Base name override for the output field. Only valid when a single input field is configured. | | field_prefix | string | (empty) | Prefix for the output field name. none means no prefix; an empty value counts as unset. | | field_suffix | string | _filtered | Suffix for the output field name. none writes into the source field, replacing its samples; an empty value counts as unset. | | config_file_path | string | — | Path to a TOML file supplying parameters; its values override inline arguments. Relative paths resolve against PLUGIN_DIR . | --- # Signal generator plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/signal-generator/ The Signal Generator Plugin lets new users produce realistic time-series data for testing InfluxDB 3 Enterprise functionality and downstream plugins without needing an external data source. Generate configurable waveform signals on a schedule to test alerts, anomaly detection, threshold checks, and dashboards from the moment you start. * **Zero dependencies**: Uses Python standard library only — no packages to install * **Composable waveforms**: Mix sine, square, triangle, sawtooth, noise, and spike signals by stacking them * **Realistic signals**: Default preset produces a signal centered at 30 with a slow sine trend, Gaussian noise, and occasional large spikes — immediately useful for alert testing * **Timestamp jitter**: Offsets point timestamps by default to simulate sensors that do not sample at perfectly uniform intervals * **Simulated gaps**: Simulates temporary data-source outages by default — bounded windows with no written points, useful for testing deadman alerts and gap handling. Set `gap_enabled=false` for continuous output * **Catch-up generation**: Generates data for the full span between executions regardless of trigger schedule — the only missing spans are the simulated gaps * **Flexible output**: Configure measurement name, field name, and custom tags per trigger ## Important CLI limitation Trigger configurations that pass JSON values, including `waveforms` and `tags`, currently cannot be created with the `influxdb3 create trigger --trigger-arguments` CLI flag. The CLI splits `--trigger-arguments` on every comma, including commas inside JSON arrays and objects, so values such as `waveforms=[{"type":"spike","value":7}]` are split into invalid fragments before they reach the plugin. Use the InfluxDB 3 Explorer UI or the `/api/v3/configure/processing_engine_trigger` API to create configured Signal Generator triggers. The default no-argument trigger can still be created with the CLI because it does not pass JSON or comma-containing values. Configured examples below use the API until the CLI parsing fix is available. ## Configuration Plugin parameters should be specified in the `trigger_arguments` field when creating a trigger with the API or InfluxDB 3 Explorer. Avoid the CLI for configured Signal Generator triggers until comma-aware parsing is available for JSON arrays and objects. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Optional parameters This plugin has no required parameters. | Parameter | Type | Default | Description | |--------------------------|-----------|---------------------|--------------------------------------------------------------------------------------------------------------| | `waveforms` |JSON string| *(default preset)* | JSON array of waveform config objects. If omitted, uses the built-in default preset. | | `measurement` | string | `signal` | Output measurement name. | | `field` | string | `value` | Output field name. | | `tags` |JSON string| *(none)* | Optional JSON object of tags to add to each data point. Example: `{"host": "server01"}`. | | `points_per_second` | float | `1.0` |Data point resolution in points per second. Controls how many points are generated per second of elapsed time.| |`jitter_amplitude_seconds`| float |10% of point interval| Maximum timestamp offset in seconds. Set to `0` to disable timestamp jitter. | | `jitter_seed` | integer | *(none)* | Optional seed for reproducible per-timestamp jitter, mainly useful for tests. | | `gap_enabled` | boolean | `true` | Enables simulated data-source outage gaps. Set to `false` for continuous output. | |`gap_min_duration_seconds`| float | `10.0` | Shortest simulated outage duration, in seconds. | |`gap_max_duration_seconds`| float | `30.0` | Longest simulated outage duration, in seconds. | |`gap_min_interval_seconds`| float | `120.0` | Shortest time between consecutive outage starts, in seconds. Must be at least `gap_max_duration_seconds`. | |`gap_max_interval_seconds`| float | `240.0` | Longest time between consecutive outage starts, in seconds. | | `gap_seed` | integer | *(none)* | Optional seed for reproducible gap timing. | | `target_database` | string | *(none)* | Optional target database. If omitted, writes to the trigger’s own database. | ### Timestamp jitter Timestamp jitter is enabled by default. If `jitter_amplitude_seconds` is omitted, the plugin uses 10% of the point interval: ```text point_interval_seconds = 1 / points_per_second default_jitter_amplitude_seconds = 0.10 * point_interval_seconds ``` Each generated timestamp receives an independent random offset from `[-jitter_amplitude_seconds, +jitter_amplitude_seconds]`. When `jitter_seed` is set, the offset is derived from the seed and the nominal timestamp, so each timestamp still gets its own offset even if scheduled executions generate one point at a time. Waveforms are still evaluated at the nominal cadence timestamps, then jitter is applied only to the timestamp that is written. With the same waveform seeds, jittered and non-jittered runs produce the same field values. Set `jitter_amplitude_seconds=0` to preserve perfectly regular timestamps. The plugin rejects jitter settings that could produce duplicate or reordered timestamps. ### Simulated gaps Simulated gaps are enabled by default. The plugin periodically simulates a data-source outage: points whose final (jittered) timestamps fall inside an outage window are not written, leaving realistic missing spans in the output. With the default configuration, each outage lasts 10–30 seconds, consecutive outages start 2–4 minutes apart, and about 11% of generated points are removed. The min/max bounds are hard guarantees: * Every outage duration lies in `[gap_min_duration_seconds, gap_max_duration_seconds]`. * The time between consecutive outage starts lies in `[gap_min_interval_seconds, gap_max_interval_seconds]`. * At least `gap_min_interval_seconds - gap_max_duration_seconds` seconds of continuous data separate consecutive outages (90 seconds at defaults). Gap windows are anchored to absolute time, so the same outage pattern is produced whether each scheduled execution writes one point or thousands, and catch-up after downtime contains the same gaps an uninterrupted run would have. When `gap_seed` is set, gap timing is fully reproducible. When omitted, the plugin generates a seed during first-run initialization and caches it, so the schedule stays coherent across executions; a restart that clears the trigger cache starts a new schedule. Missing spans are never backfilled: the generation boundary advances even when every point in a window was removed. Runs that removed points log a summary, for example: ```text Wrote 585 points to signal.value (600 generated, 15 removed by 1 gap windows) ``` Set `gap_enabled=false` to disable simulated gaps entirely. **Upgrade note:** existing triggers pick up default gaps on upgrade with no configuration change. Downstream demos that use deadman or no-data alerts will begin firing on the simulated outages — useful for exercising those alerts, but set `gap_enabled=false` if you need continuous data. ### Waveform types Waveform configuration is supplied as a JSON array. Each object requires a `type` key; all other parameters are optional and fall back to defaults. ```json [ {"type": "sine", "frequency": 0.1, "amplitude": 5.0}, {"type": "noise", "stddev": 0.3} ] ``` Multiple waveforms are summed together to produce the final signal value. #### Deterministic waveforms | Type | Parameters | Defaults | Notes | |----------|---------------------------------------------------------|-------------------------------------|---------------------------------------------------| |`constant`| `value` | `0.0` |Fixed y-offset; shifts the entire combined signal. | | `sine` | `frequency`, `amplitude`, `offset`, `phase` | `0.05 Hz`, `1.0`, `0.0`, `0.0` |0.05 Hz = \~20 s period. Anchored to absolute time.| | `square` |`frequency`, `amplitude`, `offset`, `phase`, `duty_cycle`|`0.05 Hz`, `1.0`, `0.0`, `0.0`, `0.5`|`duty_cycle` is 0–1, fraction of period spent high.| |`triangle`| `frequency`, `amplitude`, `offset`, `phase` | `0.05 Hz`, `1.0`, `0.0`, `0.0` | Linear ramp up then down. | |`sawtooth`| `frequency`, `amplitude`, `offset`, `phase` | `0.05 Hz`, `1.0`, `0.0`, `0.0` | Linear ramp up, instant drop. | #### Stochastic waveforms | Type | Parameters | Defaults | Notes | |-------|-------------------------------------------------------|-----------------------------|---------------------------------------------------------------------------------------------| |`noise`| `stddev`, `mean`, `seed` | `0.1`, `0.0`, `None` | Gaussian noise. `seed` enables reproducible sequences. | |`spike`|`probability`, `min_amplitude`, `max_amplitude`, `seed`|`0.01`, `5.0`, `10.0`, `None`|1% chance of a spike per point. Magnitude drawn uniformly from `[min, max]` with random sign.| ### Default preset When no `waveforms` argument is provided, the plugin uses this preset: ```json [ {"type": "constant", "value": 30.0}, {"type": "sine", "frequency": 0.005, "amplitude": 10.0}, {"type": "noise", "stddev": 0.5}, {"type": "spike", "probability": 0.005, "min_amplitude": 8.0, "max_amplitude": 15.0} ] ``` This produces a signal centered around 30 with a slow-moving sine wave (period \~200 s / \~3.3 minutes), light Gaussian noise (stddev 0.5), and occasional large spikes (0.5% chance per point, magnitude 8–15). Designed to be immediately useful for testing alerts and anomaly detection without any configuration. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: No additional packages required (uses Python standard library only) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. No additional Python packages are required for this plugin. ## Trigger setup ### Scheduled trigger with API configuration Use the API or InfluxDB 3 Explorer for configured triggers, especially when passing JSON values in `waveforms` or `tags`. ```bash curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Content-Type: application/json" \ -d '{ "db": "signals", "plugin_filename": "gh:influxdata/signal_generator/signal_generator.py", "trigger_name": "signal_generator_trigger", "trigger_specification": "every:10s", "trigger_arguments": { "waveforms": "[{\"type\":\"constant\",\"value\":30.0},{\"type\":\"sine\",\"frequency\":0.005,\"amplitude\":10.0},{\"type\":\"noise\",\"stddev\":0.5}]", "measurement": "signal", "field": "value", "jitter_amplitude_seconds": "0.2" }, "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` ### Default CLI trigger The CLI can create a default no-argument trigger because no JSON value is passed. Do not use the CLI for `waveforms` or `tags` until the CLI parsing issue is fixed. ```bash influxdb3 create trigger \ --database signals \ --path "gh:influxdata/signal_generator/signal_generator.py" \ --trigger-spec "every:10s" \ signal_generator_default ``` **Note:** The first execution initializes the plugin (stores the current time in cache) and does not write any data. Data begins flowing on the second execution. ## Example usage ### Example 1: Basic (defaults) Use the default preset with no configuration. Creates a signal centered around 30 with sine trend, noise, and spikes: ```bash # Create the trigger influxdb3 create trigger \ --database signals \ --path "gh:influxdata/signal_generator/signal_generator.py" \ --trigger-spec "every:10s" \ signal_basic # Enable the trigger influxdb3 enable trigger --database signals signal_basic # Query signal data (after the second execution) influxdb3 query \ --database signals \ "SELECT time, value FROM signal ORDER BY time DESC LIMIT 10" ``` ### Example 2: Custom waveforms (square + noise) Generate a square wave with added noise — useful for simulating on/off processes with sensor jitter: ```bash curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Content-Type: application/json" \ -d '{ "db": "signals", "plugin_filename": "gh:influxdata/signal_generator/signal_generator.py", "trigger_name": "signal_square_noise", "trigger_specification": "every:10s", "trigger_arguments": { "waveforms": "[{\"type\":\"square\",\"frequency\":0.02,\"amplitude\":5.0,\"duty_cycle\":0.3},{\"type\":\"noise\",\"stddev\":0.2}]" }, "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` ### Example 3: Custom output (measurement, field, tags) Write signal data to a specific measurement with custom field name and tags for multi-series dashboards: ```bash curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Content-Type: application/json" \ -d '{ "db": "signals", "plugin_filename": "gh:influxdata/signal_generator/signal_generator.py", "trigger_name": "signal_custom_output", "trigger_specification": "every:10s", "trigger_arguments": { "measurement": "cpu_temperature", "field": "temperature", "tags": "{\"host\":\"server01\",\"region\":\"us-west\"}" }, "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` ### Example 4: Multiple independent signals Run two triggers to generate multiple independent signals in the same database. Each trigger has its own cache and waveform configuration: ```bash # Signal A: slow sine wave (temperature-like) curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Content-Type: application/json" \ -d '{ "db": "signals", "plugin_filename": "gh:influxdata/signal_generator/signal_generator.py", "trigger_name": "signal_temperature", "trigger_specification": "every:10s", "trigger_arguments": { "waveforms": "[{\"type\":\"constant\",\"value\":22.0},{\"type\":\"sine\",\"frequency\":0.002,\"amplitude\":3.0},{\"type\":\"noise\",\"stddev\":0.1}]", "measurement": "environment", "field": "temperature", "tags": "{\"sensor\":\"A\"}" }, "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' # Signal B: faster oscillation with spikes (pressure-like) curl -X POST "http://localhost:8181/api/v3/configure/processing_engine_trigger" \ -H "Content-Type: application/json" \ -d '{ "db": "signals", "plugin_filename": "gh:influxdata/signal_generator/signal_generator.py", "trigger_name": "signal_pressure", "trigger_specification": "every:10s", "trigger_arguments": { "waveforms": "[{\"type\":\"constant\",\"value\":101.3},{\"type\":\"sine\",\"frequency\":0.01,\"amplitude\":0.8},{\"type\":\"spike\",\"probability\":0.01,\"min_amplitude\":2.0,\"max_amplitude\":5.0}]", "measurement": "environment", "field": "pressure", "tags": "{\"sensor\":\"B\"}" }, "trigger_settings": { "run_async": false, "error_behavior": "log" }, "disabled": false }' ``` ### Waveform JSON examples #### Single sine wave (all defaults) ```json [{"type": "sine"}] ``` Produces a sine at 0.05 Hz (20 s period), amplitude 1.0, centered at 0. #### Combined sine and noise ```json [ {"type": "constant", "value": 50.0}, {"type": "sine", "frequency": 0.01, "amplitude": 15.0}, {"type": "noise", "stddev": 1.0} ] ``` Produces a signal centered at 50 with a 100 s period sine (±15) and moderate noise. #### Square wave with custom duty cycle ```json [ {"type": "square", "frequency": 0.05, "amplitude": 10.0, "duty_cycle": 0.25} ] ``` Produces a square wave spending 25% of each period at +10 and 75% at -10. #### Full custom signal ```json [ {"type": "constant", "value": 100.0}, {"type": "triangle", "frequency": 0.005, "amplitude": 20.0}, {"type": "noise", "stddev": 2.0}, {"type": "spike", "probability": 0.02, "min_amplitude": 30.0, "max_amplitude": 60.0} ] ``` Produces a triangle wave centered at 100, with 2% spike probability and magnitude 30–60. ## Output schema ### Measurement: `signal` (default, configurable) **Tags:** Tags are optional and have no defaults. Tags are only present when specified via the `tags` trigger argument. Example with `tags={"host": "server01", "region": "us-west"}`: * `host`: `server01` * `region`: `us-west` **Fields:** * `value` (float64): The computed signal value at each timestamp. Field name is configurable via the `field` argument. **Timestamp:** * Nanosecond precision Unix epoch timestamps. Each point’s timestamp reflects simulated sample time, not the wall-clock time of the write. * Timestamp jitter is enabled by default, so adjacent timestamps are usually close to, but not exactly on, the nominal cadence grid. Set `jitter_amplitude_seconds=0` for regular intervals. * Simulated gaps are enabled by default, so output contains periodic missing spans of 10–30 seconds. Set `gap_enabled=false` for continuous data. ### Line protocol examples Without tags (default): ``` signal value=32.47 1712678399918245021 signal value=31.89 1712678401084217139 signal value=33.21 1712678401962364102 ``` With custom measurement, field, and tags: ``` cpu_temperature,host=server01,region=us-west temperature=72.3 1712678400000000000 cpu_temperature,host=server01,region=us-west temperature=71.8 1712678401000000000 ``` ## Example queries ### View the most recent signal values ```sql SELECT time, value FROM signal WHERE time > now() - INTERVAL '5 minutes' ORDER BY time DESC LIMIT 20; ``` ### Compute rolling statistics ```sql SELECT time_bucket(time, INTERVAL '1 minute') AS minute, AVG(value) AS avg_value, MIN(value) AS min_value, MAX(value) AS max_value FROM signal WHERE time > now() - INTERVAL '1 hour' GROUP BY minute ORDER BY minute DESC; ``` ### Find spikes (values far from the mean) ```sql SELECT time, value FROM signal WHERE time > now() - INTERVAL '1 hour' AND ABS(value - 30.0) > 10.0 ORDER BY time DESC; ``` ### Compare multiple signals ```sql SELECT time, temperature, pressure FROM environment WHERE time > now() - INTERVAL '30 minutes' ORDER BY time DESC LIMIT 50; ``` ## Code overview ### Files * `signal_generator.py`: The main plugin code containing all waveform factories, config parsing, time series generation, and the scheduled trigger entry point. ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Entry point for scheduled triggers. Orchestrates the full execution: parses config, reads the cache, generates timestamps and signal values, writes points, and updates the cache. Key operations: 1. Parses waveform, output, resolution, timestamp jitter, and gap configuration from trigger arguments 2. Reads `last_time` from the trigger-local cache 3. On first run: stores current time (and the auto-generated gap seed when gaps are enabled and unseeded) and returns without writing (initialization) 4. Generates timestamps in the half-open interval `(last_time, now]` 5. Evaluates the combined waveform at each nominal timestamp 6. Applies timestamp jitter without changing field values 7. Removes points whose final timestamps fall inside simulated gap windows (see [Simulated gaps](#simulated-gaps)) 8. Writes the remaining points using `write_sync` with `no_sync=True` 9. Updates `last_time` in the cache #### Waveform factories `make_constant`, `make_sine`, `make_square`, `make_triangle`, `make_sawtooth`, `make_noise`, `make_spike` — each returns a function `f(t: float) -> float` where `t` is Unix epoch seconds. Deterministic waveforms are anchored to absolute time (same `t` always produces the same value). Stochastic waveforms draw from a `random.Random` instance per factory call. #### `combine(waveform_fns)` Composes a list of waveform functions by summing their outputs: `combined(t) = sum(fn(t) for fn in waveform_fns)`. #### `generate_timestamps(start, end, points_per_second)` Generates timestamps in the half-open interval `(start, end]`. The interval is exclusive of `start` to prevent duplicate points across consecutive executions. #### `apply_timestamp_jitter(points, amplitude_seconds, seed)` Offsets generated point timestamps after signal evaluation. Values are unchanged; only timestamps are moved. #### `gap_window(gap_config, n)` / `in_gap(gap_config, t)` / `apply_gaps(points, gap_config)` Implements the simulated gap schedule: absolute time is divided into fixed strata, and stratum `n` contains one gap whose start offset and duration are derived from a `blake2b` hash of `(seed, n)`. Every window is a pure function of seed, configuration, and gap index, so results are identical regardless of how many points each scheduled call generates. `apply_gaps` filters final (jittered) timestamps and reports how many points were removed. See `GAP_DESIGN.md` for the full design. ## Troubleshooting ### Common issues #### Issue: No data appearing after enabling the trigger **Cause**: The first execution initializes the plugin (stores the current time) and does not write any data. This is by design. **Solution**: Wait for the second execution. Check the logs to confirm initialization succeeded: ```bash influxdb3 query \ --database signals \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'signal_basic' ORDER BY event_time DESC LIMIT 5" ``` Look for a log entry containing `"Signal generator initializing"` — this confirms the first run completed successfully and data will appear on the next execution. #### Issue: Config parsing error in logs **Cause**: Malformed JSON in `waveforms` or `tags` arguments, or an unknown waveform type. **Solution**: Validate your JSON before passing it as a trigger argument. Supported waveform types are: `constant`, `sine`, `square`, `triangle`, `sawtooth`, `noise`, `spike`. Check for typos and ensure the JSON array is valid: ```bash # Validate JSON locally echo '[{"type": "sine"}, {"type": "noise"}]' | python3 -m json.tool ``` #### Issue: Missing spans in query results **Cause**: Simulated gaps are enabled by default. The plugin periodically simulates a data-source outage and writes no points inside the outage window. **Solution**: This is expected behavior. To confirm a missing span is a simulated gap rather than a real failure, check the logs — runs that removed points report `... (N generated, M removed by K gap windows)`. Set `gap_enabled=false` in the trigger arguments for continuous output. #### Issue: No points generated (interval too short) **Cause**: The trigger fires more frequently than `1 / points_per_second` seconds, so no timestamps fall in the interval. **Solution**: Increase `points_per_second` to match your trigger frequency, or reduce trigger frequency. For example, if firing every 1 s with `points_per_second=1.0`, at least one point is generated per execution. If firing every 100ms, increase to `points_per_second=10.0`. #### Issue: Invalid jitter config in logs **Cause**: `jitter_amplitude_seconds` is negative, non-numeric, or too large for the configured `points_per_second`, or `jitter_seed` is not an integer. **Solution**: Reduce `jitter_amplitude_seconds`, reduce `points_per_second`, set `jitter_amplitude_seconds=0`, or use an integer `jitter_seed`. The plugin requires at least a 1 microsecond minimum gap between any two possible jittered timestamps. #### Issue: Write failures for individual points **Cause**: Intermittent write errors. The plugin logs each failed point and continues. **Solution**: Check logs for write error details: ```bash influxdb3 query \ --database signals \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'signal_basic' AND log_text LIKE '%Write failed%' ORDER BY event_time DESC LIMIT 10" ``` #### Issue: Signal is not phase-continuous after restart **Cause**: Deterministic waveforms (sine, square, triangle, sawtooth) are anchored to absolute Unix time, so they are always at the correct phase for a given wall-clock time. If the signal appears discontinuous, check that the `frequency` parameter is the same before and after the restart. **Cause of gaps in data**: Short gaps (10–30 seconds by default) are simulated outages; see [Simulated gaps](#simulated-gaps). For downtime-related gaps: if the trigger was disabled or InfluxDB was stopped, the cache retains `last_time` from the last successful execution, and on restart the plugin generates all missing points from `last_time` to the current time — the caught-up span contains the same simulated gap windows an uninterrupted run would have. ### Viewing logs ```bash influxdb3 query \ --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'signal_basic' ORDER BY event_time DESC LIMIT 20" ``` ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Signal generator plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/signal_generator) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | waveforms | JSON string | (default preset) | JSON array of waveform config objects. If omitted, uses the built-in default preset. | | measurement | string | signal | Output measurement name. | | field | string | value | Output field name. | | tags | JSON string | (none) | Optional JSON object of tags to add to each data point. Example: {"host": "server01"} . | | points_per_second | float | 1.0 | Data point resolution in points per second. Controls how many points are generated per second of elapsed time. | | jitter_amplitude_seconds | float | 10% of point interval | Maximum timestamp offset in seconds. Set to 0 to disable timestamp jitter. | | jitter_seed | integer | (none) | Optional seed for reproducible per-timestamp jitter, mainly useful for tests. | | gap_enabled | boolean | true | Enables simulated data-source outage gaps. Set to false for continuous output. | | gap_min_duration_seconds | float | 10.0 | Shortest simulated outage duration, in seconds. | | gap_max_duration_seconds | float | 30.0 | Longest simulated outage duration, in seconds. | | gap_min_interval_seconds | float | 120.0 | Shortest time between consecutive outage starts, in seconds. Must be at least gap_max_duration_seconds . | | gap_max_interval_seconds | float | 240.0 | Longest time between consecutive outage starts, in seconds. | | gap_seed | integer | (none) | Optional seed for reproducible gap timing. | | target_database | string | (none) | Optional target database. If omitted, writes to the trigger’s own database. | | Type | Parameters | Defaults | Notes | | --- | --- | --- | --- | | Type | Parameters | Defaults | Notes | | constant | value | 0.0 | Fixed y-offset; shifts the entire combined signal. | | sine | frequency , amplitude , offset , phase | 0.05 Hz , 1.0 , 0.0 , 0.0 | 0.05 Hz = ~20 s period. Anchored to absolute time. | | square | frequency , amplitude , offset , phase , duty_cycle | 0.05 Hz , 1.0 , 0.0 , 0.0 , 0.5 | duty_cycle is 0–1, fraction of period spent high. | | triangle | frequency , amplitude , offset , phase | 0.05 Hz , 1.0 , 0.0 , 0.0 | Linear ramp up then down. | | sawtooth | frequency , amplitude , offset , phase | 0.05 Hz , 1.0 , 0.0 , 0.0 | Linear ramp up, instant drop. | | Type | Parameters | Defaults | Notes | | --- | --- | --- | --- | | Type | Parameters | Defaults | Notes | | noise | stddev , mean , seed | 0.1 , 0.0 , None | Gaussian noise. seed enables reproducible sequences. | | spike | probability , min_amplitude , max_amplitude , seed | 0.01 , 5.0 , 10.0 , None | 1% chance of a spike per point. Magnitude drawn uniformly from [min, max] with random sign. | --- # Simple data replicator plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/simple-data-replicator/ The Simple Data Replicator Plugin replicates data from a local InfluxDB 3 Enterprise instance to a remote InfluxDB 3 Enterprise instance over HTTP. It supports table and field filtering, table and field renaming, and runs either on a schedule (pulling a time window from a source measurement) or on every write (replicating committed WAL batches). Data is buffered in a compressed JSONL queue file and delivered with automatic retries, so it survives transient remote failures. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. This plugin supports TOML configuration files, which can be specified using the `config_file_path` parameter; values in the file override the arguments passed to the trigger. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Scheduler mode parameters Multi-value arguments (`excluded_fields`, `field_renames`) use delimited strings on the CLI and native TOML types (lists/tables) in a config file. | Argument | Description |Required| Example (CLI) | Example (TOML) | Default | |------------------------|----------------------------------------------------------------------------------------|--------|------------------|----------------------------------------|--------------------| | `host` | Remote InfluxDB host URL. Default scheme `http`, port `8181`. | `true` | `example.com` | `host = "example.com"` | `None` | | `remote_token` | Remote InfluxDB API token. Falls back to `REMOTE_INFLUXDB_TOKEN` env var. |`false` | `apiv3_AuHk...` | `remote_token = "apiv3_AuHk..."` | `None` | | `database` | Remote database name. | `true` | `remote_db` | `database = "remote_db"` | `None` | | `source_measurement` | Measurement to replicate. | `true` | `home` | `source_measurement = "home"` | `None` | | `window` | Time window per job. Format `` (`s`, `min`, `h`, `d`, `w`). | `true` | `1h` | `window = "1h"` | `None` | | `unique_file_suffix` | Unique suffix for the queue file to avoid conflicts. | `true` | `abcd1234` | `unique_file_suffix = "abcd1234"` | `None` | | `max_size` | Maximum size for the queue file in MB. Integer ≥ 1. |`false` | `1024` | `max_size = 1024` | `512` | | `verify_ssl` | Whether to verify SSL certificates when connecting via HTTPS. |`false` | `false` | `verify_ssl = false` | `true` | | `max_retries` | Maximum number of retries for write operations. Integer ≥ 1. |`false` | `5` | `max_retries = 5` | `3` | |`queue_flush_chunk_size`| Line-protocol entries sent per remote flush request. Integer ≥ 1. |`false` | `1000` | `queue_flush_chunk_size = 1000` | `500` | | `excluded_fields` | Fields and tags to exclude. CLI: space-separated. TOML: list. |`false` | `co location` | `excluded_fields = ["co", "location"]` | `None` | | `target_table` | New name for the measurement in the remote database. |`false` | `home2` | `target_table = "home2"` |`source_measurement`| | `field_renames` | Field renames. CLI: `old:new` pairs separated by space. TOML: table. |`false` |`temp:temperature`|`field_renames = {temp = "temperature"}`| `None` | | `offset` |Time offset to apply to the window. Format `` (`s`, `min`, `h`, `d`, `w`).|`false` | `10min` | `offset = "10min"` | `0` | | `config_file_path` | Path to the TOML config file. Absolute, or relative to `PLUGIN_DIR`. |`false` | `config.toml` | — | `None` | ### Data write mode parameters Multi-value arguments (`tables`, `excluded_fields`, `tables_rename`, `field_renames`) use delimited strings on the CLI and native TOML types (lists/tables) in a config file. | Argument | Description |Required| Example (CLI) | Example (TOML) |Default| |------------------------|-----------------------------------------------------------------------------------------------------|--------|------------------------------------|-------------------------------------------------------------------------|-------| | `host` | Remote InfluxDB host URL. Default scheme `http`, port `8181`. | `true` | `example.com` | `host = "example.com"` |`None` | | `remote_token` | Remote InfluxDB API token. Falls back to `REMOTE_INFLUXDB_TOKEN` env var. |`false` | `apiv3_AuHk...` | `remote_token = "apiv3_AuHk..."` |`None` | | `database` | Remote database name. | `true` | `remote_db` | `database = "remote_db"` |`None` | | `unique_file_suffix` | Unique suffix for the queue file to avoid conflicts. | `true` | `wxyz5678` | `unique_file_suffix = "wxyz5678"` |`None` | | `tables` | Tables to replicate; all tables if omitted. CLI: space-separated. TOML: list. |`false` | `home home2` | `tables = ["home", "home2"]` |`None` | | `verify_ssl` | Whether to verify SSL certificates when connecting via HTTPS. |`false` | `false` | `verify_ssl = false` |`true` | | `max_size` | Maximum size for the queue file in MB. Integer ≥ 1. |`false` | `1024` | `max_size = 1024` | `512` | |`queue_flush_chunk_size`| Line-protocol entries sent per remote flush request. Integer ≥ 1. |`false` | `1000` | `queue_flush_chunk_size = 1000` | `500` | | `excluded_fields` |Fields/tags to exclude per table. CLI: `table:f1|f2` blocks separated by space. TOML: table of lists.|`false` | `home:co|location home2:temp` | `excluded_fields = {home = ["co", "location"], home2 = ["temp"]}` |`None` | | `tables_rename` | Table renames. CLI: `old:new` pairs separated by space. TOML: table. |`false` | `home:home2 office:office2` | `tables_rename = {home = "home2", office = "office2"}` |`None` | | `field_renames` |Field renames per table. CLI: `table:old=new|old=new` blocks separated by space. TOML: nested table. |`false` |`home:temp=temperature|hum=humidity`|`[field_renames]`
`home = {temp = "temperature", hum = "humidity"}`|`None` | | `config_file_path` | Path to the TOML config file. Absolute, or relative to `PLUGIN_DIR`. |`false` | `config.toml` | — |`None` | ### Authentication The remote API token is read from the `remote_token` argument first; if it is not set, the plugin falls back to the `REMOTE_INFLUXDB_TOKEN` environment variable. If neither is provided, the plugin returns an error. ### File path resolution The `config_file_path` and the queue directory follow the same resolution logic: * **Absolute paths** are used as-is. * **Relative paths** are resolved from the `PLUGIN_DIR` environment variable. If `PLUGIN_DIR` is not set, the plugin falls back to `INFLUXDB3_PLUGIN_DIR` and then the parent of `VIRTUAL_ENV`. If none of these are available, the plugin returns an error. #### Example TOML configuration * [simple\_data\_replicator\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/simple_data_replicator/simple_data_replicator_config_scheduler.toml) — scheduler mode * [simple\_data\_replicator\_config\_data\_writes.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/simple_data_replicator/simple_data_replicator_config_data_writes.toml) — data write mode ## Data requirements The plugin assumes the table schema is already defined in the local database; it relies on this schema to retrieve the field and tag names required for processing. The remote target database must already exist. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: * `influxdb3-python` (InfluxDB 3 Enterprise client library used to write to the remote instance) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/plugins ``` 2. Install required Python packages: ``` influxdb3 install package influxdb3-python ``` ## Trigger setup ### Scheduler mode Periodically pulls data from a local measurement within a time window and replicates it to the remote instance: ```bash influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/simple_data_replicator/simple_data_replicator.py \ --trigger-spec "every:10s" \ --trigger-arguments host=example.com,remote_token=apiv3_token,database=remote_db,source_measurement=home,window=10s,unique_file_suffix=abcd1234 \ simple_data_replicator_trigger ``` Enable the trigger to start periodic replication: ```bash influxdb3 enable trigger --database mydb simple_data_replicator_trigger ``` ### Data write mode Replicates data automatically whenever the Write-Ahead Log (WAL) is flushed after a write: ```bash influxdb3 create trigger \ --database mydb \ --trigger-spec "all_tables" \ --plugin-filename gh:influxdata/simple_data_replicator/simple_data_replicator.py \ --trigger-arguments host=example.com,remote_token=apiv3_token,database=remote_db,tables="home home2",field_renames="home:hum=humidity|temp=temperature",unique_file_suffix=wxyz5678 \ simple_data_replicator_trigger ``` Enable the trigger to start replication on WAL flush events: ```bash influxdb3 enable trigger --database mydb simple_data_replicator_trigger ``` > > > **Note:** > > > > * The plugin is triggered whenever the WAL is flushed, which happens after data is written. > * It processes the batches of data written to the database, using the arguments specified in `--trigger-arguments`. > * The plugin caches the list of measurements in the database and the tag names for each measurement for one hour to avoid repeated queries. > * In data write mode the remote flush is attempted once per invocation (no in-call retries); any data that fails to deliver stays in the queue and is retried on the next WAL flush. > * Run the trigger in the default synchronous mode. Do not use `--run-asynchronous`: concurrent invocations would access the same queue file in parallel and can corrupt it or lose data. Synchronous execution serializes invocations and keeps queue access safe. > > ### Replication with TOML configuration Recommended for complex filtering and renaming. Store the arguments in a TOML file and reference it with the `config_file_path` argument (absolute, or relative to `PLUGIN_DIR`): ```bash export PLUGIN_DIR=~/plugins cp simple_data_replicator_config_scheduler.toml $PLUGIN_DIR/ influxdb3 create trigger \ --database mydb \ --plugin-filename gh:influxdata/simple_data_replicator/simple_data_replicator.py \ --trigger-spec "every:10s" \ --trigger-arguments config_file_path=simple_data_replicator_config_scheduler.toml \ simple_data_replicator_trigger ``` ## Queue management The plugin buffers data in a compressed JSONL queue file before delivery, ensuring reliable replication. The file is unique per trigger configuration to avoid conflicts. * **Location**: the `sdr_queues` subdirectory of the resolved plugin directory (see [File path resolution](#file-path-resolution)). For example, with `PLUGIN_DIR=/opt/plugins` the queue lives in `/opt/plugins/sdr_queues/`. * **File name**: * Data write mode: `sdr_queue_writes_.jsonl.gz` * Scheduler mode: `sdr_queue_schedule_.jsonl.gz` * **Format**: gzip-compressed JSON Lines, one JSON object per line. * **Lifecycle**: data is appended before replication and removed after it is delivered successfully. * **Maximum size**: controlled by `max_size` (default 512 MB); exceeding it raises an error. Use a distinct `unique_file_suffix` for every trigger, and monitor the file size to avoid disk overflow. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs" ``` Example output: ```text +-------------------------------+-----------------+--------------+----------------------------------------------------------------------------------------------------------+ | event_time | trigger_name | log_level | log_text | +-------------------------------+-----------------+--------------+----------------------------------------------------------------------------------------------------------+ | 2025-05-14T16:31:10.033295886 | my_scheduler | INFO | [4726cf36-3b15-442e-bd9d-f9b768ad8781] Finished execution in 31ms 449us 193ns | | 2025-05-14T16:31:10.033275724 | my_scheduler | INFO | [4726cf36-3b15-442e-bd9d-f9b768ad8781] Replicated 10 lines to remote instance | | 2025-05-14T16:31:10.033232792 | my_scheduler | INFO | [4726cf36-3b15-442e-bd9d-f9b768ad8781] Queued 10 lines from table1,table2 | | 2025-05-14T16:31:00.046579787 | some_scheduler | ERROR | [528a316e-b28c-4bd2-8c05-07ad50bc1de2] Error during replication: Connection failed | | 2025-05-14T16:31:00.025482558 | my_scheduler | INFO | [cd5caa89-47db-443c-9621-5f90a129a0cc] Starting data replication process | +-------------------------------+-----------------+--------------+----------------------------------------------------------------------------------------------------------+ ``` Each entry carries the task ID in `log_text`, so logs from a single execution can be correlated. ## Troubleshooting ### Check plugin logs ```bash influxdb3 query --database _internal \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'simple_data_replicator_trigger' ORDER BY time DESC LIMIT 20" ``` ### Common issues #### “Missing remote token” Set the `remote_token` argument or the `REMOTE_INFLUXDB_TOKEN` environment variable. #### Relative `config_file_path` not found For relative paths, ensure `PLUGIN_DIR` (or `INFLUXDB3_PLUGIN_DIR`) is set. For absolute paths, verify the file exists. #### Queue file exceeds `max_size` Replication is failing or falling behind. Check the logs for delivery errors, fix the remote connection, or raise `max_size`. #### Corrupted or invalid queue file If the plugin reports persistent errors caused by a corrupted queue file, delete it to reset the queue: ```bash # Data write triggers: rm $PLUGIN_DIR/sdr_queues/sdr_queue_writes_.jsonl.gz # Scheduled triggers: rm $PLUGIN_DIR/sdr_queues/sdr_queue_schedule_.jsonl.gz ``` **Warning:** deleting the queue file permanently loses any entries that have not yet been replicated. Only delete it if you are certain the data is corrupted or you accept the loss. ## Architecture ### How It Works 1. **Trigger fires**: on a schedule (`every:`) or on each WAL flush (`all_tables`). 2. **Read rows**: scheduler mode queries the time window from `source_measurement`; data write mode reads the committed table batches. 3. **Filter and rename**: excluded tables and fields are dropped, then table and field renames are applied. 4. **Queue**: rows are converted to points and appended to the compressed JSONL queue file. If the queue already reached `max_size`, new rows are skipped (with a warning) but the existing queue is still flushed, so the buffer can drain once the remote recovers. 5. **Replicate**: the queue is flushed to the remote instance over HTTP in chunks of `queue_flush_chunk_size` lines (default 500). Scheduler mode retries failures and rate limits; data write mode attempts each flush once. 6. **Cleanup**: entries are removed from the queue only after they are delivered successfully. ### Performance Optimization * **Client caching**: the remote client is cached and reused across invocations instead of reconnecting each cycle. It is rebuilt when the connection settings change, and evicted after retries are exhausted on a connection failure so the next cycle reconnects cleanly. * **Chunked flush**: the queue is written to the remote instance in chunks of `queue_flush_chunk_size` lines (default 500), bounding request size and memory for large backlogs. Each successful chunk is removed from the queue, so a mid-flush failure does not re-send already delivered data. * **Schema caching**: the measurement list and per-measurement tag names are cached for one hour to avoid repeated queries. * **Compressed queue**: gzip-compressed JSONL keeps the on-disk buffer small. * **Retry logic**: in scheduler mode, transient failures and rate limits are retried up to `max_retries` times; in data write mode failed data stays queued for the next WAL flush. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Simple data replicator plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/simple_data_replicator) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Argument | Description | Required | Example (CLI) | Example (TOML) | Default | | --- | --- | --- | --- | --- | --- | | Argument | Description | Required | Example (CLI) | Example (TOML) | Default | | host | Remote InfluxDB host URL. Default scheme http , port 8181 . | true | example.com | host = "example.com" | None | | remote_token | Remote InfluxDB API token. Falls back to REMOTE_INFLUXDB_TOKEN env var. | false | apiv3_AuHk... | remote_token = "apiv3_AuHk..." | None | | database | Remote database name. | true | remote_db | database = "remote_db" | None | | source_measurement | Measurement to replicate. | true | home | source_measurement = "home" | None | | window | Time window per job. Format ( s , min , h , d , w ). | true | 1h | window = "1h" | None | | unique_file_suffix | Unique suffix for the queue file to avoid conflicts. | true | abcd1234 | unique_file_suffix = "abcd1234" | None | | max_size | Maximum size for the queue file in MB. Integer ≥ 1. | false | 1024 | max_size = 1024 | 512 | | verify_ssl | Whether to verify SSL certificates when connecting via HTTPS. | false | false | verify_ssl = false | true | | max_retries | Maximum number of retries for write operations. Integer ≥ 1. | false | 5 | max_retries = 5 | 3 | | queue_flush_chunk_size | Line-protocol entries sent per remote flush request. Integer ≥ 1. | false | 1000 | queue_flush_chunk_size = 1000 | 500 | | excluded_fields | Fields and tags to exclude. CLI: space-separated. TOML: list. | false | co location | excluded_fields = ["co", "location"] | None | | target_table | New name for the measurement in the remote database. | false | home2 | target_table = "home2" | source_measurement | | field_renames | Field renames. CLI: old:new pairs separated by space. TOML: table. | false | temp:temperature | field_renames = {temp = "temperature"} | None | | offset | Time offset to apply to the window. Format ( s , min , h , d , w ). | false | 10min | offset = "10min" | 0 | | config_file_path | Path to the TOML config file. Absolute, or relative to PLUGIN_DIR . | false | config.toml | — | None | | Argument | Description | Required | Example (CLI) | Example (TOML) | Default | | --- | --- | --- | --- | --- | --- | | Argument | Description | Required | Example (CLI) | Example (TOML) | Default | | host | Remote InfluxDB host URL. Default scheme http , port 8181 . | true | example.com | host = "example.com" | None | | remote_token | Remote InfluxDB API token. Falls back to REMOTE_INFLUXDB_TOKEN env var. | false | apiv3_AuHk... | remote_token = "apiv3_AuHk..." | None | | database | Remote database name. | true | remote_db | database = "remote_db" | None | | unique_file_suffix | Unique suffix for the queue file to avoid conflicts. | true | wxyz5678 | unique_file_suffix = "wxyz5678" | None | | tables | Tables to replicate; all tables if omitted. CLI: space-separated. TOML: list. | false | home home2 | tables = ["home", "home2"] | None | | verify_ssl | Whether to verify SSL certificates when connecting via HTTPS. | false | false | verify_ssl = false | true | | max_size | Maximum size for the queue file in MB. Integer ≥ 1. | false | 1024 | max_size = 1024 | 512 | | queue_flush_chunk_size | Line-protocol entries sent per remote flush request. Integer ≥ 1. | false | 1000 | queue_flush_chunk_size = 1000 | 500 | | excluded_fields | Fields/tags to exclude per table. CLI: table:f1|f2 blocks separated by space. TOML: table of lists. | false | home:co|location home2:temp | excluded_fields = {home = ["co", "location"], home2 = ["temp"]} | None | | tables_rename | Table renames. CLI: old:new pairs separated by space. TOML: table. | false | home:home2 office:office2 | tables_rename = {home = "home2", office = "office2"} | None | | field_renames | Field renames per table. CLI: table:old=new|old=new blocks separated by space. TOML: nested table. | false | home:temp=temperature|hum=humidity | [field_renames] home = {temp = "temperature", hum = "humidity"} | None | | config_file_path | Path to the TOML config file. Absolute, or relative to PLUGIN_DIR . | false | config.toml | — | None | --- # State change plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/state-change/ The State Change Plugin provides comprehensive field monitoring and threshold detection for InfluxDB 3 Enterprise data streams. Detect field value changes, monitor threshold conditions, and trigger notifications when specified criteria are met. Supports both scheduled batch monitoring and real-time data write monitoring with configurable stability checks and multi-channel alerts. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |--------------------|------|--------|----------------------------------------------------------------------------------------------------------------------| | `measurement` |string|required| Measurement to monitor for field changes | |`field_change_count`|string|required|Dot-separated field thresholds (for example, “temp:3.load:2” or “temp:3.disk.used:2”). Each count must be 1 or greater| | `senders` |string|required| Dot-separated notification channels with multi-channel alert support (Slack, Discord, etc.) | | `window` |string|required| Time window for analysis. Format: ``, units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`. Must be positive | ### Data write trigger parameters | Parameter | Type |Default | Description | |------------------|------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `measurement` |string|required| Measurement to monitor for threshold conditions | |`field_thresholds`|string|required|Threshold conditions with count-based and duration-based support (for example, “temp:30:10@status:ok:1h”). Counts must be 1 or greater; durations must be positive| | `senders` |string|required| Dot-separated notification channels with multi-channel alert support (Slack, Discord, HTTP, SMS, WhatsApp) | ### Notification parameters | Parameter | Type |Default | Description | |-------------------------|------|--------|-------------------------------------------------------------------------------------------| | `influxdb3_auth_token` |string|env var |InfluxDB 3 Enterprise API token with environment variable support for credential management| | `notification_text` |string|template| Customizable message template for scheduled notifications with dynamic variables | |`notification_count_text`|string|template| Customizable message template for count-based notifications with dynamic variables | |`notification_time_text` |string|template| Customizable message template for time-based notifications with dynamic variables | | `notification_path` |string|“notify”| Notification endpoint path | | `port_override` |number| 8181 | InfluxDB port override | ### Advanced parameters | Parameter | Type |Default| Description | |---------------------|------|-------|---------------------------------------------------------------------------------| |`state_change_window`|number| 1 | Recent values to check for stability (reduces noise from flapping fields) | |`state_change_count` |number| 1 |Changes within the stability window at which notifications start being suppressed| The stability check applies only when `state_change_window` is 2 or greater; the default of 1 leaves it off. Notifications are suppressed once the window contains `state_change_count` changes, so `state_change_count=3` is the setting that tolerates two flips. ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Relative paths are resolved against the first directory that is set: `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. Only that directory is used — the file is not looked up in the remaining ones. When `config_file_path` is set, the TOML file provides the whole configuration and inline trigger arguments are ignored. `INFLUXDB3_AUTH_TOKEN` from the environment still applies when `influxdb3_auth_token` is not set in the file. In TOML, `senders`, `field_thresholds`, and `field_change_count` use native structures (list, list of entries, table) instead of the inline string formats, though the inline strings are also accepted. Data write triggers cache the loaded configuration for 10 minutes to keep the write path fast, so configuration changes take effect within that window. Example TOML configuration files provided: * [state\_change\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/state_change/state_change_config_scheduler.toml) - for scheduled triggers * [state\_change\_config\_data\_writes.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/state_change/state_change_config_data_writes.toml) - for data write triggers For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ### Channel-specific configuration Notification channels require additional parameters based on the sender type (same as the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/)). ## Schema requirement The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Notification Sender Plugin for InfluxDB 3 Enterprise**: Required for sending notifications. See the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/). * **Python packages**: * `influxdata-plugin-utils>=0.3.0` (configuration loading, parsing, and schema introspection) * `requests` (for HTTP notifications) ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.3.0" influxdb3 install package requests ``` 3. *Optional*: For notifications, install and configure the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) ## Trigger setup ### Scheduled trigger Create a trigger for periodic field change monitoring: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/state_change/state_change_check_plugin.py" \ --trigger-spec "every:10m" \ --trigger-arguments "measurement=cpu,field_change_count=temp:3.load:2,window=10min,senders=slack,slack_webhook_url=$SLACK_WEBHOOK_URL" \ state_change_scheduler ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Data write trigger Create a trigger for real-time threshold monitoring: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/state_change/state_change_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=cpu,field_thresholds=temp:30:10@status:ok:1h,senders=slack,slack_webhook_url=$SLACK_WEBHOOK_URL" \ state_change_datawrite ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Enable triggers ```bash influxdb3 enable trigger --database mydb state_change_scheduler influxdb3 enable trigger --database mydb state_change_datawrite ``` ## Example usage ### Example 1: Scheduled field change monitoring Monitor field changes over a time window and alert when thresholds are exceeded: ```bash # Write test data with changing values (7 writes = 6 changes) influxdb3 write \ --database sensors \ "temperature,location=office value=22.5" influxdb3 write \ --database sensors \ "temperature,location=office value=25.0" influxdb3 write \ --database sensors \ "temperature,location=office value=22.8" influxdb3 write \ --database sensors \ "temperature,location=office value=26.5" influxdb3 write \ --database sensors \ "temperature,location=office value=23.0" influxdb3 write \ --database sensors \ "temperature,location=office value=27.2" influxdb3 write \ --database sensors \ "temperature,location=office value=24.0" # Create and enable the trigger influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/state_change/state_change_check_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments "measurement=temperature,field_change_count=value:5,window=1h,senders=slack,slack_webhook_url=$SLACK_WEBHOOK_URL" \ temp_change_monitor influxdb3 enable trigger --database sensors temp_change_monitor ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. **Expected output** When the field changes 5 or more times within 1 hour, a notification is sent: “Field value in table temperature changed 6 times in window 1:00:00 for tags location=office” ### Example 2: Advanced scheduled field change monitoring Monitor field changes over a time window and alert when thresholds are exceeded: ```bash influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/state_change/state_change_check_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments "measurement=temperature,field_change_count=value:5,window=1h,senders=slack,slack_webhook_url=$SLACK_WEBHOOK_URL,notification_text=Temperature sensor $field changed $changes times in $window for tags $tags" \ temp_change_monitor ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Real-time threshold detection Monitor data writes for threshold conditions: ```bash influxdb3 create trigger \ --database monitoring \ --path "gh:influxdata/state_change/state_change_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=system_metrics,field_thresholds=cpu_usage:80:5@memory_usage:90:10min,senders=discord,discord_webhook_url=$DISCORD_WEBHOOK_URL" \ system_threshold_monitor ``` Set `DISCORD_WEBHOOK_URL` to your Discord incoming webhook URL. ### Multi-condition monitoring Monitor multiple fields with different threshold types: ```bash influxdb3 create trigger \ --database application \ --path "gh:influxdata/state_change/state_change_check_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=app_health,field_thresholds=error_rate:0.05:3@response_time:500:30s@status:down:1,senders=slack.sms,slack_webhook_url=$SLACK_WEBHOOK_URL,twilio_from_number=+1234567890,twilio_to_number=+0987654321" \ app_health_monitor ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ## Code overview ### Files * `state_change_check_plugin.py`: The main plugin code containing handlers for scheduled and data write triggers * `state_change_config_scheduler.toml`: Example TOML configuration for scheduled triggers * `state_change_config_data_writes.toml`: Example TOML configuration for data write triggers * `test_state_change.py`: Pytest suite, runs without a live InfluxDB 3 Enterprise server * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`, `requests`) * `requirements-dev.txt`: Development dependencies (`pytest`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'state_change_scheduler'" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled field change monitoring. Queries data within the specified window and counts field value changes. #### `process_writes(influxdb3_local, table_batches, args)` Handles real-time threshold monitoring on data writes. Evaluates incoming data against configured thresholds. ## Troubleshooting ### Common issues #### Issue: No notifications triggered **Solution**: Verify notification channel configuration (webhook URLs, credentials). Check threshold values are appropriate for your data. Ensure the Notifier Plugin is installed and configured. Review plugin logs for error messages. #### Issue: Too many notifications **Solution**: Adjust `state_change_window` and `state_change_count` for stability filtering. Increase threshold values to reduce sensitivity. Consider longer monitoring windows for scheduled triggers. #### Issue: Authentication errors **Solution**: Set `INFLUXDB3_AUTH_TOKEN` environment variable. Verify token has appropriate database permissions. Check Twilio credentials for SMS/WhatsApp notifications. ### Field threshold formats **Count-based thresholds** * Format: `field_name:"value":count` * Example: `temp:"30.5":10` (10 consecutive occurrences of temperature = 30.5) * The count must be an integer of 1 or greater **Time-based thresholds** * Format: `field_name:"value":duration` * Example: `status:"error":5min` (status = error for 5 minutes) * Supported units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`; the duration must be positive **Multiple conditions** * Separate with `@`: `temp:"30":5@humidity:"high":10min` * Segments that fail to parse are skipped with a warning; if none remain, the run stops with an error In TOML, the same thresholds are written as entries: `field_thresholds = [["temp", 30.5, 10], ["status", "error", "5min"]]`. ### Message template variables **Scheduled notifications** * `$table`: Measurement name * `$field`: Field name * `$changes`: Number of changes detected * `$window`: Time window * `$tags`: Tag values **Data write notifications** * `$table`: Measurement name * `$field`: Field name * `$value`: Threshold value * `$duration`: Time duration or count * `$row`: Unique row identifier ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [State change plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/state_change) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[state-tracking](/influxdb3/enterprise/tags/state-tracking/)[event-detection](/influxdb3/enterprise/tags/event-detection/)[monitoring](/influxdb3/enterprise/tags/monitoring/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to monitor for field changes | | field_change_count | string | required | Dot-separated field thresholds (for example, “temp:3.load:2” or “temp:3.disk.used:2”). Each count must be 1 or greater | | senders | string | required | Dot-separated notification channels with multi-channel alert support (Slack, Discord, etc.) | | window | string | required | Time window for analysis. Format: , units: us , ms , s , min , h , d , w . Must be positive | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to monitor for threshold conditions | | field_thresholds | string | required | Threshold conditions with count-based and duration-based support (for example, “temp:30:10@status:ok:1h”). Counts must be 1 or greater; durations must be positive | | senders | string | required | Dot-separated notification channels with multi-channel alert support (Slack, Discord, HTTP, SMS, WhatsApp) | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | influxdb3_auth_token | string | env var | InfluxDB 3 Enterprise API token with environment variable support for credential management | | notification_text | string | template | Customizable message template for scheduled notifications with dynamic variables | | notification_count_text | string | template | Customizable message template for count-based notifications with dynamic variables | | notification_time_text | string | template | Customizable message template for time-based notifications with dynamic variables | | notification_path | string | “notify” | Notification endpoint path | | port_override | number | 8181 | InfluxDB port override | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | state_change_window | number | 1 | Recent values to check for stability (reduces noise from flapping fields) | | state_change_count | number | 1 | Changes within the stability window at which notifications start being suppressed | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | --- # Stateless ADTK detector plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/stateless-adtk-detector/ The ADTK Anomaly Detector Plugin provides advanced time series anomaly detection for InfluxDB 3 Enterprise using the ADTK (Anomaly Detection Toolkit) library. Apply statistical and machine learning-based detection methods to identify outliers, level shifts, volatility changes, and seasonal anomalies in your data. Features consensus-based detection requiring multiple detectors to agree before triggering alerts, reducing false positives. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |-----------------|------|--------|---------------------------------------------------------------------------------------------| | `measurement` |string|required| Measurement to analyze for anomalies | | `field` |string|required| Numeric field to evaluate | | `detectors` |string|required| Dot-separated list of advanced ADTK detectors for different anomaly types | |`detector_params`|string|required| Base64-encoded JSON parameters for each detector | | `window` |string|required|Data analysis window. Format: `` (for example, “1h”, “30min”). Must be positive| | `senders` |string|required| Dot-separated notification channels with multi-channel notification support | Duration units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`. ### Advanced parameters | Parameter | Type |Default| Description | |---------------------------|------|-------|----------------------------------------------------------------------------------------------------------| | `min_consensus` |number| 1 |Minimum detectors required to agree for consensus-based filtering to reduce false positives (1 or greater)| | `min_condition_duration` |string| “0s” | Minimum duration for configurable anomaly persistence before alerting | | `group_by_tags` | bool | false | Analyze every tag combination as its own time series | |`max_notifications_per_run`|number| 20 | Maximum number of notifications a single run may send | #### Analyzing tagged measurements By default the whole window forms a single time series. When a measurement holds several tag combinations (for example `host=server1` and `host=server2`), those rows share timestamps, and ADTK keeps only the first value of each timestamp — so one arbitrary series is analyzed and the rest of the data is ignored. Set `group_by_tags=true` to analyze each tag combination separately. Every series then gets its own detector run, its own consensus evaluation, and its own debounce state, so a measurement with N tag combinations can produce up to N notifications per run. #### Notification behavior The timestamp of the last alerted point is remembered per series, so a `window` longer than the trigger interval does not resend anomalies that earlier runs already reported. A single run sends at most `max_notifications_per_run` notifications. Anomalies beyond the limit are counted in a warning and are not resent by later runs — raise the limit if a run legitimately produces more alerts. Points that have no value for `field` are dropped before detection and reported in the log; without this a single NULL makes every detector fail. ### Notification parameters | Parameter | Type |Default | Description | |----------------------|------|--------|-----------------------------------------------------------------| |`influxdb3_auth_token`|string|env var | InfluxDB 3 Enterprise API token | | `notification_text` |string|template|Customizable notification template message with dynamic variables| | `notification_path` |string|“notify”| Notification endpoint path | | `port_override` |number| 8181 | InfluxDB port override | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. When a config file is given, it replaces the inline trigger arguments entirely. In TOML, `detectors` and `senders` accept either a list (`["QuantileAD", "PersistAD"]`) or a dot-separated string, and detector parameters may be given as a `[detector_params]` table or as a base64-encoded JSON string. #### Example TOML configuration [adtk\_anomaly\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/stateless_adtk_detector/adtk_anomaly_config_scheduler.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ### Supported ADTK detectors | Detector | Description | Required Parameters | |----------------------|-------------------------------------|------------------------| |`GeneralizedESDTestAD`| Extreme Studentized Deviate test | `alpha` (optional) | |`InterQuartileRangeAD`| Detects outliers using IQR method | None | | `ThresholdAD` |Detects values above/below thresholds|`high`, `low` (optional)| | `QuantileAD` | Detects outliers based on quantiles |`low`, `high` (optional)| | `LevelShiftAD` | Detects sudden level changes | `window` (int) | | `VolatilityShiftAD` | Detects volatility changes | `window` (int) | | `PersistAD` | Detects persistent anomalous values | None | | `SeasonalAD` | Detects seasonal pattern deviations | None | ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Python 3.11+** * **Python packages**: * `influxdata-plugin-utils>=0.3.0` (for configuration loading, parsing, and schema introspection) * `adtk` (for anomaly detection) * `pandas<3` (for data manipulation) * `requests` (for HTTP notifications) `pandas` must stay below 3.0. Window-based detectors (`LevelShiftAD`, `VolatilityShiftAD`, `PersistAD`) return `NaN` for the first `window` points, which pandas 3 refuses to store in a boolean result. With pandas 3 installed those three detectors fail with `Invalid value 'nan' for dtype 'bool'`, are skipped with a warning, and stop contributing to the consensus. * **Notification Sender Plugin** *(optional)*: Required if using the `senders` parameter. See the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/). ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.3.0" influxdb3 install package requests influxdb3 install package adtk influxdb3 install package "pandas<3" ``` 3. *(Optional)* For notifications, install the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) and create an HTTP trigger for it. ## Trigger setup ### Scheduled trigger Create a scheduled trigger for anomaly detection: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/stateless_adtk_detector/adtk_anomaly_detection_plugin.py" \ --trigger-spec "every:10m" \ --trigger-arguments "measurement=cpu,field=usage,detectors=QuantileAD.LevelShiftAD,detector_params=eyJRdWFudGlsZUFKIjogeyJsb3ciOiAwLjA1LCAiaGlnaCI6IDAuOTV9LCAiTGV2ZWxTaGlmdEFKIjogeyJ3aW5kb3ciOiA1fX0=,window=10min,senders=slack,slack_webhook_url=$SLACK_WEBHOOK_URL" \ anomaly_detector ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Enable trigger ```bash influxdb3 enable trigger --database mydb anomaly_detector ``` ## Example usage ### Example 1: Quantile-based detection Detect outliers using quantile-based detection. This plugin analyzes existing time series data and sends notifications when anomalies are detected. ```bash # Base64 encode detector parameters: {"QuantileAD": {"low": 0.05, "high": 0.95}} echo '{"QuantileAD": {"low": 0.05, "high": 0.95}}' | base64 influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/stateless_adtk_detector/adtk_anomaly_detection_plugin.py" \ --trigger-spec "every:5m" \ --trigger-arguments "measurement=temperature,field=value,detectors=QuantileAD,detector_params=eyJRdWFudGlsZUFKIjogeyJsb3ciOiAwLjA1LCAiaGlnaCI6IDAuOTV9fQ==,window=1h,senders=slack,slack_webhook_url=$SLACK_WEBHOOK_URL" \ temp_anomaly_detector ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Example 2: Multi-detector consensus Use multiple detectors with consensus requirement: ```bash # Base64 encode: {"QuantileAD": {"low": 0.1, "high": 0.9}, "LevelShiftAD": {"window": 10}} echo '{"QuantileAD": {"low": 0.1, "high": 0.9}, "LevelShiftAD": {"window": 10}}' | base64 influxdb3 create trigger \ --database monitoring \ --path "gh:influxdata/stateless_adtk_detector/adtk_anomaly_detection_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments "measurement=cpu_metrics,field=utilization,detectors=QuantileAD.LevelShiftAD,detector_params=eyJRdWFudGlsZUFEIjogeyJsb3ciOiAwLjEsICJoaWdoIjogMC45fSwgIkxldmVsU2hpZnRBRCI6IHsid2luZG93IjogMTB9fQ==,min_consensus=2,window=30min,senders=discord,discord_webhook_url=$DISCORD_WEBHOOK_URL" \ cpu_consensus_detector ``` Set `DISCORD_WEBHOOK_URL` to your Discord incoming webhook URL. ### Volatility shift detection Monitor for sudden changes in data volatility: ```bash # Base64 encode: {"VolatilityShiftAD": {"window": 20}} echo '{"VolatilityShiftAD": {"window": 20}}' | base64 influxdb3 create trigger \ --database trading \ --path "gh:influxdata/stateless_adtk_detector/adtk_anomaly_detection_plugin.py" \ --trigger-spec "every:1m" \ --trigger-arguments "measurement=stock_prices,field=price,detectors=VolatilityShiftAD,detector_params=eyJWb2xhdGlsaXR5U2hpZnRBRCI6IHsid2luZG93IjogMjB9fQ==,window=1h,min_condition_duration=5min,senders=sms,twilio_from_number=+1234567890,twilio_to_number=+0987654321" \ volatility_detector ``` ## Code overview ### Files * `adtk_anomaly_detection_plugin.py`: The main plugin code containing the scheduled handler for anomaly detection * `adtk_anomaly_config_scheduler.toml`: Example TOML configuration file * `test_adtk_anomaly_detection.py`: Pytest suite (49 tests, runs without a live InfluxDB 3 Enterprise server) * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`, `requests`, `adtk`, `pandas<3`) * `requirements-dev.txt`: Development dependencies (`pytest`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'anomaly_detector'" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled anomaly detection tasks. Queries data within the specified window, applies ADTK detectors, and sends notifications for detected anomalies. Key operations: 1. Parses configuration and decodes detector parameters 2. Queries data from source measurement 3. Applies configured ADTK detectors 4. Evaluates consensus across detectors 5. Sends notifications when anomalies are confirmed #### `parse_detectors(influxdb3_local, config, task_id)` Resolves the detectors to apply together with their parameters. Detectors that are unknown, have no entry in `detector_params`, or miss a parameter required to construct them are skipped with a warning and do not count toward `min_consensus`. #### `split_by_tags(df, tags, group_by_tags)` Splits query results into one frame per tag combination when `group_by_tags` is enabled, so detectors never mix values written under different tag sets. ## Troubleshooting ### Common issues #### Issue: Detector parameter encoding errors **Solution**: Ensure detector\_params is valid Base64-encoded JSON. Use command line Base64 encoding: `echo '{"QuantileAD": {"low": 0.05}}' | base64`. Verify JSON structure matches detector requirements. #### Issue: False positive notifications **Solution**: Increase `min_consensus` to require more detectors to agree. Add `min_condition_duration` to require anomalies to persist. Adjust detector-specific thresholds in `detector_params`. #### Issue: A newly created measurement is reported as not found **Solution**: Table and tag names are cached for one hour per trigger. Wait for the cache to expire, or recreate the trigger to clear it. #### Issue: Anomalies of some tag combinations are never detected **Solution**: Set `group_by_tags=true`. Without it, rows of different tag combinations share timestamps and only the first series survives. #### Issue: A detector is skipped with a warning **Solution**: The warning names the reason: the detector is not in the supported list (check the spelling), has no entry in `detector_params`, or misses a required parameter (`window` for `LevelShiftAD` and `VolatilityShiftAD`). `Invalid value 'nan' for dtype 'bool'` means pandas 3 is installed — downgrade to `pandas<3`. #### Issue: No anomalies are ever reported **Solution**: Check the warnings. `min_consensus` must not exceed the number of detectors that were actually applied — skipped detectors reduce that count. `min_condition_duration` must be shorter than `window`, otherwise no anomaly can persist long enough within a single query window. #### Issue: Missing dependencies **Solution**: Install required packages: `influxdata-plugin-utils`, `adtk`, `pandas`, `requests`. Ensure the Notifier Plugin is installed for notifications. #### Issue: Data quality issues **Solution**: Verify sufficient data points in the specified window. Check for null values or data gaps that affect detection. Ensure field contains numeric data suitable for analysis. ### Base64 parameter encoding Generate properly encoded detector parameters: ```bash # Single detector echo '{"QuantileAD": {"low": 0.05, "high": 0.95}}' | base64 -w 0 # Multiple detectors echo '{"QuantileAD": {"low": 0.1, "high": 0.9}, "LevelShiftAD": {"window": 15}}' | base64 -w 0 # Threshold detector echo '{"ThresholdAD": {"high": 100, "low": 10}}' | base64 -w 0 ``` ### Message template variables Available variables for notification templates: * `$table`: Measurement name * `$field`: Field name with anomaly * `$value`: Anomalous value * `$detectors`: List of detecting methods * `$tags`: Tag values * `$timestamp`: Anomaly timestamp ### Detector configuration reference For detailed detector parameters and options, see the [ADTK documentation](https://adtk.readthedocs.io/en/stable/api/detectors.html). ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Stateless ADTK detector plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/stateless_adtk_detector) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[anomaly-detection](/influxdb3/enterprise/tags/anomaly-detection/)[adtk](/influxdb3/enterprise/tags/adtk/)[stateless](/influxdb3/enterprise/tags/stateless/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to analyze for anomalies | | field | string | required | Numeric field to evaluate | | detectors | string | required | Dot-separated list of advanced ADTK detectors for different anomaly types | | detector_params | string | required | Base64-encoded JSON parameters for each detector | | window | string | required | Data analysis window. Format: (for example, “1h”, “30min”). Must be positive | | senders | string | required | Dot-separated notification channels with multi-channel notification support | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | min_consensus | number | 1 | Minimum detectors required to agree for consensus-based filtering to reduce false positives (1 or greater) | | min_condition_duration | string | “0s” | Minimum duration for configurable anomaly persistence before alerting | | group_by_tags | bool | false | Analyze every tag combination as its own time series | | max_notifications_per_run | number | 20 | Maximum number of notifications a single run may send | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | influxdb3_auth_token | string | env var | InfluxDB 3 Enterprise API token | | notification_text | string | template | Customizable notification template message with dynamic variables | | notification_path | string | “notify” | Notification endpoint path | | port_override | number | 8181 | InfluxDB port override | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | | Detector | Description | Required Parameters | | --- | --- | --- | | Detector | Description | Required Parameters | | GeneralizedESDTestAD | Extreme Studentized Deviate test | alpha (optional) | | InterQuartileRangeAD | Detects outliers using IQR method | None | | ThresholdAD | Detects values above/below thresholds | high , low (optional) | | QuantileAD | Detects outliers based on quantiles | low , high (optional) | | LevelShiftAD | Detects sudden level changes | window (int) | | VolatilityShiftAD | Detects volatility changes | window (int) | | PersistAD | Detects persistent anomalous values | None | | SeasonalAD | Detects seasonal pattern deviations | None | --- # Stock plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/stock-plugin/ The Stock Portfolio Tracker plugin periodically fetches current stock, ETF, and mutual fund prices from Yahoo Finance and writes per-holding rows plus per-portfolio and per-category roll-up totals to InfluxDB 3 Enterprise. Configure your holdings inline as trigger arguments or in a TOML file, group them into named portfolios (e.g. `401k`, `brokerage`) and user-defined categories (e.g. `Retirement`, `Investment`), and the plugin will track the value of each holding and each grouping over time. Built-in gating skips fetches during market closures (with a clear log line so dashboard gaps have a known cause), polls mutual funds only once per day after NAV close, and carries forward the last known price for skipped symbols so portfolio totals stay accurate. ### Scope Yahoo Finance supports tickers from many global exchanges (e.g. `.L` for London, `.T` for Tokyo, `.PA` for Paris), so the price-fetch path works for international holdings out of the box. The **market-hours gating** is also internationalizable: `market_calendar` and `market_timezone` configure which exchange’s calendar to use. Defaults are US-centric (`NYSE` / `America/New_York`) because that’s the most common case, but any exchange name accepted by [pandas\_market\_calendars](https://pandas-market-calendars.readthedocs.io/) works (LSE, TSX, JPX, XETR, ASX, HKEX, etc.). Currency conversion is **not** performed — all values are stored in whatever currency yfinance returns per symbol; this is recorded on the `currency` field of `stock_holdings` rows. If your portfolio mixes currencies, do conversion at query time. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. The plugin also supports a TOML configuration file for the full portfolio shape; the trigger-argument form is best for one-off testing. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines the supported trigger type and configuration parameters. This metadata enables the InfluxDB 3 Explorer UI to display and configure the plugin. ### Optional parameters | Parameter | Type | Default | Description | |---------------------------|-------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `database` |string | `stocks` | Target database for writes. | | `portfolio` |string |`AAPL:1|MSFT:1|GOOG:1`|Inline holdings: pipe-separated `SYMBOL:QUANTITY[:PORTFOLIO_NAME]` entries (e.g. `AAPL:10:401k|MSFT:5:401k|GOOG:2.5:brokerage`). Portfolio defaults to `main`. When omitted and no TOML config is found, falls back to the default shown.| | `categories` |string | none | Inline category map: pipe-separated `PORTFOLIO:CATEGORY` entries (e.g. `401k:Retirement|brokerage:Investment`). | | `config_path` |string | `stock_plugin.toml` | Path to the TOML config file, relative to the InfluxDB plugin directory (or absolute). The default file is loaded when it exists; an explicit `config_path` that does not exist is an error. | |`write_during_closed_hours`|boolean| `true` | See the TOML table below. Also settable as a trigger argument. | | `mutual_fund_check_time` |string | `18:00` | See the TOML table below. Also settable as a trigger argument. | | `market_calendar` |string | `NYSE` | See the TOML table below. Also settable as a trigger argument. | | `market_timezone` |string | `America/New_York` | See the TOML table below. Also settable as a trigger argument. | *If neither `portfolio` nor a TOML file with `[holdings.]` is provided, the plugin runs with the default holdings `AAPL:1|MSFT:1|GOOG:1` in the `main` portfolio.* ### TOML configuration The TOML file is the recommended way to configure anything more than a handful of holdings. The plugin reads it from `config_path` (default: `/stock_plugin.toml`). Trigger arguments take precedence over TOML keys of the same name, so a TOML file can hold the full portfolio shape while a trigger argument overrides a single setting. Holdings and categories are the exception, because each is spelled differently per source: | Setting | Trigger argument | TOML | |----------|------------------------------------|------------------------------| | Holdings |`portfolio=AAPL:10:401k|MSFT:5:401k`| `[holdings.401k]` tables | |Categories| `categories=401k:Retirement` |`[portfolio_categories]` table| Each spelling is read only from its own source: a top-level `portfolio` or `categories` key in the TOML file is ignored, as is a trigger argument named `holdings` or `portfolio_categories`. When both sources are present, the trigger argument replaces the TOML tables entirely rather than merging with them. | Key | Type | Default | Description | |---------------------------|-------|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `database` |string | `stocks` | Target database for writes. | |`write_during_closed_hours`|boolean| `true` | When `false`, stocks/ETFs are skipped outside the configured exchange’s regular session (the calendar handles holidays and early closes). Mutual funds always follow their own daily check schedule. | | `mutual_fund_check_time` |string | `"18:00"` | Time of day in `market_timezone` after which the plugin fetches mutual fund NAV. Mutual funds are fetched at most once per local calendar day, at the first tick at or after this time. Bootstrap exception: a mutual fund with no cached asset type is fetched on its first tick regardless of time. | | `market_calendar` |string | `"NYSE"` | Exchange calendar used for the market-hours check. Any name accepted by [pandas\_market\_calendars](https://pandas-market-calendars.readthedocs.io/) (e.g. `NYSE`, `LSE`, `TSX`, `JPX`, `XETR`, `ASX`, `HKEX`). | | `market_timezone` |string |`"America/New_York"`| IANA timezone for the exchange’s local time. Used for `mutual_fund_check_time` comparisons and for resolving the “today” date the calendar consults. | | `[portfolio_categories]` | table | empty | Maps portfolio name to category name. Portfolios not listed are uncategorized (omitted from `category_totals`). | | `[holdings.]` | table | default holdings |Holdings for each portfolio. Each entry is `SYMBOL = quantity`. Fractional quantities supported; the quantity must be a finite number (`inf` and `nan` are rejected). Quote symbols containing dots, for example `"VOD.L" = 10`. Duplicate same-symbol entries in one portfolio are aggregated. The portfolio name `_total` is reserved. When no `[holdings.*]` section is present, the plugin falls back to `AAPL:1|MSFT:1|GOOG:1`.| The trigger spec is the source of truth for cadence. For example, `--trigger-spec "every:15m"` runs the plugin every 15 minutes. #### Example TOML configuration [stock\_plugin.toml.example](stock_plugin.toml.example) ## Software requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Python 3.11 or higher** * **Python packages** (installed into the plugin venv): * `yfinance` — Yahoo Finance scraper for price data * `pandas_market_calendars` — exchange calendars for accurate market-hours and holiday gating * `influxdata-plugin-utils>=0.3.0` — shared configuration, parsing, and write helpers ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package yfinance pandas_market_calendars influxdata-plugin-utils ``` 3. Copy `stock_plugin.toml.example` to `/stock_plugin.toml` and edit it with your holdings and categories. ## Trigger setup ### Basic scheduled trigger (TOML config) ```bash influxdb3 create trigger \ --database stocks \ --path "stock_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments config_path=stock_plugin.toml \ --error-behavior log \ stock_portfolio_tracker ``` ### Inline holdings without TOML ```bash influxdb3 create trigger \ --database stocks \ --path "stock_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments 'portfolio=AAPL:10:401k|MSFT:5:401k|GOOG:2:brokerage,categories=401k:Retirement|brokerage:Investment' \ --error-behavior log \ stock_portfolio_tracker ``` ### Skip writes when the configured market is closed Set `write_during_closed_hours = false` in your TOML config and create the trigger as normal. With the default NYSE calendar, stocks and ETFs will skip outside 9:30–16:00 ET on weekdays and on NYSE holidays. Their portfolio total values are still maintained via carry-forward. On a cold cache, such as after an InfluxDB restart, a symbol that would otherwise be skipped can be fetched once so the plugin has a last-known price to carry forward on later skipped ticks. The summary log calls these out as cold-cache bootstrap fetches. ## Example usage ### Latest portfolio value by category ```bash influxdb3 query \ --database stocks \ "SELECT category, value, portfolio_count, symbol_count FROM category_totals WHERE time = (SELECT MAX(time) FROM category_totals)" ``` **Expected output** ``` +------------+-----------+-----------------+--------------+ | category | value | portfolio_count | symbol_count | +------------+-----------+-----------------+--------------+ | Investment | 257908.71 | 2 | 17 | | Retirement | 1990604.5 | 3 | 5 | +------------+-----------+-----------------+--------------+ ``` ### Total portfolio value over time ```bash influxdb3 query \ --database stocks \ "SELECT time, value FROM portfolio_totals WHERE portfolio = '_total' ORDER BY time DESC LIMIT 10" ``` ### Daily P&L per holding (using previous\_close) ```bash influxdb3 query \ --database stocks \ "SELECT symbol, portfolio, (price - previous_close) * quantity AS daily_change FROM stock_holdings WHERE time = (SELECT MAX(time) FROM stock_holdings) AND previous_close IS NOT NULL ORDER BY daily_change DESC" ``` ### Filter for fully fresh rows Rows in `portfolio_totals` and `category_totals` carry `missing_symbols`, `skipped_symbols`, and `carried_symbols` counts so dashboards can distinguish fully fresh data from carry-forward or partial fetches: ```sql -- "All symbols fetched fresh this tick" SELECT * FROM portfolio_totals WHERE missing_symbols = 0 AND skipped_symbols = 0; -- "All values present (fresh + carry-forward), no failures" SELECT * FROM portfolio_totals WHERE missing_symbols = 0; ``` ## Code overview ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Entry point for the scheduled trigger. Stamps one UTC timestamp for the whole run and delegates to `_main` with the runtime-injected `LineBuilder` and the live `influxdb3_local`. All side-effecting work lives in `_main` so its logic can be reasoned about with injected dependencies. #### `_main(local, args, fetcher, line_builder_cls, now_ns, task_id)` Drives the full plugin flow: 1. Resolve config from trigger args + TOML. 2. Determine configured market state via `pandas_market_calendars` and parse the mutual-fund check time. 3. For each configured holding, decide whether to fetch (gated by asset type, cached state, and config), fetch via `yfinance.fast_info`, and build a `HoldingRow`. 4. Build carry-forward `HoldingRow`s for intentionally-skipped symbols whose last known price is cached. 5. Aggregate per-portfolio totals + a grand `_total` row. 6. Aggregate per-category totals across portfolios. 7. Write `stock_holdings`, `portfolio_totals`, and `category_totals` as a single batched payload. 8. Log a single summary line. #### `resolve_config(args)` Merges the TOML file with the trigger arguments and validates the result. The TOML path comes from `config_path`; relative paths resolve against the plugin directory (`PLUGIN_DIR`, `INFLUXDB3_PLUGIN_DIR`, or the `VIRTUAL_ENV` parent). Returns a `ResolvedConfig`, raising `ValueError` on any invalid value. ### Measurements and fields #### `stock_holdings` One row per symbol per successful fetch. Carry-forward symbols are NOT written here (their last fresh row remains the most recent record in this measurement). * **Tags**: `symbol`, `portfolio`, `asset_type` (`equity`, `etf`, `mutualfund`, `other`), `category` (omitted if portfolio is uncategorized) * **Fields**: `price`, `quantity`, `value`, `currency`, `previous_close`, `day_open`, `day_high`, `day_low` #### `portfolio_totals` One row per configured portfolio plus a `_total` grand-total row, written every tick. * **Tags**: `portfolio`, `category` (omitted if uncategorized; always omitted on `_total`) * **Fields**: `value`, `symbol_count`, `missing_symbols`, `skipped_symbols`, `carried_symbols` #### `category_totals` One row per defined category, rolled up across portfolios in that category. Uncategorized portfolios and the `_total` row are excluded to avoid double-counting. * **Tags**: `category` * **Fields**: `value`, `symbol_count`, `portfolio_count`, `missing_symbols`, `skipped_symbols`, `carried_symbols` ### Internal cache keys The plugin uses `influxdb3_local.cache` (no TTL) for state across runs: | Key | Value | Purpose | |-----------------------|-----------------|--------------------------------------------------------------------------------------| | `asset_type:` | string | Cached `yfinance.fast_info.quote_type` so we don’t re-derive it each run. | |`last_mf_date:`|`YYYY-MM-DD` (ET)|Last calendar day a mutual fund was fetched. Used to enforce once-per-day NAV polling.| | `last_price:` | float | Last known price. Used to carry forward portfolio value when a symbol is skipped. | Cache is cleared on server restart. The plugin self-bootstraps: any symbol with a missing cache key gets a fresh fetch on its next tick regardless of skip rules. ## Troubleshooting ### Common issues #### Issue: No holdings are configured **Solution:** With no `portfolio` argument and no TOML file, the plugin uses the default holdings `AAPL:1|MSFT:1|GOOG:1`. To track your own holdings, provide `portfolio` trigger arguments or a TOML file with at least one `[holdings.]` table. If you use a TOML file, confirm that `config_path` points to the file in the InfluxDB 3 Enterprise plugin directory. #### Issue: Market-hours checks skip expected writes **Solution:** Set `write_during_closed_hours = true` or choose the correct `market_calendar` and `market_timezone` for your exchange. When `write_during_closed_hours` is false, the plugin uses `pandas_market_calendars` to skip equity and ETF fetches outside the configured exchange session. Mutual funds are still gated by `mutual_fund_check_time`. #### Issue: Yahoo Finance returns missing prices **Solution:** Verify the ticker symbol and check whether Yahoo Finance exposes current price data for that instrument. A symbol whose price is missing or not a finite number is counted as a fetch failure and reported in the summary log; the optional `previous_close`, `day_open`, `day_high`, and `day_low` fields are simply omitted when unusable. The plugin carries forward the last known price for intentionally skipped symbols, but it cannot value a new holding until the first successful fetch. ### Debugging tips Check `system.processing_engine_logs` for the trigger summary line. It reports fetched, skipped, carried, and missing symbol counts for each scheduled run. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Stock plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/stock_plugin) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | database | string | stocks | Target database for writes. | | portfolio | string | AAPL:1|MSFT:1|GOOG:1 | Inline holdings: pipe-separated SYMBOL:QUANTITY[:PORTFOLIO_NAME] entries (e.g. AAPL:10:401k|MSFT:5:401k|GOOG:2.5:brokerage ). Portfolio defaults to main . When omitted and no TOML config is found, falls back to the default shown. | | categories | string | none | Inline category map: pipe-separated PORTFOLIO:CATEGORY entries (e.g. 401k:Retirement|brokerage:Investment ). | | config_path | string | stock_plugin.toml | Path to the TOML config file, relative to the InfluxDB plugin directory (or absolute). The default file is loaded when it exists; an explicit config_path that does not exist is an error. | | write_during_closed_hours | boolean | true | See the TOML table below. Also settable as a trigger argument. | | mutual_fund_check_time | string | 18:00 | See the TOML table below. Also settable as a trigger argument. | | market_calendar | string | NYSE | See the TOML table below. Also settable as a trigger argument. | | market_timezone | string | America/New_York | See the TOML table below. Also settable as a trigger argument. | | Setting | Trigger argument | TOML | | --- | --- | --- | | Setting | Trigger argument | TOML | | Holdings | portfolio=AAPL:10:401k|MSFT:5:401k | [holdings.401k] tables | | Categories | categories=401k:Retirement | [portfolio_categories] table | | Key | Type | Default | Description | | --- | --- | --- | --- | | Key | Type | Default | Description | | database | string | stocks | Target database for writes. | | write_during_closed_hours | boolean | true | When false , stocks/ETFs are skipped outside the configured exchange’s regular session (the calendar handles holidays and early closes). Mutual funds always follow their own daily check schedule. | | mutual_fund_check_time | string | "18:00" | Time of day in market_timezone after which the plugin fetches mutual fund NAV. Mutual funds are fetched at most once per local calendar day, at the first tick at or after this time. Bootstrap exception: a mutual fund with no cached asset type is fetched on its first tick regardless of time. | | market_calendar | string | "NYSE" | Exchange calendar used for the market-hours check. Any name accepted by pandas_market_calendars (e.g. NYSE , LSE , TSX , JPX , XETR , ASX , HKEX ). | | market_timezone | string | "America/New_York" | IANA timezone for the exchange’s local time. Used for mutual_fund_check_time comparisons and for resolving the “today” date the calendar consults. | | [portfolio_categories] | table | empty | Maps portfolio name to category name. Portfolios not listed are uncategorized (omitted from category_totals ). | | [holdings.] | table | default holdings | Holdings for each portfolio. Each entry is SYMBOL = quantity . Fractional quantities supported; the quantity must be a finite number ( inf and nan are rejected). Quote symbols containing dots, for example "VOD.L" = 10 . Duplicate same-symbol entries in one portfolio are aggregated. The portfolio name _total is reserved. When no [holdings.*] section is present, the plugin falls back to AAPL:1|MSFT:1|GOOG:1 . | | Key | Value | Purpose | | --- | --- | --- | | Key | Value | Purpose | | asset_type: | string | Cached yfinance.fast_info.quote_type so we don’t re-derive it each run. | | last_mf_date: | YYYY-MM-DD (ET) | Last calendar day a mutual fund was fetched. Used to enforce once-per-day NAV polling. | | last_price: | float | Last known price. Used to carry forward portfolio value when a symbol is skipped. | --- # Synthefy forecasting plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/synthefy-forecasting/ The Synthefy Forecasting Plugin integrates the Synthefy Forecasting API with InfluxDB 3 Enterprise to enable on-demand time series forecasting via HTTP requests. It reads time series data from InfluxDB, generates forecasts using Synthefy’s foundation models, and writes the results back to InfluxDB for visualization and alerting. **Key Features:** * **On-Demand Forecasting**: Generate forecasts on-demand via HTTP requests * **Multiple Models**: Support for various Synthefy models * **Metadata Support**: Use additional fields as covariates for improved accuracy * **Tag Filtering**: Filter input data by tags (for example, location, device); supports multiple values per tag (`IN (...)`) * **Line Protocol Writes**: Reliable, batched writing via `write_sync` / `write_sync_to_db` ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger, and/or in the JSON body of each HTTP request. Body values override trigger arguments. A `null` in the body counts as unset, so the trigger argument applies; send `{}` for `tags` or `[]` for `metadata_fields` to clear them. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Authentication for the Synthefy API The Synthefy API key is **never** read from trigger arguments or the request body. It must be provided via either of: * HTTP request header: `X-Synthefy-Api-Key: ` * Environment variable: `SYNTHEFY_API_KEY` If both are set, the header takes precedence. ### Request body parameters | Parameter | Type | Default | Description | |---------------------|-------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `measurement` | string | required | Source measurement (table) containing historical data | | `field` | string | `"value"` | Field name to forecast | | `tags` |string | dict| `""` |Tag filters. Trigger args: dot-separated string `key:val1@val2.key2:val3`. Request body: JSON object mapping tag name to a string or list of strings. See [Tag filter format](#tag-filter-format).| | `time_range` | string | `"30d"` | Historical window. Format: ``. Units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`, `m`, `q`, `y` (`m`/`q`/`y` are approximate). | | `forecast_horizon` | string | `"7d"` | Forecast duration. Format: `` (same units as `time_range`) or ` points`. | | `model` | string | `"sfm-tabular"` | Synthefy model identifier (for example, `sfm-tabular`, `Migas-latest`) | |`output_measurement` | string |`"{measurement}_forecast"`| Destination measurement for forecast results | | `metadata_fields` |string | list| `""` | Trigger args: space-separated list of field names (`"humidity pressure"`). Request body: JSON list of strings. Used as covariates. | |`max_forecast_points`| integer | `10000` |Upper bound on the forecast points one request may produce. A time-based `forecast_horizon` is divided by the series’ own step, so a dense series with a long horizon builds a very large payload.| | `database` | string | `""` | Optional override database for **writes only**. If unset, forecasts are written to the trigger’s database. Reads always go to the trigger’s database. | #### Tag filter format The plugin supports multi-value tag filters that are translated to `tag IN ('a', 'b', ...)` in SQL. **Trigger arguments (string form)** — based on the downsampler convention: * `.` separates `key:value` pairs * `:` separates the tag name from its value(s) * `@` separates multiple values for the same tag * Quote a value with `'...'` or `"..."` if it contains `:`, `@` or `.`. A quote inside a value, as in `Bob's`, needs no escaping. An unclosed quote is rejected. Examples: ``` tags="room:Bedroom" tags="room:Bedroom@Kitchen.location:Hall" tags="room:'Some other room'@Bedroom.device:sensor1" tags="owner:Bob's.room:Bedroom" ``` **Request body (JSON form)**: ```json { "tags": { "room": ["Bedroom", "Kitchen"], "location": "Hall" } } ``` The body also accepts the string form above. Send `{}` to clear the tag filters configured on the trigger. #### Forecast points and tags When a tag filter has a single value, that value is added as a tag on every forecast point. When it has multiple values (an `IN (...)` filter), no value is written for that tag — the response covers several tag values at once. ## Requirements ### Dependencies * Python 3.11 or higher * `pandas` — Data manipulation * `requests` — HTTP client for the Synthefy API * `influxdata-plugin-utils` — Shared configuration, schema and write helpers ### Installation steps Using the InfluxDB 3 Enterprise package manager: ```bash influxdb3 install package pandas influxdb3 install package requests influxdb3 install package influxdata-plugin-utils ``` ### Prerequisites * InfluxDB 3 Enterprise Core or Enterprise installed and running * Synthefy API key — create one at [https://console.synthefy.com/api-keys](https://console.synthefy.com/api-keys) ## Quick Start / Testing Setup Create a database and write some sample data: ```bash influxdb3 create database mydb NOW=$(date +%s) for i in {0..168}; do TIMESTAMP=$((NOW - (168 - i) * 3600))000000000 influxdb3 write --database mydb "temperature,location=NYC value=$((70 + RANDOM % 10)),humidity=$((60 + RANDOM % 15)),pressure=$((1000 + RANDOM % 20)) ${TIMESTAMP}" done ``` **Note**: In InfluxDB 3 Enterprise, tables/measurements are created automatically when you first write data to them. **Quick check that data exists:** ```bash influxdb3 query --database mydb "SELECT COUNT(*) FROM temperature" influxdb3 query --database mydb "SELECT * FROM temperature ORDER BY time DESC LIMIT 5" ``` ## Trigger setup ### HTTP trigger Create and enable the trigger: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/synthefy_forecasting/synthefy_forecasting.py" \ --trigger-spec "request:forecast" \ --trigger-arguments measurement=temperature,field=value \ temperature_forecast_http influxdb3 enable trigger --database mydb temperature_forecast_http ``` Trigger arguments act as defaults; any field can be overridden in the request body. Then call via HTTP. **Always** pass the Synthefy API key as a header (or set the `SYNTHEFY_API_KEY` env var on the InfluxDB process): ```bash TOKEN=$(influxdb3 create token --admin --offline | grep token | cut -d'=' -f2) curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "temperature", "field": "value", "time_range": "30d", "forecast_horizon": "7d", "model": "sfm-tabular" }' ``` **Important Notes:** * Endpoint is `/api/v3/engine/forecast` (matches the `request:forecast` trigger spec). * Authentication for InfluxDB itself is handled by the framework via the `Authorization` header. * The Synthefy API key must be in the `X-Synthefy-Api-Key` header or in the `SYNTHEFY_API_KEY` env var; it cannot be passed via trigger arguments or request body. ## Example usage ### Basic forecast ```bash TOKEN=$(influxdb3 create token --admin --offline | grep token | cut -d'=' -f2) curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "temperature", "field": "value" }' ``` ### Forecast with single-tag filter ```bash curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "temperature", "field": "value", "tags": { "location": "NYC" }, "time_range": "30d" }' ``` ### Forecast with multi-value tag filter ```bash curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "temperature", "field": "value", "tags": { "location": ["NYC", "SF"], "device": "sensor1" }, "time_range": "30d" }' ``` ### Forecast with metadata covariates ```bash curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "temperature", "field": "value", "metadata_fields": ["humidity", "pressure"], "time_range": "30d" }' ``` ### Trigger arguments form The same parameters can also be set as trigger arguments (note the dot/colon/`@` syntax for `tags` and the space-separated list for `metadata_fields`): ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/synthefy_forecasting/synthefy_forecasting.py" \ --trigger-spec "request:forecast" \ --trigger-arguments 'measurement=temperature,field=value,tags=location:NYC@SF.device:sensor1,metadata_fields=humidity pressure,time_range=30d,forecast_horizon=7d' \ temperature_forecast_http ``` ### Advanced model ```bash curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "sales", "field": "revenue", "model": "Migas-latest", "forecast_horizon": "30d", "time_range": "90d" }' ``` ### Writing the forecast to a different database ```bash curl -X POST http://localhost:8181/api/v3/engine/forecast \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ -H "X-Synthefy-Api-Key: YOUR_SYNTHEFY_API_KEY" \ -d '{ "measurement": "temperature", "field": "value", "database": "forecasts" }' ``` ## Output Format Forecasts are written to a new measurement (default: `{measurement}_forecast`) using `write_sync` (or `write_sync_to_db` when `database` is set) with `no_sync=True`, batched into a single line-protocol payload. * **Measurement**: `{measurement}_forecast` (configurable via `output_measurement`) * **Tags**: Single-value tag filters from the request + `model={model_name}` * **Fields**: * `{field_name}`: Forecasted values * `value_{quantile}`: Quantile forecasts when available (for example, `value_0.1`, `value_0.9`) Example Line Protocol output: ``` temperature_forecast,location=NYC,model=sfm-tabular value=72.5,value_0.1=71.2,value_0.9=73.8 1704672000000000000 ``` ## Querying Forecasts Forecast points sit in the future, so query by an upcoming time window rather than a past one: ```sql SELECT * FROM temperature_forecast WHERE time >= now() AND time <= now() + INTERVAL '7 days' ORDER BY time ``` Historical data — past 30 days: ```sql SELECT time, value FROM temperature WHERE time >= now() - INTERVAL '30 days' ORDER BY time ``` ## Supported Models The plugin supports models available through the Synthefy Forecasting API. Key supported models include: * `sfm-tabular`: Synthefy Foundation Model for tabular/multivariate time series * `Migas-latest`: Latest Migas foundation model Check the [Synthefy documentation](https://docs.synthefy.com) for the most up-to-date model list and availability. ## Code overview ### Files * `synthefy_forecasting.py`: HTTP trigger entry point, argument parsing, InfluxDB query construction, Synthefy API calls, and forecast writes. * `requirements.txt`: Python packages required by the plugin. * `README.md`: Setup, usage, and troubleshooting documentation. ### Key functions * `process_request(influxdb3_local, query_parameters, request_headers, request_body, args=None)`: handles HTTP requests, merges trigger arguments with request-body overrides, validates input, calls Synthefy, and writes forecast points. * `build_history_query(measurement, field, metadata_fields, tag_filters, start_time)`: builds the parameterized SQL query for historical data. * `dataframe_to_synthefy_request(influxdb3_local, df, field, forecast_horizon, metadata_fields, model, max_forecast_points, task_id)`: converts InfluxDB query rows into the Synthefy forecast request payload and enforces the point limit. * `forecast_response_to_line_builders(influxdb3_local, forecast_response, output_measurement, tag_filters, model, field_name, task_id)`: converts Synthefy forecast results into InfluxDB line protocol builders. ## Troubleshooting ### Common issues ### Missing API key `{"message":"Missing API key"}` — set `X-Synthefy-Api-Key` in the request headers, or `SYNTHEFY_API_KEY` in the environment of the InfluxDB process. ### Measurement not found `{"message":"Measurement '' not found"}` — the plugin verifies the measurement exists by querying `information_schema.columns`. Make sure the measurement has been written to in the trigger’s database. ### Field does not exist `{"message":"Field '' does not exist in ''"}` — the requested `field` (or `metadata_fields` entries) was not found in the measurement schema. ### No data found `{"message":"No data found"}` — the query returned zero rows. Widen `time_range`, relax `tags` filters, or verify timestamps fall inside the window. ### Invalid interval format `Invalid interval format: ''. Expected ''.` — `time_range` and `forecast_horizon` must be `` (units: `us`, `ms`, `s`, `min`, `h`, `d`, `w`, `m`, `q`, `y`) or, for `forecast_horizon`, ` points`. ### Forecast horizon too large `forecast_horizon '' resolves to N points … above the max_forecast_points limit` — the horizon is divided by the interval between the last two historical points, so a dense series produces many points. Shorten `forecast_horizon`, use the ` points` form, or raise `max_forecast_points`. ### Timestamps collapse onto each other `N history timestamps differ by less than a microsecond …` or `The series' step of … is finer than a microsecond …` — timestamps are sent to Synthefy with microsecond precision, so a series stepping in nanoseconds cannot be represented. Resample it to a coarser step. ### History holds repeated timestamps `History holds N repeated timestamps, so the window covers more than one series …` — the query matched several tag series and their values are interleaved in one input sequence. Add a `tags` filter that selects a single series. ### Synthefy API errors If Synthefy API calls fail: * Verify the API key is correct * Check API URL accessibility and rate limits * Check network connectivity ### Write errors If writes fail: * Ensure the database exists (the trigger database, or the override `database` if used) * Ensure the plugin has write permissions * Check the `[task_id] Failed to write forecasts after N attempts: …` error in the InfluxDB logs ### Query file limit exceeded (InfluxDB 3 Enterprise Core) If you see “Query would scan X Parquet files, exceeding the file limit” errors, narrow `time_range` (for example, `90d` instead of `730d`), or upgrade to InfluxDB 3 Enterprise Enterprise (which compacts files and removes the limit). ## Limitations * Currently supports a single time series per request (one `field` plus optional covariates). A request whose window matches several tag series is logged as a warning. * Forecast horizon calculation assumes regular time intervals. * Timestamps are exchanged with microsecond precision; a series whose step is finer is rejected. * In the trigger-arguments string form, a tag value containing `:`, `@` or `.` must be quoted; the JSON request body needs no quoting. ## License Apache 2.0 ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Synthefy forecasting plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/synthefy_forecasting) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Source measurement (table) containing historical data | | field | string | "value" | Field name to forecast | | tags | string | dict | "" | Tag filters. Trigger args: dot-separated string key:val1@val2.key2:val3 . Request body: JSON object mapping tag name to a string or list of strings. See Tag filter format . | | time_range | string | "30d" | Historical window. Format: . Units: us , ms , s , min , h , d , w , m , q , y ( m / q / y are approximate). | | forecast_horizon | string | "7d" | Forecast duration. Format: (same units as time_range ) or points . | | model | string | "sfm-tabular" | Synthefy model identifier (for example, sfm-tabular , Migas-latest ) | | output_measurement | string | "{measurement}_forecast" | Destination measurement for forecast results | | metadata_fields | string | list | "" | Trigger args: space-separated list of field names ( "humidity pressure" ). Request body: JSON list of strings. Used as covariates. | | max_forecast_points | integer | 10000 | Upper bound on the forecast points one request may produce. A time-based forecast_horizon is divided by the series’ own step, so a dense series with a long horizon builds a very large payload. | | database | string | "" | Optional override database for writes only . If unset, forecasts are written to the trigger’s database. Reads always go to the trigger’s database. | --- # Synthefy nori regression plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/synthefy-nori-regression/ > > > **Note:** This plugin requires InfluxDB 3 Enterprise.8.2 or later (it uses the synchronous write API). > > Predict a numeric field in an InfluxDB 3 Enterprise measurement from other columns on the same rows with**Nori**, Synthefy’s in-context-learning tabular regression model, called through the Synthefy inference gateway. The plugin reads a window of rows, trains on the rows where the target field is present, predicts the rows where it is null (imputation / backfill), and writes the predicted values back into InfluxDB. Nori is a tabular regression foundation model: you give it labeled feature rows (`X_train`,`y_train`) and query rows (`X_test`) in a single request, and it predicts a value for each query row in one forward pass, with no training or fine-tuning step. This plugin applies Nori to an InfluxDB measurement. You choose a target field and a set of feature columns; the plugin uses the rows where the target is present as the in-context training set and predicts the target for the rows where it is null, writing each prediction back at its own row’s timestamp. It is plain tabular regression: Nori sees only the feature columns you name, with no time or ordering assumptions. Typical uses: * Backfill a field that dropped out (a sensor went offline while its neighbors kept reporting). * Impute a missing metric from correlated ones (for example, predict `pressure` from `temperature`and `humidity`). * Derive a field that is expensive to measure directly from cheaper ones recorded alongside it. Key features: * **In-context tabular regression**: no training step; the recent labeled rows are the context. * **Imputation / backfill**: predicts the rows where the target is null and writes them back. * **Scheduled or on-demand**: run on an interval, or call an HTTP endpoint with an explicit window. * **Idempotent by default**: rows that already hold a prediction are skipped, so a repeating schedule does not re-send and re-pay for the same rows. * **Bounded cost**: row caps and a batch size keep one run’s billed rows predictable. * **Single-series guarantee**: a run that resolves to more than one series fails before it calls the gateway, rather than training one model on two mixed series. ## Configuration Plugin parameters may be given as key-value pairs in the `--trigger-arguments` flag of`influxdb3 create trigger`, in the `trigger_arguments` field of the API, or entirely from a TOML file via `config_file_path` — see [TOML configuration](#toml-configuration). For the HTTP trigger, a documented subset may also be sent in the JSON request body. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that declares the supported trigger types (`scheduled`, `http`) and every parameter each accepts, so the[InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI can render a configuration form. ### Authentication for the Nori gateway The Nori gateway API key is a secret and is **never** read from trigger arguments or the request body (both are logged). It is resolved, in order: 1. a non-empty `X-Nori-Api-Key: ` request header (HTTP trigger only), then 2. the `SYNTHEFY_NORI_API_KEY` environment variable set on the InfluxDB host (required for the scheduled trigger). The key is intentionally **not** accepted in the `Authorization` header: InfluxDB parses`Authorization` for its own request authorization, so a key placed there never reaches the plugin. Use the custom `X-Nori-Api-Key` header instead. Get a Nori API key from the [Synthefy console](https://console.synthefy.com/). One key covers every model slug its group is granted (see [Supported models](#supported-models)), so you do not normally need a key per variant. This plugin does not create keys. The gateway endpoint itself is **not** a parameter: the request carries the operator’s API key and the training data, so a caller must never be able to choose its destination. An operator running a private gateway can point the plugin at it with the `NORI_GATEWAY_URL` environment variable on the InfluxDB host. It must be an `https://` URL; plain `http://` is accepted only for a loopback host (`localhost`, `127.0.0.1` or `::1`), so a local mock gateway still works in testing. ### Required parameters | Parameter | Type |Default | Description | |----------------|------|--------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `measurement` |string|required| Source measurement (table) to read from. | | `field` |string|required| The numeric field to predict. The plugin trains on the rows where it is present and predicts the rows where it is null. | |`feature_fields`|string|required|Numeric feature columns (X) used to predict `field`, **space-separated** (for example `temp humidity`). Use spaces, not commas (`--trigger-arguments` splits argument pairs on commas) and not dots (a field name may contain a `.`).| | `model` |string|required| The Nori gateway slug to call. There is no default: the slug selects a priced model, so the plugin will not choose one for you. See [Supported models](#supported-models). Trigger argument only. | A column name that contains a space cannot be expressed in `feature_fields` as a trigger argument, because every string form splits on whitespace. Name such a column from a TOML array (`feature_fields = ["air temp", "humidity"]`) or from a JSON list in the HTTP body. ### Optional parameters | Parameter | Type | Default | Description | |--------------------|-------|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `window` |string | `30d` | Time window of rows to read, ending at the trigger’s call time. Units: `s`, `min`, `h`, `d`, `w`, with an integer magnitude. | | `start_time` |string | *(none)* | ISO 8601 start of a fixed window. Given alone, the window ends now. | | `end_time` |string | *(none)* | ISO 8601 end of a fixed window. Given alone, the window starts one `window` earlier. | | `tags` |string | *(none)* | Filter to a single series. Format: `key:val key2:val2` (space-separated pairs, one value per key). A token without a `:` is rejected. Required when the window holds more than one series. | |`output_measurement`|string |`_regressed`| Measurement to write predictions to. Must differ from `measurement`. | | `target_database` |string | *(trigger db)* | Write predictions to a different database. | | `dry_run` |boolean| `false` | Log the first few predictions and return them all, without writing anything. | | `skip_existing` |boolean| `true` | Skip rows that already hold a prediction in `output_measurement`. Set `false` to refresh earlier predictions with newer training data. | | `min_history` |integer| `50` | Minimum labeled rows required to train; the run is skipped below this. | | `max_train_rows` |integer| `1000` | Cap on labeled rows sent as the training context; the most recent rows are kept. This is the main cost control — the gateway bills per training row and column. | | `max_predict_rows` |integer| `5000` | Cap on rows predicted per run; the most recent rows are kept and the rest wait for a later run. | | `max_read_rows` |integer| `50000` |Ceiling on rows read from InfluxDB in one run, applied as a `LIMIT` on the query. The most recent rows are read, and a truncated read is logged with a warning. This bounds the plugin’s memory: a row costs roughly 0.7 KB while it is held, so the default is about 35 MB.| |`predict_batch_size`|integer| `1000` | Rows per gateway call. Each batch re-sends the training context and is billed separately, so a larger value costs less. | | `request_timeout` |string | `300s` | Timeout for one gateway call. A model that has scaled to zero cold-starts on the first request, measured between roughly one and four minutes depending on the variant, so keep this well above the warm response time. | | `max_retries` |integer| `3` | Maximum attempts per gateway call and per write. `1` disables retry. | | `config_file_path` |string | *(none)* | Path to a TOML file supplying every parameter, relative to `PLUGIN_DIR`. Cannot be combined with other inline arguments or a request body. | Two constraints are checked before anything runs: `min_history` must not exceed `max_train_rows`(no run could otherwise ever qualify), and `output_measurement` must differ from `measurement`. All of the integer parameters must be at least `1`. ### HTTP request body parameters On the HTTP trigger, these keys may be sent in the JSON request body: `measurement`, `field`, `feature_fields`, `tags`, `window`, `start_time`, `end_time`, `dry_run`. `feature_fields` may be a JSON list (`{"feature_fields": ["temp", "humidity"]}`) or a space-separated string, and `tags` may be a JSON object (`{"tags": {"site": "A"}}`). **A trigger argument pins its value.** The body may fill in what the trigger left open, but it cannot change what the trigger already set — that is rejected. So an operator who wants the request to choose the measurement creates the trigger without one, and an operator who wants it fixed sets it as a trigger argument. This matters because `output_measurement` defaults to`_regressed`: without the pin, a body-supplied `measurement` would move the write target too. Every other parameter is **operator-only** and is rejected by name if it appears in the body. The endpoint is reachable by anyone holding a database token, so the model slug (which selects a billed model), the write targets (`output_measurement`, `target_database`), the row caps, the timeout and`config_file_path` stay under the control of whoever created the trigger. `gateway_url` is not a parameter at all, in either place — use the `NORI_GATEWAY_URL` environment variable. Passing it (or a parameter from the plugin’s earlier forecasting revision: `mode`,`horizon`, `step`, `lags`, `rolling`, `tz`) is rejected with a message naming the replacement, rather than ignored. ### TOML configuration Set the `PLUGIN_DIR` environment variable and reference the file with the `config_file_path` trigger argument (relative paths resolve against `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`). The TOML file then supplies **all** parameters — it is mutually exclusive with inline trigger arguments and with an HTTP request body. See[`synthefy_nori_regression_config_scheduler.toml`](synthefy_nori_regression_config_scheduler.toml) for an annotated template. ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/synthefy_nori_regression/synthefy_nori_regression.py" \ --trigger-spec "every:1h" \ --trigger-arguments config_file_path=synthefy_nori_regression_config_scheduler.toml \ nori_from_toml ``` ## Requirements ### Software requirements * **InfluxDB 3 Enterprise Core or Enterprise**, version 3.8.2 or later, with the Processing Engine enabled (`influxdb3 serve --plugin-dir /path/to/plugins`). * **Python packages**: `influxdata-plugin-utils>=0.3.0`, `requests`. * A **Nori API key** from the [Synthefy console](https://console.synthefy.com/), reachable from the InfluxDB host over HTTPS. ### Installation steps 1. Install the Python dependencies into the InfluxDB 3 Enterprise Processing Engine environment: ``` influxdb3 install package influxdata-plugin-utils requests ``` 2. Reference the plugin directly from this repository with the `gh:` prefix (the form used in the examples below): `--path "gh:influxdata/synthefy_nori_regression/synthefy_nori_regression.py"`. Alternatively, copy`synthefy_nori_regression.py` into your plugin directory (the one passed to `influxdb3 serve --plugin-dir`) and use `--path synthefy_nori_regression.py`. 3. Set the Nori gateway key on the InfluxDB host, so the scheduled trigger can read it: ``` export SYNTHEFY_NORI_API_KEY="" ``` ### Data requirements * The measurement holds at least `min_history` rows where the target `field` is present **and**every `feature_fields` column is present. Those rows are the training context. * It holds at least one row where the target is null and every feature is present. Those rows are what gets predicted; if there are none, the run is a no-op. * The window resolves to a **single series**. If the measurement holds several series (one per`site`, say), pass a `tags` filter that isolates one, or create one trigger per series. * The features actually explain the target. Nori sees no time and no row order, so a target that depends on time rather than on the feature columns is not a good fit for this plugin. ### Schema requirements The plugin reads `information_schema.columns` before it queries data, and fails with a message naming the offending column if the schema cannot serve the request: | Column | Required type | |---------------------------|-----------------------------------------------------------------| | `time` | timestamp (every InfluxDB measurement has one) | | `field` (the target) |numeric field: `Int64`, `UInt64`, `Int32`, `Float64` or `Float32`| |each `feature_fields` entry| numeric field, and neither the target nor `time` | | each `tags` key | a tag column (`Dictionary(Int32, Utf8)`), not a field | | the source tag columns | none named `model`, `source` or `target` | A string or boolean column named as a feature is rejected here rather than coerced to null, which would otherwise surface much later as `only 0 labeled rows`. The last row matters because every output point carries `model`, `source` and `target` tags for provenance. A source tag with one of those names would overwrite the provenance on write *and* make the `skip_existing` lookup contradict itself, so the run would silently re-send and re-pay for the same rows on every tick. The plugin refuses the configuration instead. ## Trigger setup ### Scheduled trigger Every 15 minutes, fill any rows of `sensors` (for `site=A`) that are missing `pressure`, predicting it from `temp` and `humidity`: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/synthefy_nori_regression/synthefy_nori_regression.py" \ --trigger-spec "every:15m" \ --trigger-arguments measurement=sensors,field=pressure,feature_fields="temp humidity",tags=site:A,model=synthefy/nori-30m \ nori_sensors_pressure ``` Because `skip_existing` defaults to `true`, each subsequent run only sends the rows that still have no prediction. Once the window is fully imputed, the trigger stops calling the gateway entirely. ### HTTP trigger ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/synthefy_nori_regression/synthefy_nori_regression.py" \ --trigger-spec "request:nori_regress" \ nori_http ``` ## Example usage ### Example 1: impute a missing field on a schedule Write sample data. The plugin needs at least `min_history` complete rows to train on, so this example lowers that to `3` — a real deployment should leave it at the default and train on far more. The last two rows carry `temp` and `humidity` but no `pressure`, and those are the ones that get imputed: ```bash influxdb3 write --database mydb --precision s " sensors,site=A temp=20.0,humidity=40.0,pressure=1000.0 1767225600 sensors,site=A temp=22.0,humidity=41.0,pressure=1000.7 1767225660 sensors,site=A temp=24.0,humidity=42.0,pressure=1001.4 1767225720 sensors,site=A temp=25.0,humidity=45.0 1767229200 sensors,site=A temp=21.0,humidity=41.0 1767229260 " ``` ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/synthefy_nori_regression/synthefy_nori_regression.py" \ --trigger-spec "every:15m" \ --trigger-arguments measurement=sensors,field=pressure,feature_fields="temp humidity",tags=site:A,model=synthefy/nori-30m,min_history=3 \ nori_example ``` Read the predictions back after the trigger runs: ```bash influxdb3 query --database mydb " SELECT time, value, model, target, site FROM sensors_regressed ORDER BY time DESC LIMIT 5 " ``` **Expected output:** ``` +---------------------+--------+-------------------+----------+------+ | time | value | model | target | site | +---------------------+--------+-------------------+----------+------+ | 2026-01-01T01:01:00 | 998.2 | synthefy/nori-30m | pressure | A | | 2026-01-01T01:00:00 | 999.0 | synthefy/nori-30m | pressure | A | +---------------------+--------+-------------------+----------+------+ ``` ### Example 2: on-demand HTTP regression Call the HTTP endpoint (exposed at `/api/v3/engine/`), passing the Nori key in the header: ```bash curl -X POST http://localhost:8181/api/v3/engine/nori_regress \ -H "X-Nori-Api-Key: $SYNTHEFY_NORI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"measurement":"sensors","field":"pressure","feature_fields":["temp","humidity"],"tags":{"site":"A"}}' ``` **Expected output:** ```json {"status": "success", "task_id": "...", "result": {"status": "success", "written": 24}} ``` A run that had nothing to do reports its real outcome instead of a bare success: ```json {"status": "skipped", "task_id": "...", "result": {"status": "skipped", "written": 0}} ``` A run stopped part-way by a gateway fault keeps the batches it already paid for and reports the shortfall, so a caller never reads a partial result as a complete one: ```json {"status": "partial", "task_id": "...", "result": {"status": "partial", "written": 8, "remaining": 12}} ``` The top-level `status` is one of `success`, `partial`, `skipped`, `dry_run` or `failed`. ### Example 3: backfill a specific window ```bash curl -X POST http://localhost:8181/api/v3/engine/nori_regress \ -H "X-Nori-Api-Key: $SYNTHEFY_NORI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"measurement":"sensors","field":"pressure","feature_fields":["temp","humidity"],"tags":{"site":"A"},"start_time":"2026-01-01T00:00:00Z","end_time":"2026-02-01T00:00:00Z"}' ``` Either bound may be given alone: `start_time` on its own reads up to now, and `end_time` on its own reads the `window` before it. ### Example 4: dry run (preview without writing) ```bash curl -X POST http://localhost:8181/api/v3/engine/nori_regress \ -H "X-Nori-Api-Key: $SYNTHEFY_NORI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"measurement":"sensors","field":"pressure","feature_fields":["temp","humidity"],"tags":{"site":"A"},"dry_run":true}' ``` ## Output format Each prediction is written as a point: * **Measurement:** `output_measurement` (default `_regressed`). * **Tags:** `model` (the slug), `source` (the input measurement), `target` (the predicted field), plus every tag of the source series (not only the tags you filtered on), so a point can always be traced back to the series it was predicted for. * **Field:** `value` (float): the predicted target value. * **Timestamp:** the predicted row’s own timestamp (nanoseconds). Example line protocol: ``` sensors_regressed,model=synthefy/nori-30m,source=sensors,target=pressure,site=A value=1001.2 1767225600000000000 ``` ## Cost and metering Every gateway call is a billed request, priced from the **training** rows and columns you send (`max_train_rows` x the number of `feature_fields`), with a per-request floor. Three settings control what a run costs: * `max_train_rows` bounds the priced rows in every call. * `predict_batch_size` bounds the number of calls: each batch re-sends the same training context and is billed again, so a larger batch size is cheaper. * `skip_existing` (on by default) stops a repeating schedule from paying for rows it has already predicted. With it off, an `every:15m` trigger over a 30-day window re-sends each row roughly 2,880 times. ## Querying predictions ```bash influxdb3 query --database mydb " SELECT date_trunc('hour', time) AS hour, count(*) AS predicted, avg(value) AS mean_value FROM sensors_regressed WHERE target = 'pressure' GROUP BY 1 ORDER BY 1 DESC " ``` ## Notes * **What it predicts:** rows in the window where the target `field` is null but every`feature_fields` column is present. Rows where the target is already present become the training set. It never overwrites an existing target value. * **One series per run:** the plugin counts the distinct tag combinations in the window and fails before calling the gateway if there is more than one, because predictions are written back at each row’s own timestamp and two series would train as one model. * **Features only:** Nori sees just the columns you name in `feature_fields`. Row order does not matter, and no time-derived features are added. * **Non-finite predictions:** the gateway returns JSON `null` for a row it cannot produce a finite value for. Those rows are skipped and counted in the log; a batch that is entirely null fails rather than reporting a successful run that wrote nothing. * **A partial run keeps what it paid for, and says so:** if a later batch fails, the predictions the earlier batches already returned are still written, because those batches were already billed. The run reports `{"status": "partial", "written": N, "remaining": M}` rather than `success`, and the remaining rows are picked up by the next run. ## Supported models The `model` argument is the Nori gateway slug your API key is granted. It is **required**: the slug selects a priced model, so the plugin will not choose one on your behalf. Synthefy’s own client and local package take the same position. Synthefy publishes the current models, their sizes and their slugs at[docs.synthefy.com/nori/quickstart#models](https://docs.synthefy.com/nori/quickstart#models). That list is the authoritative one: it changes when Synthefy releases a variant, and a slug not on it will not route. Which model predicts better depends on your data, and the larger ones cost more per request and take longer to cold-start. Try a couple with `dry_run=true` before committing a schedule to one. The bare `synthefy/nori` slug has been retired and no longer routes; the plugin rejects it with a pointed message rather than letting the gateway answer `404`. One API key from the[Synthefy console](https://console.synthefy.com/) works for every slug it is granted. ## Code overview ### Files * `synthefy_nori_regression.py`: the plugin (metadata docstring and implementation). * `synthefy_nori_regression_config_scheduler.toml`: annotated TOML configuration template. * `test_synthefy_nori_regression.py`: unit tests (`pytest influxdata/synthefy_nori_regression/`); no engine or network needed. * `requirements.txt`: Python dependencies. * `manifest.toml`: packaging metadata. ### Key functions * `process_scheduled_call(influxdb3_local, call_time, args)`: scheduled entry point; anchors the window to `call_time`. * `process_request(influxdb3_local, query_parameters, request_headers, request_body, args)`: HTTP entry point; applies the request-body allowlist. * `_load_config(args, body)`: merges trigger arguments, the TOML file and the allowlisted body keys, then validates them. * `_resolve_schema(influxdb3_local, cfg)`: reads column names *and* types, rejecting a non-numeric target or feature. * `_resolve_window(cfg, now)`: resolves `window` / `start_time` / `end_time` into one range, honouring each bound on its own. * `_regress(...)`: enforces the single-series rule, splits labeled from null-target rows, applies the caps and `skip_existing`, and batches the gateway calls. * `_call_nori(...)`: sends the in-context regression request and validates the response. * `_write_predictions(...)`: writes the predictions with `write_sync` so a write error surfaces during trigger execution. ## Troubleshooting ### Common issues Each heading below quotes the text the plugin actually logs or returns, so a message can be searched for directly. Every failure is logged with a `task_id`; use it to correlate the caller-facing message with the full detail in `processing_engine_logs`. #### Missing API key The plugin cannot find a Nori gateway key. **Solution:** set `SYNTHEFY_NORI_API_KEY` on the InfluxDB host, or pass an`X-Nori-Api-Key: ` header when calling the HTTP trigger (see[Authentication](#authentication-for-the-nori-gateway)). An empty header value is ignored and the environment variable is used instead. #### Gateway returns 403 or 404 * **`HTTP 403 ... please check the api-key you provided`:** the key is wrong, revoked, or malformed. * **`HTTP 404 ... please check the model you provided`:** the `model` slug does not exist or your key’s group was not granted it. Confirm the spelling against[Supported models](#supported-models). **Solution:** re-copy the key from the [Synthefy console](https://console.synthefy.com/) and check the slug. Neither status is retried, because neither is transient. #### Request body may not set … The HTTP request body contained an operator-only parameter (for example `model` or`target_database`). **Solution:** set it as a trigger argument or in the TOML config file. Only the query-shape keys listed in [HTTP request body parameters](#http-request-body-parameters) may come from the body. #### `gateway_url` is not a parameter The endpoint moved out of the configuration entirely, because the request carries the Nori API key. **Solution:** set `NORI_GATEWAY_URL` on the InfluxDB host. It must be an `https://` URL. #### Not enough labeled rows, or nothing to predict * **`only N labeled rows (< min_history)`:** fewer than `min_history` rows have both the target and every feature present. Widen `window`, lower `min_history`, or check that`measurement`/`field`/`feature_fields`/`tags` select the data you expect. * **`no rows to predict`:** every target value in the window is already present. The plugin only fills rows where the target is null. * **`every row in the window already holds a prediction`:** `skip_existing` did its job. Set`skip_existing=false` to recompute them with newer training data. #### The window holds N series The measurement holds more than one series and your `tags` filter did not isolate one, so a single model would be trained on mixed series. **Solution:** add a `tags` filter that selects one series, or create one trigger per series. The error message lists the first few series it found. #### Feature or target column rejected A column does not exist, is not a numeric field, or clashes with the target field or the reserved names `time`/`y`. **Solution:** fix the column names, and check the types with`SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'sensors'`. #### Cold-start latency and timeouts The models scale to zero, so the first request after an idle period is slow: measurements have ranged from roughly one minute to nearly four, with the larger variants slower, and it can return a `503` or a non-JSON body from the fronting proxy once. **Solution:** the default `request_timeout` of `300s` and `max_retries` of `3` are set to absorb this; a `503`, a `429` and a connection error are retried with backoff. A read timeout is **not**retried — it has already spent the whole budget, and it usually means the key’s group was never granted the slug. Raise `request_timeout` only if you see genuine timeouts on a warm model. ## Limitations * One series per run; create one trigger per series for a multi-series measurement. Multi-series imputation in a single run is a possible enhancement. * Imputes only rows where the target is null; it never overwrites an existing value. * Prediction quality depends on how well `feature_fields` explain the target. Nori adds no time-derived features, so this plugin is not a time-series forecaster. * A feature column whose name contains a space is only reachable via a TOML array or the HTTP JSON body, not via `--trigger-arguments`. * Each gateway call is billed; see [Cost and metering](#cost-and-metering). ## License Apache 2.0. ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Synthefy nori regression plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/synthefy_nori_regression) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Source measurement (table) to read from. | | field | string | required | The numeric field to predict. The plugin trains on the rows where it is present and predicts the rows where it is null. | | feature_fields | string | required | Numeric feature columns (X) used to predict field , space-separated (for example temp humidity ). Use spaces, not commas ( --trigger-arguments splits argument pairs on commas) and not dots (a field name may contain a . ). | | model | string | required | The Nori gateway slug to call. There is no default: the slug selects a priced model, so the plugin will not choose one for you. See Supported models . Trigger argument only. | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | window | string | 30d | Time window of rows to read, ending at the trigger’s call time. Units: s , min , h , d , w , with an integer magnitude. | | start_time | string | (none) | ISO 8601 start of a fixed window. Given alone, the window ends now. | | end_time | string | (none) | ISO 8601 end of a fixed window. Given alone, the window starts one window earlier. | | tags | string | (none) | Filter to a single series. Format: key:val key2:val2 (space-separated pairs, one value per key). A token without a : is rejected. Required when the window holds more than one series. | | output_measurement | string | _regressed | Measurement to write predictions to. Must differ from measurement . | | target_database | string | (trigger db) | Write predictions to a different database. | | dry_run | boolean | false | Log the first few predictions and return them all, without writing anything. | | skip_existing | boolean | true | Skip rows that already hold a prediction in output_measurement . Set false to refresh earlier predictions with newer training data. | | min_history | integer | 50 | Minimum labeled rows required to train; the run is skipped below this. | | max_train_rows | integer | 1000 | Cap on labeled rows sent as the training context; the most recent rows are kept. This is the main cost control — the gateway bills per training row and column. | | max_predict_rows | integer | 5000 | Cap on rows predicted per run; the most recent rows are kept and the rest wait for a later run. | | max_read_rows | integer | 50000 | Ceiling on rows read from InfluxDB in one run, applied as a LIMIT on the query. The most recent rows are read, and a truncated read is logged with a warning. This bounds the plugin’s memory: a row costs roughly 0.7 KB while it is held, so the default is about 35 MB. | | predict_batch_size | integer | 1000 | Rows per gateway call. Each batch re-sends the training context and is billed separately, so a larger value costs less. | | request_timeout | string | 300s | Timeout for one gateway call. A model that has scaled to zero cold-starts on the first request, measured between roughly one and four minutes depending on the variant, so keep this well above the warm response time. | | max_retries | integer | 3 | Maximum attempts per gateway call and per write. 1 disables retry. | | config_file_path | string | (none) | Path to a TOML file supplying every parameter, relative to PLUGIN_DIR . Cannot be combined with other inline arguments or a request body. | | Column | Required type | | --- | --- | | Column | Required type | | time | timestamp (every InfluxDB measurement has one) | | field (the target) | numeric field: Int64 , UInt64 , Int32 , Float64 or Float32 | | each feature_fields entry | numeric field, and neither the target nor time | | each tags key | a tag column ( Dictionary(Int32, Utf8) ), not a field | | the source tag columns | none named model , source or target | --- # System metrics plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/system-metrics/ The System Metrics Plugin provides comprehensive system monitoring capabilities for InfluxDB 3 Enterprise, collecting CPU, memory, disk, and network metrics from the host system. Monitor detailed performance insights including per-core CPU statistics, memory usage breakdowns, disk I/O performance, and network interface statistics. Features configurable metric collection with robust error handling and retry logic for reliable monitoring. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Optional parameters | Parameter | Type | Default | Description | |-----------------|-------|-----------|------------------------------------------------------------------------------------------------| | `hostname` |string |`localhost`| Hostname to tag all metrics with for system identification | | `include_cpu` |boolean| `true` | Include comprehensive CPU metrics collection (overall and per-core statistics) | |`include_memory` |boolean| `true` | Include memory metrics collection (RAM usage, swap statistics, page faults) | | `include_disk` |boolean| `true` | Include disk metrics collection (partition usage, I/O statistics, performance) | |`include_network`|boolean| `true` | Include network metrics collection (interface statistics and error counts) | | `max_retries` |integer| `3` |Retry attempts per metric type; the group is skipped and the run continues once they are used up| *Note: This plugin has no required parameters. All parameters have sensible defaults.* Boolean parameters accept `true`/`false`, `1`/`0`, `yes`/`no`, and `on`/`off`. A value the plugin cannot interpret is reported in the logs and the run collects nothing, so fix the trigger arguments and the next run recovers. ### Environment variables Every parameter can also come from an environment variable named `INFLUXDB3_SYSTEM_METRICS_` in upper case — for example, `INFLUXDB3_SYSTEM_METRICS_HOSTNAME` sets `hostname`. The environment is the lowest layer: a trigger argument overrides it, and the TOML file overrides both. `INFLUXDB3_SYSTEM_METRICS_CONFIG_FILE_PATH` names the TOML file when the trigger doesn’t carry a `config_file_path` argument. ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Relative paths are resolved against the first directory that is set: `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. Only that directory is used — the file is not looked up in the remaining ones. Values in the TOML file override the inline trigger arguments. A file that cannot be read, is not named `.toml`, or holds an invalid value stops the run with a configuration error rather than falling back to the inline arguments. #### Example TOML configuration [system\_metrics\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/system_metrics/system_metrics_config_scheduler.toml) For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled. * **Python packages**: `influxdata-plugin-utils>=0.4.0`, `psutil` ### Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.4.0" influxdb3 install package psutil ``` ## Trigger setup ### Basic Scheduled Trigger ```bash influxdb3 create trigger \ --database system_monitoring \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --trigger-spec "every:30s" \ system_metrics_trigger ``` ### Using Configuration File ```bash influxdb3 create trigger \ --database system_monitoring \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --trigger-spec "every:1m" \ --trigger-arguments config_file_path=system_metrics_config_scheduler.toml \ system_metrics_config_trigger ``` ### Custom Configuration ```bash influxdb3 create trigger \ --database system_monitoring \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --trigger-spec "every:30s" \ --trigger-arguments hostname=web-server-01,include_disk=false,max_retries=5 \ system_metrics_custom_trigger ``` ## Example usage ### Monitor Web Server Performance ```bash # Create trigger for web server monitoring every 15 seconds influxdb3 create trigger \ --database web_monitoring \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --trigger-spec "every:15s" \ --trigger-arguments hostname=web-server-01,include_network=true \ web_server_metrics ``` ### Database Server Monitoring ```bash # Focus on CPU and disk metrics for database server influxdb3 create trigger \ --database db_monitoring \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --trigger-spec "every:30s" \ --trigger-arguments hostname=db-primary,include_disk=true,include_cpu=true,include_network=false \ database_metrics ``` ### High-Frequency System Monitoring ```bash # Collect all metrics every 10 seconds with higher retry tolerance influxdb3 create trigger \ --database system_monitoring \ --path "gh:influxdata/system_metrics/system_metrics.py" \ --trigger-spec "every:10s" \ --trigger-arguments hostname=critical-server,max_retries=10 \ high_freq_metrics ``` ### Query collected metrics This plugin collects system metrics automatically. After the trigger runs, query to view the collected data: ```bash influxdb3 query \ --database system_monitoring \ "SELECT * FROM system_cpu WHERE time >= now() - interval '5 minutes' LIMIT 5" ``` **Expected output** \+——+——–+——-+——–+——+——–+——-+——–+——-+——-+————+——————+ | host | cpu | user | system | idle | iowait | nice | irq | load1 | load5 | load15 | time | +——+——–+——-+——–+——+——–+——-+——–+——-+——-+————+——————+ | srv1 | total | 12.5 | 5.3 | 81.2 | 0.8 | 0.0 | 0.2 | 0.85 | 0.92 | 0.88 | 2024-01-15 10:00 | | srv1 | total | 13.1 | 5.5 | 80.4 | 0.7 | 0.0 | 0.3 | 0.87 | 0.93 | 0.88 | 2024-01-15 10:01 | | srv1 | total | 11.8 | 5.1 | 82.0 | 0.9 | 0.0 | 0.2 | 0.83 | 0.91 | 0.88 | 2024-01-15 10:02 | | srv1 | total | 14.2 | 5.8 | 79.0 | 0.8 | 0.0 | 0.2 | 0.89 | 0.92 | 0.88 | 2024-01-15 10:03 | | srv1 | total | 12.9 | 5.4 | 80.6 | 0.9 | 0.0 | 0.2 | 0.86 | 0.92 | 0.88 | 2024-01-15 10:04 | +——+——–+——-+——–+——+——–+——-+——–+——-+——-+————+——————+ ## Code overview ### Main Functions #### `process_scheduled_call()` The main entry point for scheduled triggers. Loads the configuration, then runs each enabled collector and writes the lines it built. A collector is retried up to `max_retries` times, and its lines are written only once it completes. ```python def process_scheduled_call(influxdb3_local, call_time, args=None): config: Config = load_config( parse_trigger_args(args, SETTINGS), parse_toml(args.get("config_file_path"), SETTINGS), validators=SETTING_VALIDATORS, ) for config_key, metric_type, collect in _COLLECTORS: if not config[config_key]: continue lines = _collect_with_retry( influxdb3_local, collect, metric_type, hostname, max_retries, task_id ) write_data(influxdb3_local, lines, batch=False, retries=0) ``` ### Measurements and Fields #### system\_cpu Overall CPU statistics and metrics: * **Tags**: `host`, `cpu=total` * **Fields**: `user`, `system`, `idle`, `iowait`, `nice`, `irq`, `softirq`, `steal`, `guest`, `guest_nice`, `frequency_current`, `frequency_min`, `frequency_max`, `ctx_switches`, `interrupts`, `soft_interrupts`, `syscalls`, `load1`, `load5`, `load15` The state shares (`user` through `guest_nice`) are derived from the change in the CPU time counters between two consecutive runs, so each value covers the interval between the previous run and the current one. They are absent on the first run after the trigger is created or restarted; the remaining fields are written from the first run on. #### system\_cpu\_cores Per-core CPU statistics: * **Tags**: `host`, `core` (core number) * **Fields**: `usage`, `user`, `system`, `idle`, `iowait`, `nice`, `irq`, `softirq`, `steal`, `guest`, `guest_nice`, `frequency_current`, `frequency_min`, `frequency_max` Shares are derived the same way as in `system_cpu`; `usage` is the busy share of the core, everything except `idle` and `iowait`. #### system\_memory System memory statistics: * **Tags**: `host` * **Fields**: `total`, `available`, `used`, `free`, `active`, `inactive`, `buffers`, `cached`, `shared`, `slab`, `percent` #### system\_swap Swap memory statistics: * **Tags**: `host` * **Fields**: `total`, `used`, `free`, `percent`, `sin`, `sout` #### system\_memory\_faults Memory page fault information (when available): * **Tags**: `host` * **Fields**: `page_faults`, `major_faults`, `minor_faults`, `rss`, `vms`, `dirty`, `uss`, `pss` #### system\_disk\_usage Disk partition usage: * **Tags**: `host`, `device`, `mountpoint`, `fstype` * **Fields**: `total`, `used`, `free`, `percent` #### system\_disk\_io Disk I/O statistics: * **Tags**: `host`, `device` * **Fields**: `reads`, `writes`, `read_bytes`, `write_bytes`, `read_time`, `write_time`, `busy_time`, `read_merged_count`, `write_merged_count` #### system\_disk\_performance Disk performance rates, derived from the change in the I/O counters between two consecutive runs of the plugin: * **Tags**: `host`, `device` * **Fields**: `read_bytes_per_sec`, `write_bytes_per_sec`, `read_iops`, `write_iops`, `avg_read_latency_ms`, `avg_write_latency_ms`, `util_percent` Each value covers the interval between the previous run and the current one, so the shorter the trigger interval, the finer the resolution. A device gets no line when there is nothing to compare against: on the first run after the trigger is created or restarted, and when its counters were reset (for example after the device was re-attached). #### system\_network Network interface statistics: * **Tags**: `host`, `interface` * **Fields**: `bytes_sent`, `bytes_recv`, `packets_sent`, `packets_recv`, `errin`, `errout`, `dropin`, `dropout` ## Troubleshooting ### Common issues #### Issue: Permission errors for disk I/O metrics **Solution**: The plugin will continue collecting other metrics even if some require elevated permissions. Run InfluxDB with appropriate permissions if disk I/O metrics are required. #### Issue: Missing Python packages **Solution**: Install the required packages: ```bash influxdb3 install package "influxdata-plugin-utils>=0.4.0" influxdb3 install package psutil ``` #### Issue: No `system_disk_performance` data, or CPU shares are missing **Solution**: Both are derived from two consecutive runs. Wait for the second run of the trigger; if the values stay missing, check the logs for `No previous disk I/O sample` and `No previous CPU sample`, which repeat when the cached counters are lost on every run. #### Issue: One metric group is missing while the others are written **Solution**: A collector that keeps failing is skipped so the rest of the run survives. Look for `Failed to collect metrics after N retries` in the logs, followed by `skipped after repeated failures`, which names every group left out of that run. #### Issue: No metrics at all and a configuration error in the logs **Solution**: An invalid setting stops the run before any collection. Look for `Configuration error` in the logs, which names the offending key and value, and fix the trigger arguments or the TOML file. An unreadable, wrongly named or invalid config file is reported the same way, so check the path in `config_file_path` resolves under `PLUGIN_DIR` and names a `.toml` file. #### Issue: High CPU usage from plugin **Solution**: Increase the trigger interval (for example, from `every:10s` to `every:30s`). Disable unnecessary metric types. Reduce the number of disk partitions monitored. ### Viewing Logs Logs are stored in the trigger’s database in the `system.processing_engine_logs` table: ```bash influxdb3 query \ --database YOUR_DATABASE \ "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'system_metrics_trigger' ORDER BY event_time DESC LIMIT 10" ``` ### Verifying Data Collection Check that metrics are being collected: ```bash # List all system metric measurements influxdb3 query \ --database system_monitoring \ "SHOW MEASUREMENTS WHERE measurement =~ /^system_/" # Check recent CPU metrics influxdb3 query \ --database system_monitoring \ "SELECT COUNT(*) FROM system_cpu WHERE time >= now() - interval '1 hour'" ``` ## Logging Logs are stored in the `_internal` database (or the database where the trigger is created) in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database _internal "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Log columns: * **event\_time**: Timestamp of the log event * **trigger\_name**: Name of the trigger that generated the log * **log\_level**: Severity level (INFO, WARN, ERROR) * **log\_text**: Message describing the action or error ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [System metrics plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/system_metrics) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[monitoring](/influxdb3/enterprise/tags/monitoring/)[system-metrics](/influxdb3/enterprise/tags/system-metrics/)[performance](/influxdb3/enterprise/tags/performance/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | hostname | string | localhost | Hostname to tag all metrics with for system identification | | include_cpu | boolean | true | Include comprehensive CPU metrics collection (overall and per-core statistics) | | include_memory | boolean | true | Include memory metrics collection (RAM usage, swap statistics, page faults) | | include_disk | boolean | true | Include disk metrics collection (partition usage, I/O statistics, performance) | | include_network | boolean | true | Include network metrics collection (interface statistics and error counts) | | max_retries | integer | 3 | Retry attempts per metric type; the group is skipped and the run continues once they are used up | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | --- # Threshold deadman checks plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/threshold-deadman-checks/ The Threshold Deadman Checks Plugin provides comprehensive monitoring capabilities for time series data in InfluxDB 3 Enterprise, combining real-time threshold detection with deadman monitoring. Monitor field values against configurable thresholds, detect data absence patterns, and trigger multi-level alerts based on aggregated metrics. Features both scheduled batch monitoring and real-time data write monitoring with configurable trigger counts and severity levels. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |-------------|------|--------|----------------------------------------------------------------------------------------------------------------------------------------| |`measurement`|string|required| Measurement to monitor for deadman alerts and aggregation-based conditions | | `senders` |string|required| Dot-separated notification channels with multi-channel notification integration | | `window` |string|required|Time window for periodic data presence checking. Format: ``, units: `s`, `min`, `h`, `d`, `w`. Must be a positive duration| ### Data write trigger parameters | Parameter | Type |Default | Description | |------------------|------|--------|------------------------------------------------------------------------------------------------------| | `measurement` |string|required| Measurement to monitor for real-time threshold violations in dual monitoring mode | |`field_conditions`|string|required|Real-time threshold conditions with multi-level alerting (INFO, WARN, ERROR, CRITICAL severity levels)| | `senders` |string|required| Dot-separated notification channels with multi-channel notification integration | ### Threshold check parameters | Parameter | Type |Default| Description | |--------------------------|-------|-------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`field_aggregation_values`|string | none | Multi-level aggregation conditions with aggregation support for avg, min, max, count, sum, median, stddev, first\_value, last\_value, var, and approx\_median values | | `deadman_check` |boolean| false | Enable deadman detection to monitor for data absence and missing data streams | | `interval` |string |“5min” | Aggregation time interval used in `DATE_BIN`. Format: ``, units: `s`, `min`, `h`, `d`, `w` | | `trigger_count` |number | 1 |Breaches required before alerting. Threshold checks count consecutive breaches per row identifier, including across the time bins of a single run; deadman checks count consecutive runs without data| ### Notification parameters | Parameter | Type |Default | Description | |-----------------------------|------|--------|-----------------------------------------------------------------------------------------| | `influxdb3_auth_token` |string|env var | InfluxDB 3 Enterprise API token with environment variable support | | `notification_deadman_text` |string|template| Customizable deadman alert template message with dynamic variables | |`notification_threshold_text`|string|template| Customizable threshold alert template message with dynamic variables | | `notification_text` |string|template|Customizable notification template message for data write triggers with dynamic variables| | `notification_path` |string|“notify”| Notification endpoint path with retry logic and exponential backoff | | `port_override` |number| 8181 | InfluxDB port override for notification delivery | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Relative paths are resolved against the first directory that is set: `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. Only that directory is used — the file is not looked up in the remaining ones. When `config_file_path` is set, the TOML file provides the whole configuration and inline trigger arguments are ignored. `INFLUXDB3_AUTH_TOKEN` from the environment still applies when `influxdb3_auth_token` is not set in the file. In TOML, `senders`, `field_conditions`, and `field_aggregation_values` use native structures instead of the inline string formats. Data write triggers cache the loaded configuration for 10 minutes to keep the write path fast, so configuration changes take effect within that window. Example TOML configuration files provided: * [threshold\_deadman\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/threshold_deadman_checks/threshold_deadman_config_scheduler.toml) - for scheduled triggers * [threshold\_deadman\_config\_data\_writes.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/threshold_deadman_checks/threshold_deadman_config_data_writes.toml) - for data write triggers For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ### Channel-specific configuration Notification channels require additional parameters based on the sender type (same as the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/)). ## Schema requirement The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. ## Software requirements * **InfluxDB v3 Core/Enterprise**: with the Processing Engine enabled. * **Python packages**: `influxdata-plugin-utils>=0.3.0`, `requests` * **Notification Sender Plugin for InfluxDB 3 Enterprise**: This plugin is required for sending notifications. See the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/). ## Installation steps 1. **Start InfluxDB 3 Enterprise with the Processing Engine enabled** (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. **Install required Python packages**: ``` influxdb3 install package "influxdata-plugin-utils>=0.3.0" influxdb3 install package requests ``` 3. **Optional**: For notifications, install and configure the [influxdata/notifier plugin](/influxdb3/enterprise/plugins/library/official/notifier/) ## Trigger setup ### Scheduled trigger Create a trigger for periodic threshold and deadman checks: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:10m" \ --trigger-arguments "measurement=cpu,senders=slack,field_aggregation_values=temp:avg@>=30-ERROR,window=10min,trigger_count=3,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \ threshold_scheduler ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Data write trigger Create a trigger for real-time threshold monitoring: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=cpu,field_conditions=temp>30-WARN:status==ok-INFO,senders=slack,trigger_count=2,slack_webhook_url=$SLACK_WEBHOOK_URL" \ threshold_datawrite ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Enable triggers ```bash influxdb3 enable trigger --database mydb threshold_scheduler influxdb3 enable trigger --database mydb threshold_datawrite ``` ## Example usage ### Example 1: Basic threshold monitoring Write test data and monitor for threshold violations: ```bash # Write test data influxdb3 write \ --database sensors \ "heartbeat,host=server1 status=1" influxdb3 write \ --database sensors \ "heartbeat,host=server1 status=0" # Create and enable the trigger influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:5m" \ --trigger-arguments "measurement=heartbeat,senders=slack,window=5min,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \ heartbeat_monitor influxdb3 enable trigger --database sensors heartbeat_monitor # Query to verify data influxdb3 query \ --database sensors \ "SELECT * FROM heartbeat ORDER BY time DESC LIMIT 5" ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. **Expected output** When no data is received within the window, a deadman alert is sent: “CRITICAL: No heartbeat data from heartbeat between 2025-06-01T10:00:00Z and 2025-06-01T10:05:00Z” ### Example 2: Deadman monitoring Monitor for data absence and alert when no data is received: ```bash influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments "measurement=heartbeat,senders=sms,window=10min,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321,notification_deadman_text=CRITICAL: No heartbeat data from \$table between \$time_from and \$time_to" \ heartbeat_monitor ``` ### Multi-level threshold monitoring Monitor aggregated values with different severity levels: ```bash influxdb3 create trigger \ --database monitoring \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:5m" \ --trigger-arguments "measurement=system_metrics,senders=slack.discord,field_aggregation_values='cpu_usage:avg@>=80-WARN cpu_usage:avg@>=95-ERROR memory_usage:max@>=90-WARN',window=5min,interval=1min,trigger_count=3,slack_webhook_url=$SLACK_WEBHOOK_URL,discord_webhook_url=$DISCORD_WEBHOOK_URL" \ system_threshold_monitor ``` Set `SLACK_WEBHOOK_URL` and `DISCORD_WEBHOOK_URL` to your webhook URLs. ### Real-time field condition monitoring Monitor data writes for immediate threshold violations: ```bash influxdb3 create trigger \ --database applications \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=response_times,field_conditions=latency>500-WARN:latency>1000-ERROR:error_rate>0.05-CRITICAL,senders=http,trigger_count=1,http_webhook_url=$HTTP_WEBHOOK_URL,notification_text=[\$level] Application alert: \$field \$op_sym \$compare_val (actual: \$actual)" \ app_performance_monitor ``` Set `HTTP_WEBHOOK_URL` to your HTTP webhook endpoint. ### Combined monitoring Monitor both aggregation thresholds and deadman conditions: ```bash influxdb3 create trigger \ --database comprehensive \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:10m" \ --trigger-arguments "measurement=temperature_sensors,senders=whatsapp,field_aggregation_values='temperature:avg@>=35-WARN temperature:max@>=40-ERROR',window=15min,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321" \ comprehensive_sensor_monitor ``` ## Code overview ### Files * `threshold_deadman_checks_plugin.py`: The main plugin code containing handlers for scheduled and data write triggers * `threshold_deadman_config_scheduler.toml`: Example TOML configuration for scheduled triggers * `threshold_deadman_config_data_writes.toml`: Example TOML configuration for data write triggers * `test_threshold_deadman_checks.py`: Pytest suite (59 tests, runs without a live InfluxDB 3 Enterprise server) * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`, `requests`) * `requirements-dev.txt`: Development dependencies (`pytest`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'threshold_scheduler'" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled threshold and deadman checks. Queries data within the specified window, evaluates aggregation-based conditions, and checks for data absence. #### `process_writes(influxdb3_local, table_batches, args)` Handles real-time threshold monitoring on data writes. Evaluates incoming data against configured field conditions with multi-level severity. ## Troubleshooting ### Common issues #### Issue: No alerts triggered **Solution**: Verify threshold values are appropriate for your data ranges. Check that notification channels are properly configured. Ensure the Notifier Plugin is installed and accessible. Review plugin logs for configuration errors. #### Issue: False positive alerts **Solution**: Increase `trigger_count` to require more consecutive breaches. In scheduled mode every `DATE_BIN` bin of the window counts as a breach, so keep `window`, `interval`, and `trigger_count` aligned. Adjust threshold values to be less sensitive. Consider longer aggregation intervals for noisy data. #### Issue: Missing deadman alerts **Solution**: Verify `deadman_check=true` is set in configuration. Check that the measurement name matches existing data. Ensure the time window is appropriate for your data frequency. #### Issue: Authentication issues **Solution**: Set `INFLUXDB3_AUTH_TOKEN` environment variable. Verify API token has required database permissions. Check Twilio credentials for SMS/WhatsApp notifications. ### Configuration formats **Aggregation conditions (scheduled)** * Format: `field:aggregation@operator value-level` * Example: `temp:avg@>=30-ERROR` * Multiple conditions: `"temp:avg@>=30-WARN humidity:min@<40-INFO"` **Field conditions (data write)** * Format: `field operator value-level` * Example: `temp>30-WARN:status==ok-INFO` * Supported operators: `>`, `<`, `>=`, `<=`, `==`, `!=` **Supported aggregations** * `avg`: Average value * `min`: Minimum value * `max`: Maximum value * `count`: Count of records * `sum`: Sum of values * `median`: Median value * `stddev`: Standard deviation * `first_value`: First value in time interval * `last_value`: Last value in time interval * `var`: Variance of values * `approx_median`: Approximate median (faster than exact median) ### Message template variables **Deadman notifications** * `$table`: Measurement name * `$time_from`: Start of checked period * `$time_to`: End of checked period **Threshold notifications (scheduled)** * `$level`: Alert severity level * `$table`: Measurement name * `$field`: Field name * `$aggregation`: Aggregation type * `$op_sym`: Operator symbol * `$compare_val`: Threshold value * `$actual`: Actual measured value * `$row`: Unique identifier **Threshold notifications (data write)** * `$level`: Alert severity level * `$field`: Field name * `$op_sym`: Operator symbol * `$compare_val`: Threshold value * `$actual`: Actual field value * `$trigger_count`: Consecutive matches required before alerting * `$row`: Unique identifier ### Row identification The `row` variable uniquely identifies alert contexts using format: `measurement:field[:aggregation]:level:tag1=value1:tag2=value2` (`aggregation` is present for scheduled threshold checks only). Tags without a value are omitted. Trigger counts are maintained independently for each unique combination of measurement, field, aggregation, severity level, tag values, **and the condition’s operator and threshold** — two conditions that differ only by threshold never share a count. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Threshold deadman checks plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/threshold_deadman_checks) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[monitoring](/influxdb3/enterprise/tags/monitoring/)[thresholds](/influxdb3/enterprise/tags/thresholds/)[deadman](/influxdb3/enterprise/tags/deadman/)[alerting](/influxdb3/enterprise/tags/alerting/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to monitor for deadman alerts and aggregation-based conditions | | senders | string | required | Dot-separated notification channels with multi-channel notification integration | | window | string | required | Time window for periodic data presence checking. Format: , units: s , min , h , d , w . Must be a positive duration | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to monitor for real-time threshold violations in dual monitoring mode | | field_conditions | string | required | Real-time threshold conditions with multi-level alerting (INFO, WARN, ERROR, CRITICAL severity levels) | | senders | string | required | Dot-separated notification channels with multi-channel notification integration | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | field_aggregation_values | string | none | Multi-level aggregation conditions with aggregation support for avg, min, max, count, sum, median, stddev, first_value, last_value, var, and approx_median values | | deadman_check | boolean | false | Enable deadman detection to monitor for data absence and missing data streams | | interval | string | “5min” | Aggregation time interval used in DATE_BIN . Format: , units: s , min , h , d , w | | trigger_count | number | 1 | Breaches required before alerting. Threshold checks count consecutive breaches per row identifier, including across the time bins of a single run; deadman checks count consecutive runs without data | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | influxdb3_auth_token | string | env var | InfluxDB 3 Enterprise API token with environment variable support | | notification_deadman_text | string | template | Customizable deadman alert template message with dynamic variables | | notification_threshold_text | string | template | Customizable threshold alert template message with dynamic variables | | notification_text | string | template | Customizable notification template message for data write triggers with dynamic variables | | notification_path | string | “notify” | Notification endpoint path with retry logic and exponential backoff | | port_override | number | 8181 | InfluxDB port override for notification delivery | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | --- # Valuecounter plugin Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/valuecounter/ The Value Counter Plugin counts how many times each unique value of one or more watched fields appears in a source table, then writes the result as a rollup measurement with field keys of the form `_=i`. It is a direct port of the [Telegraf `valuecounter` aggregator](https://github.com/influxdata/telegraf/tree/master/plugins/aggregators/valuecounter) to the InfluxDB 3 Enterprise Processing Engine. The plugin ships with two independent modes — a data-write (WAL) trigger that accumulates counts as rows arrive, and a scheduled trigger that queries the source table on every fire and aggregates the window since the previous successful fire. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. Some plugins support TOML configuration files, which can be specified using the plugin’s `config_file_path` parameter. If a plugin supports multiple trigger specifications, some parameters may depend on the trigger specification that you use. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters |Parameter| Type | Default | Description | |---------|------|-------------------------|--------------------------------------------------------------------------------------------------| |`fields` |string| required |Space-separated list of field names whose unique values are counted (for example, `status method`)| | `table` |string|required (scheduled only)|Source table to query. Mode A derives the source table from the trigger spec and rejects this key | ### Data write trigger parameters | Parameter | Type | Default | Description | |----------------|-------|----------------|--------------------------------------------------------------------------------------------------------------------------------| |`period_seconds`|integer| `60` | Emission period in seconds, at least `1`. Cache TTL is set to `2 * period_seconds` | | `period` |string | `60s` |Emission period as a duration, at least `1s`. Units: `s`, `min`, `h`, `d`, `w`. Overridden by `period_seconds` when both are set| |`output_suffix` |string | `_valuecounts` | Suffix appended to the source measurement name for rollup output. Must be non-empty | |`dest_database` |string |trigger’s own DB| Optional database to write rollups to via `write_sync_to_db` | ### Scheduled trigger parameters | Parameter | Type | Default | Description | |---------------|------|----------------|-----------------------------------------------------------------------------------| |`output_suffix`|string| `_valuecounts` |Suffix appended to the source measurement name for rollup output. Must be non-empty| |`dest_database`|string|trigger’s own DB| Optional database to write rollups to via `write_sync_to_db` | Mode B is drift-based: the trigger’s `every:` spec is the only cadence knob. The plugin queries the window between the previous successful fire and the current `call_time`, so it self-aligns to whatever the scheduler delivers. Mode B rejects `period_seconds` and `period` keys. ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Relative paths are resolved against `PLUGIN_DIR`, then `INFLUXDB3_PLUGIN_DIR`, then the parent of `VIRTUAL_ENV`. If `config_file_path` is set, no other inline arguments may be set on the trigger. The TOML accepts the same keys as inline arguments. #### Example TOML configuration * [valuecounter\_config.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/valuecounter/valuecounter_config.toml) — shows both Mode A (WAL) and Mode B (scheduled) shapes For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ## Schema requirement The plugin reads tag column names from `information_schema.columns` for the source table; the source measurement must exist before the trigger fires. The watched fields named in `fields` must be of categorical type (string or low-cardinality numeric). The rollup measurement is created on first emission; do not pre-create it with a conflicting schema. The rollup measurement name is always ``. The default suffix (`_valuecounts`) keeps the rollup separate from the source table so the WAL trigger does not feed itself. An empty `output_suffix` is rejected at trigger creation. ## Software Requirements * **InfluxDB 3 Enterprise**: with the Processing Engine enabled * **Python packages**: `influxdata-plugin-utils>=0.3.0` ## Installation steps 1. Start InfluxDB 3 Enterprise with the Processing Engine enabled (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. Install required Python packages: ``` influxdb3 install package "influxdata-plugin-utils>=0.3.0" ``` ## Trigger setup > > > **Do not install both Mode A and Mode B triggers against the same source table.** The two modes share no state and will write duplicate rollup rows. Choose one mode per source table. > > ### Data write trigger (Mode A) Count unique field values as rows are written. Emission is period-gated by the plugin’s own bookkeeping — the trigger emits when at least `period_seconds` have elapsed since the last successful emission for a given series. ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/valuecounter/valuecounter.py" \ --trigger-spec "table:http_requests" \ --trigger-arguments "fields=status method,period_seconds=60" \ --error-behavior log \ http_status_counter ``` Mode A’s `--trigger-spec` must name a single table (`table:`). `all_tables` is not supported because one `fields` list cannot apply to heterogeneous tables. ### Scheduled trigger (Mode B) Query the source table on a fixed cadence and aggregate the window since the previous fire. Best when writes are bursty or sparse, or when a deterministic emission cadence matters more than per-write freshness. ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/valuecounter/valuecounter.py" \ --trigger-spec "every:60s" \ --trigger-arguments "table=http_requests,fields=status method" \ --error-behavior log \ http_status_counter_scheduled ``` The first fire after install (or after a server restart that wipes the cache) only establishes the cadence anchor and emits no rollup. The second fire emits a rollup covering the window from the first fire to the second. ### Enable triggers ```bash influxdb3 enable trigger --database mydb http_status_counter ``` ## Example usage ### Example 1: HTTP status code rollup (Mode A) Roll up HTTP status codes to per-minute counts per host and endpoint: ```bash # Create and enable the trigger influxdb3 create trigger \ --database web \ --path "gh:influxdata/valuecounter/valuecounter.py" \ --trigger-spec "table:http_requests" \ --trigger-arguments "fields=status,period_seconds=60" \ --error-behavior log \ http_status_counter influxdb3 enable trigger --database web http_status_counter # Write some sample requests influxdb3 write \ --database web \ "http_requests,host=web-1,endpoint=/api/users status=200i http_requests,host=web-1,endpoint=/api/users status=200i http_requests,host=web-1,endpoint=/api/users status=404i http_requests,host=web-1,endpoint=/api/users status=500i" # After period_seconds elapses, write one more row to drive emission sleep 60 influxdb3 write \ --database web \ "http_requests,host=web-1,endpoint=/api/users status=200i" # Query the rollup influxdb3 query \ --database web \ "SELECT * FROM http_requests_valuecounts WHERE time >= now() - 5m" ``` **Expected output** ```text http_requests_valuecounts,host=web-1,endpoint=/api/users status_200=2i,status_404=1i,status_500=1i ``` ### Example 2: Log level counts (Mode B) Roll up application log levels per service over a fixed five-minute cadence: ```bash influxdb3 create trigger \ --database observability \ --path "gh:influxdata/valuecounter/valuecounter.py" \ --trigger-spec "every:5m" \ --trigger-arguments "table=app_logs,fields=level" \ --error-behavior log \ log_level_counter influxdb3 enable trigger --database observability log_level_counter ``` Each fire produces one rollup row per `(service, env)` tag combination, with fields like `level_INFO=120i`, `level_WARN=8i`, `level_ERROR=2i`. ### Example 3: Cross-database rollup (Mode B) Write rollups to a separate analytics database while leaving the source database untouched: ```bash influxdb3 create trigger \ --database production \ --path "gh:influxdata/valuecounter/valuecounter.py" \ --trigger-spec "every:1m" \ --trigger-arguments "table=payment_attempts,fields=result,dest_database=analytics" \ --error-behavior log \ payment_result_rollup ``` The rollup measurement `payment_attempts_valuecounts` is written to the `analytics` database. ## Code overview ### Files * `valuecounter.py`: The main plugin code containing `process_writes` (Mode A) and `process_scheduled_call` (Mode B) * `valuecounter_config.toml`: Example TOML configuration with both Mode A and Mode B shapes * `test_valuecounter.py`: Pytest suite (79 tests, runs without a live InfluxDB 3 Enterprise server) * `requirements.txt`: Runtime dependencies (`influxdata-plugin-utils>=0.3.0`) * `requirements-dev.txt`: Development dependencies (`pytest`) ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'your_trigger_name'" ``` Every log line is prefixed with a per-fire `task_id` (eight hex characters) so log records from a single trigger fire can be correlated. ### Main functions #### `process_writes(influxdb3_local, table_batches, args)` Handles Mode A WAL-trigger invocations. Accumulates per-value counts into trigger-local cache entries, period-gates emission per series, and writes one rollup row per series via `write_sync` (or `write_sync_to_db` if `dest_database` is set). Uses a snapshot-subtract pattern on emit so concurrent increments arriving during a write are preserved across emissions. #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles Mode B scheduled invocations. On the first fire, only the cadence anchor is established (no rollup emitted). On subsequent fires, the plugin queries the source table for the window since the previous successful fire, groups results by tag set and watched field, and writes one rollup row per series. The anchor advances only on a successful write or a genuinely empty window — write failures cause the next fire to cover a larger window and naturally retry. ## Troubleshooting ### Common issues #### Issue: No rollup rows appear **Solution (Mode A)**: Mode A emission is tied to write traffic. If no rows are written for at least `period_seconds`, emission stops. Confirm the source table is receiving writes and that `period_seconds` is shorter than the inter-write interval. Check `system.processing_engine_logs` for the trigger. **Solution (Mode B)**: The first fire after install (or server restart) establishes the cadence anchor and emits no rollup — look for a log line containing `vc-scheduled: first fire`. Subsequent fires emit normally. #### Issue: Rollup measurement schema explosion **Solution**: The plugin creates one rollup field per unique value of each watched field. Configuring `fields` against a high-cardinality column (for example, `user_id`) creates one column per distinct value and stresses the Parquet schema. Use only on bounded-cardinality categorical fields such as HTTP status, log level, or A/B variant. #### Issue: Field key contains an invalid character **Solution**: Watched-field values that contain characters other than `[A-Za-z0-9_]` are sanitized — every disallowed character is replaced with `_`. Two distinct raw values that sanitize to the same key (for example, `"not found"` and `"not_found"`) are summed into one rollup field. When this happens, the plugin emits a warning to `system.processing_engine_logs` listing the colliding raw values. #### Issue: Source table tag list changed and rollups are missing tags **Solution**: The plugin caches tag-column names from `information_schema.columns` for one hour per source table. Schema changes are picked up within an hour. To force a refresh, delete the cache key `shared:tags:
`. #### Issue: Both modes installed against the same source table **Solution**: The two modes share no state and will both emit rollups, producing duplicate rows. Drop one trigger and use the other exclusively. ### Operational notes * Do not enable `--run-asynchronous` on a Mode A trigger. The plugin assumes serialized per-trigger invocations; concurrent fires race on the per-series counts dict and the in-memory index. * Re-running a fire (for example, via `influxdb3 test schedule_plugin`) overwrites the previous emission idempotently because every rollup row uses the same timestamp for a given fire (Mode A: `now_ns`; Mode B: `call_time_ns`). Combined with InfluxDB 3 Enterprise’s last-write-wins semantics, replays do not produce duplicate rollup rows. * Server restart wipes the trigger-local cache. Mode A loses any accumulated-but-not-yet-emitted counts; Mode B’s next fire is treated as a “first fire” and skips one period of source data. Both behaviors are documented above. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Valuecounter plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/valuecounter) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[official](/influxdb3/enterprise/tags/official/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | fields | string | required | Space-separated list of field names whose unique values are counted (for example, status method ) | | table | string | required (scheduled only) | Source table to query. Mode A derives the source table from the trigger spec and rejects this key | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | period_seconds | integer | 60 | Emission period in seconds, at least 1 . Cache TTL is set to 2 * period_seconds | | period | string | 60s | Emission period as a duration, at least 1s . Units: s , min , h , d , w . Overridden by period_seconds when both are set | | output_suffix | string | _valuecounts | Suffix appended to the source measurement name for rollup output. Must be non-empty | | dest_database | string | trigger’s own DB | Optional database to write rollups to via write_sync_to_db | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | output_suffix | string | _valuecounts | Suffix appended to the source measurement name for rollup output. Must be non-empty | | dest_database | string | trigger’s own DB | Optional database to write rollups to via write_sync_to_db | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) | --- # Processing engine Python API reference Source: https://docs.influxdata.com/influxdb3/enterprise/plugins/python-api-reference/ The InfluxDB 3 processing engine provides a Python API that plugins use at runtime. Nothing in this API is importable from a plugin—the runtime injects the objects instead. InfluxDB passes `influxdb3_local` (an `InfluxDB3Local` instance) as the first argument to every trigger entry point and installs `LineBuilder` into Python builtins before the plugin runs, so both single-file and multi-file plugins can use them without an import. * [Trigger entry points](#trigger-entry-points) * [Log messages](#log-messages) * [Query data](#query-data) * [Write data](#write-data) * [Cache data](#cache-data) * [Plugin cancellation](#plugin-cancellation) * [LineBuilder](#linebuilder) * [TableBatch](#tablebatch) * [Exceptions](#exceptions) For usage patterns and examples, see [Extend plugins](/influxdb3/enterprise/extend-plugin/). ## Trigger entry points A plugin defines one entry point that matches the trigger type it is attached to. You define the function; InfluxDB calls it when the trigger fires. Multi-file plugins must define the entry point in the plugin directory’s `__init__.py`. | Trigger type | Trigger specification | Entry point function | |----------------|-------------------------------------|-----------------------------------------------------| |Data write (WAL)| `table:TABLE_NAME` or `all_tables` | [`process_writes()`](#process_writes) | | Scheduled |`every:DURATION` or `cron:EXPRESSION`|[`process_scheduled_call()`](#process_scheduled_call)| | HTTP request | `request:REQUEST_PATH` | [`process_request()`](#process_request) | ### `process_writes()` ```python def process_writes( influxdb3_local, table_batches: Sequence[TableBatch], args: Mapping[str, str] | None = None, ) -> None ``` Entry point for data write (WAL) triggers. InfluxDB calls the function with the rows that were just written to the database, grouped by table. * `table_batches`: the written rows as a sequence of [`TableBatch`](#tablebatch) dictionaries. * `args`: trigger arguments as a string-to-string mapping. ### `process_scheduled_call()` ```python def process_scheduled_call( influxdb3_local, schedule_time: datetime, args: Mapping[str, str] | None = None, ) -> None ``` Entry point for scheduled triggers. * `schedule_time`: the trigger’s fire time as a naive `datetime`. The engine builds the value with `datetime.fromtimestamp()`, so it is expressed in the server’s local timezone and truncated to whole seconds. * `args`: trigger arguments as a string-to-string mapping. ### `process_request()` ```python def process_request( influxdb3_local, query_params: Mapping[str, str], request_headers: Mapping[str, str], request_body: bytes, args: Mapping[str, str] | None = None, ) ``` Entry point for HTTP request triggers. * `query_params`: URL query parameters as a string-to-string mapping. * `request_headers`: request headers as a string-to-string mapping. Header names are lowercased, and the client’s `Authorization` header is stripped before the plugin sees it. * `request_body`: the raw request bytes. * `args`: trigger arguments as a string-to-string mapping. #### Return value `process_request` returns an HTTP-style response in one of the following forms: * **A Flask-style response object**—duck-typed, not an actual `flask.Response`. The object must define a `__flask_response__()` method that returns `True`, a `status_code` attribute, a `headers` mapping, and a `get_data()` method that returns `str`. A genuine `flask.Response` is rejected: it does not define `__flask_response__`, and its `get_data()` returns bytes. * **A tuple** of `(body, status, headers)` following Flask conventions.`status` and `headers` are optional; `headers` may be the second element only when `status` is omitted. * **A bare string, dictionary, list, or iterator**, rendered with defaults: dictionaries and lists are JSON-encoded with `Content-Type: application/json`; strings and other iterables are concatenated with a `text/html` default content type. The status code defaults to `200` when not provided. Any other return value—`None`, or an object that is not a string, dictionary, list, or iterable—fails the request with status `500` and the error `Unsupported return type from Python function`. Returning `bytes` is accepted as an iterable but yields an empty body, because its items are integers; encode to `str` instead. ## Log messages ```python influxdb3_local.info(*args: object) -> None influxdb3_local.warn(*args: object) -> None influxdb3_local.error(*args: object) -> None ``` Each method converts its arguments to strings, joins them with spaces, and logs the result at the named level (`INFO`, `WARN`, or `ERROR`). All three levels are stored in the `system.processing_engine_logs` table, where you can [query them using SQL](/influxdb3/enterprise/admin/query-system-data/#query-trigger-logs), and appear in `influxdb3 test` output. ## Query data ```python influxdb3_local.query( query: str, args: Mapping[str, str] | None = None, database: str | None = None, ) -> list[dict[str, Any]] ``` Executes a SQL query and returns a list of row dictionaries. `args` supplies values for named SQL parameters, referenced in the query with a leading `$`: ```python influxdb3_local.query( "SELECT host, usage FROM cpu WHERE host = $host", {"host": "host1"}, ) ``` Parameter values must be strings; passing an `int` or `float` raises `TypeError`. `database` selects which database to query and defaults to the database the trigger is attached to. In a cluster, a node that serves queries runs the query itself; a node that does not forwards it to a querier node over Arrow Flight. Each row dictionary contains every column in the result schema: * Column values are native Python `int`, `float`, `bool`, or `str` values. * Timestamp columns are nanosecond timestamps represented as integers. * Dictionary-encoded columns (tags) are materialized as their string labels. * SQL `NULL` values are returned as `None`. Columns of any other Arrow type raise `ValueError`. Supported types are `Int64`, `UInt64`, `Float64`, `Boolean`, `Utf8`, `LargeUtf8`, `Timestamp` (nanosecond), and `Dictionary` of `Utf8`. Raises [`QueryError`](#exceptions) on execution failures, invalid SQL, or an invalid database name. ## Write data All write methods take a [`LineBuilder`](#linebuilder) line and raise an exception when the target database is `_internal`. `write` and `write_to_db` queue writes; queued writes are flushed once the plugin returns, so they land after any `write_sync` or `write_sync_to_db` calls the plugin made.`write_sync` and `write_sync_to_db` write synchronously through the write buffer while the plugin is still running. ### `write` ```python influxdb3_local.write(line: LineBuilder) -> None ``` Queues a line protocol write to the current database. ### `write_to_db` ```python influxdb3_local.write_to_db(db_name: str, line: LineBuilder) -> None ``` Queues a line protocol write to the named database. ### `write_sync` ```python influxdb3_local.write_sync(line: LineBuilder, no_sync: bool) -> None ``` Writes line protocol to the current database synchronously.`no_sync` is required; pass `no_sync=True` to skip waiting for WAL synchronization. Use [`write_sync_to_db`](#write_sync_to_db) to target a different database. ### `write_sync_to_db` ```python influxdb3_local.write_sync_to_db( db_name: str, line: LineBuilder, no_sync: bool, ) -> None ``` Writes line protocol to the named database synchronously.`no_sync` is required; pass `no_sync=True` to skip waiting for WAL synchronization. Use [`write_sync`](#write_sync) to write to the trigger’s database. ### Writes to explicit schema databases In a database that uses [`explicit` schema mode](/influxdb3/enterprise/admin/databases/enforce-schema/), the server rejects a plugin write to an undeclared table or column with the same explicit-schema error message it returns to HTTP writes. * `write_sync` and `write_sync_to_db` raise an exception when the write is rejected. * `write` and `write_to_db` queue the write, so the failure surfaces after the plugin run. The server logs an `ERROR` entry in `system.processing_engine_logs`, for example: ``` error running schedule plugin: error writing back lines ``` For a schedule trigger, the trigger keeps running and the next run executes. If your plugin needs to know that a write failed, use `write_sync` and check `system.processing_engine_logs`. ## Cache data ```python influxdb3_local.cache: Cache ``` The `influxdb3_local.cache` property provides an in-memory cache that persists values between plugin executions. The trigger-local cache (the default) is scoped to the database and trigger pair; the global cache is shared by every trigger in the process. Set `use_global=True` in cache calls to use the global cache. The server clears both caches on restart. Reading the `cache` property runs an expiry sweep, though the sweep only does work once per cleanup interval. Individual expired entries are dropped on read regardless. ### `cache.put()` ```python influxdb3_local.cache.put( key: str, value: Any, ttl: float | None = None, use_global: bool | None = None, ) -> None ``` Stores a Python object in the trigger-local cache or the global shared cache. `ttl` controls expiry in seconds. Values in test caches (`influxdb3 test`) default to a 30-minute TTL when not provided; production caches persist values indefinitely unless a `ttl` is set. Raises `ValueError` when `ttl` is negative, `NaN`, or too large to represent as a duration. ### `cache.get()` ```python influxdb3_local.cache.get( key: str, default: Any | None = None, use_global: bool | None = None, ) -> Any | None ``` Fetches a value from the trigger-local or global cache. Expired entries are evicted on read. Returns the stored value, the provided default, or `None` when the key is absent. ### `cache.delete()` ```python influxdb3_local.cache.delete( key: str, use_global: bool | None = None, ) -> bool ``` Removes a cached value. Returns `True` when the key existed in the selected cache. ## Plugin cancellation Once the current plugin run has been cancelled—the server is shutting down, or the trigger was disabled or deleted—the logging, write, and query methods raise `KeyboardInterrupt`. `KeyboardInterrupt` subclasses `BaseException` rather than `Exception`, so a plugin’s `except Exception` handler does not swallow it, and a long-running loop unwinds instead of hanging the shutdown or the disable. The message reads `influxdb3 is shutting down; aborting plugin execution` for every cancellation cause, including a plain trigger disable. The `cache` property and the cache methods do not check for cancellation and keep working. ## LineBuilder `LineBuilder` constructs InfluxDB line protocol in plugins. It handles line protocol escaping and nanosecond timestamps, and plugins can use it without an import. Construct a builder with `LineBuilder(measurement)`, then call the tag, field, and timestamp methods on the resulting instance. In this section, `line` is a `LineBuilder` instance. Tag and field methods return the same `LineBuilder` instance, so you can chain calls: ```python line = LineBuilder("weather") line.tag("location", "us-midwest").float64_field("temperature", 82.5) influxdb3_local.write(line) ``` Raises `InvalidMeasurementError` when the measurement name contains spaces. Tag and field keys that are empty or contain spaces, commas, or equals signs raise `InvalidKeyError`. | Method | Adds | |---------------------------|----------------------------------------------------------------| | `tag(key, value)` | A tag; the value is stringified | | `int64_field(key, value)` | A signed integer field | |`uint64_field(key, value)` | An unsigned integer field; negative values raise `ValueError` | |`float64_field(key, value)`|A float field; integral values are rendered with a trailing `.0`| |`string_field(key, value)` | A string field with quotes and backslashes escaped | | `bool_field(key, value)` | A boolean field, rendered as `t` or `f` | | `time_ns(timestamp_ns)` | The nanosecond timestamp for the line | | `build()` | Renders the line as line protocol | ### `line.build()` ```python line.build() -> str ``` When you call `build()` on a `LineBuilder` instance, it renders the accumulated measurement, tags, fields, and optional timestamp as a line protocol string.`build()` raises `InvalidLineError` when called without any fields. For format details, see the [line protocol reference](/influxdb3/enterprise/reference/line-protocol/). ## TableBatch A batch of WAL rows for a single table, delivered to [`process_writes`](#process_writes). A `TableBatch` is a plain dictionary—read it with `table_batch["table_name"]` and `table_batch["rows"]`, not attribute access. | Key | Type | Description | |------------|------------------------------------|------------------------------| |`table_name`| `str` |The table the batch belongs to| | `rows` |`Sequence[MutableMapping[str, Any]]`| The written rows | Each row is itself a dictionary that holds every column in the table schema (tags, fields, and time) keyed by column name. Columns with no value for that row are `None`. Time columns are nanosecond integers. ## Exceptions | Exception | Raised | |-------------------------|----------------------------------------------------------------------------| | `InfluxDBError` | Base exception for `LineBuilder` errors | |`InvalidMeasurementError`| When the measurement name contains spaces | | `InvalidKeyError` |When a tag or field key is empty or contains spaces, commas, or equals signs| | `InvalidLineError` | When building line protocol fails—for example, no fields were added | | `QueryError` | By [`query`](#query-data) when a query fails or names an invalid database | ### Catching QueryError The `QueryError` class is defined by the engine’s native extension module, which a plugin cannot import, and the name is not injected into plugin globals or builtins.`except QueryError` therefore raises `NameError`. Catch `Exception` instead and inspect `type(err).__name__` to distinguish it: ```python try: results = influxdb3_local.query("SELECT * FROM cpu") except Exception as err: if type(err).__name__ == "QueryError": influxdb3_local.error("query failed:", err) ``` #### Related * [Extend plugins with API features and state management](/influxdb3/enterprise/plugins/extend-plugin/) * [influxdb3 create trigger](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/) * [influxdb3 test](/influxdb3/enterprise/reference/cli/influxdb3/test/) * [Processing engine reference](/influxdb3/enterprise/reference/processing-engine/) [processing engine](/influxdb3/enterprise/tags/processing-engine/)[plugins](/influxdb3/enterprise/tags/plugins/)[API](/influxdb3/enterprise/tags/api/)[python](/influxdb3/enterprise/tags/python/) | Trigger type | Trigger specification | Entry point function | | --- | --- | --- | | Trigger type | Trigger specification | Entry point function | | Data write (WAL) | table:TABLE_NAME or all_tables | process_writes() | | Scheduled | every:DURATION or cron:EXPRESSION | process_scheduled_call() | | HTTP request | request:REQUEST_PATH | process_request() | | Method | Adds | | --- | --- | | Method | Adds | | tag(key, value) | A tag; the value is stringified | | int64_field(key, value) | A signed integer field | | uint64_field(key, value) | An unsigned integer field; negative values raise ValueError | | float64_field(key, value) | A float field; integral values are rendered with a trailing .0 | | string_field(key, value) | A string field with quotes and backslashes escaped | | bool_field(key, value) | A boolean field, rendered as t or f | | time_ns(timestamp_ns) | The nanosecond timestamp for the line | | build() | Renders the line as line protocol | | Key | Type | Description | | --- | --- | --- | | Key | Type | Description | | table_name | str | The table the batch belongs to | | rows | Sequence[MutableMapping[str, Any]] | The written rows | | Exception | Raised | | --- | --- | | Exception | Raised | | InfluxDBError | Base exception for LineBuilder errors | | InvalidMeasurementError | When the measurement name contains spaces | | InvalidKeyError | When a tag or field key is empty or contains spaces, commas, or equals signs | | InvalidLineError | When building line protocol fails—for example, no fields were added | | QueryError | By query when a query fails or names an invalid database | --- # Query data in InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/ Learn to query data in InfluxDB 3 Enterprise. ### [Execute queries](/influxdb3/enterprise/query-data/execute-queries/) Use tools and libraries to query data from InfluxDB 3 Enterprise. ### [Query data with SQL](/influxdb3/enterprise/query-data/sql/) Learn to query data in InfluxDB 3 Enterprise using SQL. ### [Query data with InfluxQL](/influxdb3/enterprise/query-data/influxql/) Learn to use InfluxQL to query data in InfluxDB 3 Enterprise. [query](/influxdb3/enterprise/tags/query/) --- # Execute queries Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/execute-queries/ Use tools and libraries to query data from an InfluxDB 3 Enterprise database. InfluxDB client libraries and Flight clients can use the Flight+gRPC protocol to query with SQL or InfluxQL and retrieve data in the[Arrow in-memory format](https://arrow.apache.org/docs/format/Columnar.html). HTTP clients can use the InfluxDB v1 `/query` REST API to query with InfluxQL and retrieve data in JSON format. Learn how to connect to InfluxDB and query your data using the following tools: ## [Use the v3 query API](/influxdb3/enterprise/query-data/execute-queries/influxdb-v3-api/) Use SQL or InfluxQL and the InfluxDB v3 HTTP query API to query data in InfluxDB 3 Enterprise. ```sh curl --get http://localhost:8181/api/v3/query_sql \ --header "Authorization: Token DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` [Read more ](/influxdb3/enterprise/query-data/execute-queries/influxdb-v3-api/) ## [Use the influxdb3 CLI](/influxdb3/enterprise/query-data/execute-queries/influxdb3-cli/) Use the `influxdb3 query` command to query data in InfluxDB 3 Enterprise with SQL. ```sh influxdb3 query \ --database DATABASE_NAME \ "SELECT * FROM home" ``` [Read more ](/influxdb3/enterprise/query-data/execute-queries/influxdb3-cli/) ## [Use the v1 query API and InfluxQL](/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/) Use the InfluxDB v1 HTTP query API to query data in InfluxDB 3 Enterprise with InfluxQL. ```sh curl --get http://localhost:8181/query \ --header "Authorization: Token DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` [Read more ](/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/) [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/)[influxql](/influxdb3/enterprise/tags/influxql/) --- # Use the InfluxDB v1 HTTP query API and InfluxQL to query data Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/execute-queries/influxdb-v1-api/ * InfluxQL Use the InfluxDB v1 HTTP query API to query data in InfluxDB 3 Enterprise with InfluxQL. The examples below use **cURL** to send HTTP requests to the InfluxDB v1 HTTP API, but you can use any HTTP client. > [!Warning] > #### InfluxQL feature support > > InfluxQL is being rearchitected to work with the InfluxDB 3 storage engine. > This process is ongoing and some InfluxQL features are still being implemented. > For information about the current implementation status of InfluxQL features, > see [InfluxQL feature support](/influxdb3/enterprise/reference/influxql/feature-support/). Use the v1 `/query` endpoint and the `GET` request method to query data with InfluxQL: ``` GET http://localhost:8181/query ``` ## Authenticate API requests InfluxDB 3 Enterprise requires each API request to be authenticated with a[token](/influxdb3/enterprise/admin/tokens/). With InfluxDB v1-compatible endpoints in InfluxDB 3, you can use database tokens in InfluxDB 1.x username and password schemes, in the InfluxDB v2 `Authorization: Token` scheme, or in the OAuth `Authorization: Bearer` scheme. * [Authenticate with a username and password scheme](#authenticate-with-a-username-and-password-scheme) * [Authenticate with a token scheme](#authenticate-with-a-token-scheme) ### Authenticate with a username and password scheme With InfluxDB v1-compatible endpoints in InfluxDB 3, you can use the InfluxDB 1.x convention of username and password to authenticate database reads by passing a [token](/influxdb3/enterprise/admin/tokens/)with read access to the database as the `password` credential. When authenticating requests to the v1 API `/query` endpoint, InfluxDB 3 Enterprise checks that the `password` (`p`) value is an authorized [token](/influxdb3/enterprise/admin/tokens/)with read access to the database . InfluxDB 3 Enterprise ignores the `username` (`u`) parameter in the request. Use one of the following authentication schemes with clients that support Basic authentication or query parameters: * [Basic authentication](#basic-authentication) * [Query string authentication](#query-string-authentication) #### Basic authentication Use the `Authorization` header with the `Basic` scheme to authenticate v1 API `/query` requests. When authenticating requests, InfluxDB 3 Enterprise checks that the `password` part of the decoded credential is an authorized [token](/influxdb3/enterprise/admin/tokens/)with read access to the database . InfluxDB 3 Enterprise ignores the `username` part of the decoded credential. ##### Syntax ```http Authorization: Basic ``` Encode the `[USERNAME]:DATABASE_TOKEN` credential using base64 encoding, and then append the encoded string to the `Authorization: Basic` header. ##### Example The following example shows how to use cURL with the `Basic` authentication scheme: ```sh curl --get "http://localhost:8181/query" \ --user "any:DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` #### Query string authentication In the URL, pass the `p` query parameter to authenticate `/query` requests. When authenticating requests, InfluxDB 3 Enterprise checks that the `p` (*password*) value is an authorized [token](/influxdb3/enterprise/admin/tokens/)with read access to the database and ignores the `u` (*username*) parameter. ##### Syntax ```sh http://localhost:8181/query/?u=any&p=DATABASE_TOKEN ``` ##### Example The following example shows how to use cURL with query string authentication: ```sh curl --get "http://localhost:8181/query" \ --data-urlencode "p=DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` ### Authenticate with a token scheme Use the `Authorization: Bearer` or the `Authorization: Token` scheme to pass a [token](/influxdb3/enterprise/admin/tokens/)with read access to the database for authenticating v1 API `/query` requests. `Bearer` and `Token` are equivalent in InfluxDB 3 Enterprise. The `Token` scheme is used in the InfluxDB 2.x API.`Bearer` is defined by the [OAuth 2.0 Framework](https://www.rfc-editor.org/rfc/rfc6750#page-14). Support for one or the other may vary across InfluxDB API clients. #### Syntax ```http Authorization: Bearer DATABASE_TOKEN ``` ```http Authorization: Token DATABASE_TOKEN ``` #### Examples Use `Bearer` to authenticate a query request: ```sh curl --get "http://localhost:8181/query" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` Use `Token` to authenticate a query request: ```sh curl --get "http://localhost:8181/query" \ --header "Authorization: Token DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` ## Query parameters For InfluxDB 3 Enterprise v1 API `/query` requests, set parameters as listed in the following table: | Parameter | Allowed in |Ignored| Value | |--------------------------------|------------|-------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `chunked` |Query string|Honored|Returns points in streamed batches instead of in a single response. If set to `true`, InfluxDB chunks responses by series or by every 10,000 points, whichever occurs first.| | `chunked_size` |Query string|Honored| **Requires `chunked` to be set to `true`**. If set to a specific value, InfluxDB chunks responses by series or by this number of points. | | `db` \* |Query string|Honored| Database name | | `epoch` |Query string|Honored| [Timestamp precision](#timestamp-precision) | | `p` |Query string|Honored| For [query string authentication](#query-string-authentication), a [token](/influxdb3/enterprise/admin/tokens/) | |with read access to the database| | | | `pretty` | Query string | Ignored | N/A`q` \* | Query string | Honored | URL-encoded InfluxQL query`rp` | Query string | Honored, but discouraged | Retention policy`u` | Query string | Ignored | For [query string authentication](#query-string-authentication), any arbitrary string`Authorization` | Header | Honored | `Bearer DATABASE_TOKEN`, `Token DATABASE_TOKEN`, or `Basic ` \* = Required ### Timestamp precision Use one of the following values for timestamp precision: * `ns`: nanoseconds * `us`: microseconds * `ms`: milliseconds * `s`: seconds * `m`: minutes * `h`: hours Replace the following configuration values: * `DATABASE_NAME`: the name of the [database](/influxdb3/enterprise/admin/databases/) to query * `DATABASE_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/)with read access to the database ## Return results as JSON or CSV By default, the `/query` endpoint returns results in **JSON**, but it can also return results in **CSV**. To return results as CSV, include the `Accept` header with the `application/csv` or `text/csv` MIME type: ```sh curl --get http://localhost:8181/query \ --header "Authorization: Bearer DATABASE_TOKEN" \ --header "Accept: application/csv" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SELECT * FROM home" ``` #### Related * [Use compatibility APIs and client libraries to write data](/influxdb3/enterprise/write-data/http-api/compatibility-apis/) [query](/influxdb3/enterprise/tags/query/)[influxql](/influxdb3/enterprise/tags/influxql/)[python](/influxdb3/enterprise/tags/python/) | Parameter | Allowed in | Ignored | Value | | --- | --- | --- | --- | | Parameter | Allowed in | Ignored | Value | | chunked | Query string | Honored | Returns points in streamed batches instead of in a single response. If set to true , InfluxDB chunks responses by series or by every 10,000 points, whichever occurs first. | | chunked_size | Query string | Honored | Requires chunked to be set to true . If set to a specific value, InfluxDB chunks responses by series or by this number of points. | | db * | Query string | Honored | Database name | | epoch | Query string | Honored | Timestamp precision | | p | Query string | Honored | For query string authentication , a token | | with read access to the database | | | | --- # Use the InfluxDB v3 HTTP query API Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/execute-queries/influxdb-v3-api/ * InfluxQL * SQL Use the InfluxDB 3 HTTP query API to query data in InfluxDB 3 Enterprise. The API provides `GET` and `POST` endpoints for querying data and system information using SQL or InfluxQL. > [!Note] > #### Query using gRPC or HTTP > > InfluxDB 3 supports HTTP and Flight (gRPC) query APIs. > For more information about using Flight, see the [InfluxDB 3 (`influxdb3-`) client libraries](https://github.com/InfluxCommunity/). The examples below use **cURL** to send HTTP requests to the InfluxDB 3 HTTP API, but you can use any HTTP client. * [Query using SQL and the HTTP API](#query-using-sql-and-the-http-api) * [Query using InfluxQL and the HTTP API](#query-using-influxql-and-the-http-api) ## Query using SQL and the HTTP API Use the `/api/v3/query_sql` endpoint with the `GET` or `POST` request methods. * `GET`: Pass parameters in the URL query string (for simple queries) * `POST`: Pass parameters in a JSON object (for complex queries and readability in your code) Include the following parameters: * `q`: *(Required)* The **SQL** query to execute. * `db`: *(Required)* The database to execute the query against. * `params`: A JSON object containing parameters to be used in a *parameterized query*. * `format`: The format of the response (`json`, `jsonl`, `csv`, `pretty`, or `parquet`). JSONL (`jsonl`) is preferred because it streams results back to the client.`pretty` is for human-readable output. Default is `json`. ### Example: Query passing URL-encoded parameters The following example sends an HTTP `GET` request with a URL-encoded SQL query: ```bash curl "http://localhost:8181/api/v3/query_sql?db=servers&q=select+*+from+cpu+limit+5" \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### Example: Query passing JSON parameters The following example sends an HTTP `POST` request with parameters in a JSON payload: ``` curl http://localhost:8181/api/v3/query_sql \ --header "Authorization: Bearer AUTH_TOKEN" --json '{"db": "server", "q": "select * from cpu limit 5"}' ``` ### Query system information Use the HTTP API `/api/v3/query_sql` endpoint to retrieve system information about your database server and table schemas in InfluxDB 3 Enterprise. #### Examples > [!Note] > #### system\_ sample data > > In examples, tables with `"table_name":"system_` are user-created tables for CPU, memory, disk, > network, and other resource statistics collected and written > by the user–for example, using the `psutil` Python library or[Telegraf](/telegraf/v1/get-started/) to collect > and write system metrics to an InfluxDB 3 database. ##### Show tables The following example sends a `GET` request that executes a `show tables` query to retrieve all user-created tables (`"table_schema":"iox"`), system tables, and information schema tables for a database: ```bash curl "http://localhost:8181/api/v3/query_sql?db=mydb&format=jsonl&q=show%20tables" \ --header "Authorization: Bearer AUTH_TOKEN" ``` The response body contains the following JSONL: ```jsonl {"table_catalog":"public","table_schema":"iox","table_name":"system_cpu","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_cpu_cores","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_memory","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_memory_faults","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_disk_usage","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_disk_io","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_disk_performance","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"iox","table_name":"system_network","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"distinct_caches","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"last_caches","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"parquet_files","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"processing_engine_plugins","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"processing_engine_triggers","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"system","table_name":"queries","table_type":"BASE TABLE"} {"table_catalog":"public","table_schema":"information_schema","table_name":"tables","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"views","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"columns","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"df_settings","table_type":"VIEW"} {"table_catalog":"public","table_schema":"information_schema","table_name":"schemata","table_type":"VIEW"} ``` A table has one of the following `table_schema` values: * `iox`: tables created by the user of the database. * `system`: tables used by the system to show information about the running database server. Some of these tables show stored information such as configurations, while others, such as the `queries` table, hold ephemeral state in memory. * `information_schema`: views that show schema information for tables in the database. #### View column information for a table The following query sends a `POST` request that executes an SQL query to retrieve information about columns in the sample `system_swap` table schema: *Note: when you send a query in JSON, you must escape single quotes that surround field names.* ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "mydb", "q": "SELECT * FROM information_schema.columns WHERE table_schema = '"'iox'"' AND table_name = '"'system_swap'"'", "format": "jsonl" }' ``` The output is similar to the following: ```jsonl {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"free","ordinal_position":0,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"host","ordinal_position":1,"is_nullable":"NO","data_type":"Dictionary(Int32, Utf8)"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"percent","ordinal_position":2,"is_nullable":"YES","data_type":"Float64","numeric_precision":24,"numeric_precision_radix":2} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"sin","ordinal_position":3,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"sout","ordinal_position":4,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"time","ordinal_position":5,"is_nullable":"NO","data_type":"Timestamp(Nanosecond, None)"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"total","ordinal_position":6,"is_nullable":"YES","data_type":"UInt64"} {"table_catalog":"public","table_schema":"iox","table_name":"system_swap","column_name":"used","ordinal_position":7,"is_nullable":"YES","data_type":"UInt64"} ``` #### Recently executed queries To view recently executed queries, query the `queries` system table: ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{ "db": "mydb", "q": "SELECT * FROM system.queries LIMIT 2", "format": "jsonl" }' ``` The output is similar to the following: ```jsonl {"id":"cdd63409-1822-4e65-8e3a-d274d553dbb3","phase":"success","issue_time":"2025-01-20T17:01:40.690067","query_type":"sql","query_text":"show tables","partitions":0,"parquet_files":0,"plan_duration":"PT0.032689S","permit_duration":"PT0.000202S","execute_duration":"PT0.000223S","end2end_duration":"PT0.033115S","compute_duration":"P0D","max_memory":0,"success":true,"running":false,"cancelled":false} {"id":"47f8d312-5e75-4db2-837a-6fcf94c09927","phase":"success","issue_time":"2025-01-20T17:02:32.627782","query_type":"sql","query_text":"show tables","partitions":0,"parquet_files":0,"plan_duration":"PT0.000583S","permit_duration":"PT0.000015S","execute_duration":"PT0.000063S","end2end_duration":"PT0.000662S","compute_duration":"P0D","max_memory":0,"success":true,"running":false,"cancelled":false} ``` ## Query using InfluxQL and the HTTP API Use the `/api/v3/query_influxql` endpoint with the `GET` or `POST` request methods. * `GET`: Pass parameters in the URL query string (for simple queries) * `POST`: Pass parameters in a JSON object (for complex queries and readability in your code) Include the following parameters: * `q`: *(Required)* The **InfluxQL** query to execute. * `db`: *(Required)* The database to execute the query against. * `params`: A JSON object containing parameters to be used in a *parameterized query*. * `format`: The format of the response (`json`, `jsonl`, `csv`, `pretty`, or `parquet`). JSONL (`jsonl`) is preferred because it streams results back to the client.`pretty` is for human-readable output. Default is `json`. ### Example: Query passing URL-encoded parameters The following example sends an HTTP `GET` request with a URL-encoded InfluxQL query: ```bash curl "http://localhost:8181/api/v3/query_influxql?db=servers&q=select+*+from+cpu+limit+5" \ --header "Authorization: Bearer AUTH_TOKEN" ``` ### Example: Query passing JSON parameters The following example sends an HTTP `POST` request with parameters in a JSON payload: ``` curl http://localhost:8181/api/v3/query_influxql \ --header "Authorization: Bearer AUTH_TOKEN" \ --json '{"db": "server", "q": "select * from cpu limit 5"}' ``` #### Related [query](/influxdb3/enterprise/tags/query/)[influxql](/influxdb3/enterprise/tags/influxql/)[sql](/influxdb3/enterprise/tags/sql/)[python](/influxdb3/enterprise/tags/python/) --- # Use the influxdb3 CLI to query data Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/execute-queries/influxdb3-cli/ Use the [`influxdb3 query` command](/influxdb3/enterprise/reference/cli/influxdb3/query/)to query data in InfluxDB 3 Enterprise with SQL or InfluxQL. Provide the following with your command: * **Authorization token**: Your InfluxDB 3 Enterprise [admin token](/influxdb3/enterprise/admin/tokens/admin) with read permissions on the database. Provide this using one of the following: * `--token` command option * `INFLUXDB3_AUTH_TOKEN` environment variable * **Database name**: The name of the database to query. Provide this using one of the following: * `-d`, `--database` command option * `INFLUXDB3_DATABASE_NAME` environment variable * **Query language** *(Optional)*: The query language of the query. Use the `-l`, `--language` option to specify one of the following query languages: * `sql` *(default)* * `influxql` * **Query**: SQL or InfluxQL query to execute. Provide the query in one of the following ways: * a string * the `--file` option and the path to a file that contains the query * from stdin [SQL](#sql)[InfluxQL](#influxql) [string](#string)[file](#file)[stdin](#stdin) ```bash influxdb3 query \ --token AUTH_TOKEN \ --database DATABASE_NAME \ "SELECT * FROM home" ``` ```bash influxdb3 query \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --file ./query.sql ``` ```bash cat ./query.sql | influxdb3 query --token AUTH_TOKEN --database DATABASE_NAME ``` [string](#string-2)[file](#file-2)[stdin](#stdin-2) ```bash influxdb3 query \ --token AUTH_TOKEN \ --language influxql \ --database DATABASE_NAME \ "SELECT * FROM home" ``` ```bash influxdb3 query \ --token AUTH_TOKEN \ --language influxql \ --file ./query.influxql ``` ```bash cat ./query.influxql | influxdb3 query \ --token AUTH_TOKEN \ --language influxql \ --database DATABASE_NAME ``` In the examples above and below, replace the following: * `DATABASE_NAME`: Name of the database to query ## Output format The `influxdb3 query` command supports the following output formats: * `pretty` *(default)* * `json` * `jsonl` * `csv` * `parquet` *(must [output to a file](#output-query-results-to-a-parquet-file))* Use the `--format` flag to specify the output format: ```sh influxdb3 query \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --format json \ "SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' LIMIT 5" ``` [](#view-example-pretty-formatted-results) View example pretty-formatted results ``` +----+------+-------------+------+---------------------+ | co | hum | room | temp | time | +----+------+-------------+------+---------------------+ | 0 | 35.9 | Living Room | 21.1 | 2022-01-01T08:00:00 | | 0 | 35.9 | Kitchen | 21.0 | 2022-01-01T08:00:00 | | 0 | 35.9 | Living Room | 21.4 | 2022-01-01T09:00:00 | | 0 | 36.2 | Kitchen | 23.0 | 2022-01-01T09:00:00 | | 0 | 36.0 | Living Room | 21.8 | 2022-01-01T10:00:00 | +----+------+-------------+------+---------------------+ ``` [](#view-example-json-formatted-results) View example JSON-formatted results ```json [{"co":0,"hum":35.9,"room":"Living Room","temp":21.1,"time":"2022-01-01T08:00:00"},{"co":0,"hum":35.9,"room":"Kitchen","temp":21.0,"time":"2022-01-01T08:00:00"},{"co":0,"hum":35.9,"room":"Living Room","temp":21.4,"time":"2022-01-01T09:00:00"},{"co":0,"hum":36.2,"room":"Kitchen","temp":23.0,"time":"2022-01-01T09:00:00"},{"co":0,"hum":36.0,"room":"Living Room","temp":21.8,"time":"2022-01-01T10:00:00"}] ``` [](#view-example-json-line-formatted-results) View example JSON-line-formatted results ```jsonl {"co":0,"hum":35.9,"room":"Living Room","temp":21.1,"time":"2022-01-01T08:00:00"} {"co":0,"hum":35.9,"room":"Kitchen","temp":21.0,"time":"2022-01-01T08:00:00"} {"co":0,"hum":35.9,"room":"Living Room","temp":21.4,"time":"2022-01-01T09:00:00"} {"co":0,"hum":36.2,"room":"Kitchen","temp":23.0,"time":"2022-01-01T09:00:00"} {"co":0,"hum":36.0,"room":"Living Room","temp":21.8,"time":"2022-01-01T10:00:00"} ``` [](#view-example-csv-formatted-results) View example CSV-formatted results ```csv co,hum,room,temp,time 0,35.9,Living Room,21.1,2022-01-01T08:00:00 0,35.9,Kitchen,21.0,2022-01-01T08:00:00 0,35.9,Living Room,21.4,2022-01-01T09:00:00 0,36.2,Kitchen,23.0,2022-01-01T09:00:00 0,36.0,Living Room,21.8,2022-01-01T10:00:00 ``` ## Output query results to a Parquet file To output query results to a Parquet file, provide the following options with the `influxdb3 query` command: * `--format`: `parquet` * `-o`, `--output`: the filepath to the Parquet file to store results in ```sh influxdb3 query \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --format parquet \ --output path/to/results.parquet \ "SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' LIMIT 5" ``` #### Related * [influxdb3 query](/influxdb3/enterprise/reference/cli/influxdb3/query/) * [SQL reference documentation](/influxdb3/enterprise/reference/sql/) * [InfluxQL reference documentation](/influxdb3/enterprise/reference/influxql/) [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/)[influxql](/influxdb3/enterprise/tags/influxql/)[influxdb3](/influxdb3/enterprise/tags/influxdb3/)[CLI](/influxdb3/enterprise/tags/cli/) --- # Query data with InfluxQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/influxql/ Learn to query data in InfluxDB 3 Enterprise using InfluxQL. * [Explore your schema with InfluxQL](#explore-your-schema-with-influxql) * [Perform a basic InfluxQL query](#perform-a-basic-influxql-query) * [Aggregate data with InfluxQL](#aggregate-data-with-influxql) * [Troubleshoot InfluxQL errors](#troubleshoot-influxql-errors) * [Use parameterized queries with InfluxQL](#use-parameterized-queries-with-influxql) ### [Explore your schema with InfluxQL](/influxdb3/enterprise/query-data/influxql/explore-schema/) Use InfluxQL `SHOW` statements to return information about your data schema. ##### List measurements ```sql SHOW MEASUREMENTS ``` ##### List field keys in a measurement ```sql SHOW FIELD KEYS FROM "measurement" ``` ##### List tag keys in a measurement ```sql SHOW TAG KEYS FROM "measurement" ``` ##### List tag values for a specific tag key ```sql SHOW TAG VALUES FROM "measurement" WITH KEY = "tag-key" WHERE time > now() - 1d ``` [Read more ](/influxdb3/enterprise/query-data/influxql/explore-schema/) ### [Perform a basic InfluxQL query](/influxdb3/enterprise/query-data/influxql/basic-query/) A basic InfluxQL query that queries data from InfluxDB most commonly includes `SELECT`, `FROM`, and `WHERE` clauses. ```sql SELECT temp, room FROM home WHERE time >= now() - 1d ``` [Read more ](/influxdb3/enterprise/query-data/influxql/basic-query/) ### [Aggregate data with InfluxQL](/influxdb3/enterprise/query-data/influxql/aggregate-select/) Use InfluxQL aggregate and selector functions to perform aggregate operations on your time series data. ##### Aggregate fields by groups ```sql SELECT MEAN(temp) AS mean, FIRST(hum) as first, FROM home GROUP BY tag ``` ##### Aggregate by time-based intervals ```sql SELECT MEAN(temp), sum(hum), FROM home WHERE time >= now() - 24h GROUP BY time(1h),room ``` [Read more ](/influxdb3/enterprise/query-data/influxql/aggregate-select/) ### [Troubleshoot InfluxQL errors](/influxdb3/enterprise/query-data/influxql/troubleshoot/) Learn how to troubleshoot and fix common InfluxQL errors. ### [Use parameterized queries with InfluxQL](/influxdb3/enterprise/query-data/influxql/parameterized-queries/) Use parameterized queries to prevent injection attacks and make queries more reusable. ```sql SELECT * FROM home WHERE time >= $min_time AND temp >= $min_temp AND room = $room ``` [Read more ](/influxdb3/enterprise/query-data/influxql/parameterized-queries/) [query](/influxdb3/enterprise/tags/query/)[influxql](/influxdb3/enterprise/tags/influxql/) --- # Aggregate data with InfluxQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/influxql/aggregate-select/ An InfluxQL query that aggregates data includes the following clauses: \* Required * \* `SELECT`: Specify fields and calculations to output from a measurement or use the wildcard alias (`*`) to select all fields and tags from a measurement. * \* `FROM`: Specify the measurement to query data from. * `WHERE`: Only retrieve data that meets the specified conditions–for example, time is in a time range, contains specific tag values, or contains a field value outside specified thresholds. * `GROUP BY`: Group data by tag values and time intervals. > [!Note] > For simplicity, the term *“aggregate”* in this guide refers to applying > both aggregate and selector functions to a dataset. Learn how to apply aggregate operations to your queried data: * [Aggregate and selector functions](#aggregate-and-selector-functions) * [Aggregate functions](#aggregate-functions) * [Selector functions](#selector-functions) * [Example aggregate queries](#example-aggregate-queries) > [!Note] > #### InfluxDB v1 to InfluxDB 3 data model > > InfluxQL was designed around the InfluxDB v1 data model, but can still be used > to query data from InfluxDB 3 Enterprise. When using the InfluxDB 3 Enterprise > InfluxQL implementation, the data model is different in the following ways: > > * an InfluxDB v1 **database and retention policy** combination is combined > into a single InfluxDB 3 **database** entity. > * an InfluxDB v1 **measurement** is equivalent to an InfluxDB 3 **table**. ## Aggregate and selector functions Both aggregate and selector functions return a limited number of rows from each group. Aggregate functions return a single row, whereas some selector functions let you specify the number of rows to return from each group. For example, if you `GROUP BY room` and perform an aggregate operation in your `SELECT` clause, results include an aggregate value for each unique value of `room`. ### Aggregate functions Use **aggregate functions** to aggregate values in a specified field for each group and return a single row per group containing the aggregate field value. [View InfluxQL aggregate functions](/influxdb3/enterprise/reference/influxql/functions/aggregates/) ##### Basic aggregate query ```sql SELECT MEAN(co) from home ``` ### Selector functions Use **selector functions** to “select” a value from a specified field. [View InfluxQL selector functions](/influxdb3/enterprise/reference/influxql/functions/selectors/) ##### Basic selector query ```sql SELECT TOP(co, 3) from home ``` ## Example aggregate queries * [Perform an ungrouped aggregation](#perform-an-ungrouped-aggregation) * [Group and aggregate data](#group-and-aggregate-data) * [Downsample data by applying interval-based aggregates](#downsample-data-by-applying-interval-based-aggregates) > [!Note] > #### Sample data > > The following examples use the [Home sensor data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-the-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. ### Perform an ungrouped aggregation To aggregate *all* queried values in a specified field: * Use aggregate or selector functions in your `SELECT` statement. * Do not include a `GROUP BY` clause to leave your data ungrouped. ```sql SELECT MEAN(co) AS "average co" FROM home ``` [](#view-example-results) View example results name: home |time| average co | |----|-----------------| | 0 |5.269230769230769| ### Group and aggregate data To apply aggregate or selector functions to grouped data: * Use aggregate or selector functions in your `SELECT` statement. * Include a `GROUP BY` clause with a comma-delimited list of tags to group by. Keep the following in mind when using `GROUP BY`: * `GROUP BY` can use column aliases that are defined in the `SELECT` clause. ```sql SELECT MEAN(temp) AS "average temp" FROM home GROUP BY room ``` [](#view-example-results) View example results name: home tags: room=Kitchen |time| average temp | |----|------------------| | 0 |22.623076923076926| name: home tags: room=Living Room |time| average temp | |----|-----------------| | 0 |22.16923076923077| #### Downsample data by applying interval-based aggregates A common use case when querying time series is downsampling data by applying aggregates to time-based groups. To group and aggregate data into time-based groups: * In your `SELECT` clause, apply [aggregate](/influxdb3/enterprise/reference/influxql/functions/aggregates/)or [selector](/influxdb3/enterprise/reference/influxql/functions/selectors/)functions to queried fields. * In your `WHERE` clause, include time bounds for the query. Interval-based aggregates produce a row for each specified time interval. If no time bounds are specified in the `WHERE` clause, the query uses the default time range (1970-01-01T00:00:00Z to now) and returns a row for each interval in that time range. * In your `GROUP BY` clause: * Use the [`time()` function](/influxdb3/enterprise/reference/influxql/functions/date-time/#time)to specify the time interval to group by. * *Optional*: Specify other tags to group by. The following example retrieves unique combinations of time intervals and rooms with their minimum, maximum, and average temperatures. ```sql SELECT MAX(temp) AS "max temp", MIN(temp) AS "min temp", MEAN(temp) AS "average temp" FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T20:00:00Z' GROUP BY time(2h), room ``` [](#view-example-results-1) View example results name: home tags: room=Kitchen | time |max temp|min temp| average temp | |--------------------|--------|--------|------------------| |2022-01-01T08:00:00Z| 23 | 21 | 22 | |2022-01-01T10:00:00Z| 22.7 | 22.4 |22.549999999999997| |2022-01-01T12:00:00Z| 22.8 | 22.5 | 22.65 | |2022-01-01T14:00:00Z| 22.8 | 22.7 | 22.75 | |2022-01-01T16:00:00Z| 22.7 | 22.4 |22.549999999999997| |2022-01-01T18:00:00Z| 23.3 | 23.1 |23.200000000000003| |2022-01-01T20:00:00Z| 22.7 | 22.7 | 22.7 | name: home tags: room=Living Room | time |max temp|min temp| average temp | |--------------------|--------|--------|------------------| |2022-01-01T08:00:00Z| 21.4 | 21.1 | 21.25 | |2022-01-01T10:00:00Z| 22.2 | 21.8 | 22 | |2022-01-01T12:00:00Z| 22.4 | 22.2 |22.299999999999997| |2022-01-01T14:00:00Z| 22.3 | 22.3 | 22.3 | |2022-01-01T16:00:00Z| 22.6 | 22.4 | 22.5 | |2022-01-01T18:00:00Z| 22.8 | 22.5 | 22.65 | |2022-01-01T20:00:00Z| 22.2 | 22.2 | 22.2 | #### Related * [InfluxQL aggregate functions](/influxdb3/enterprise/reference/influxql/functions/aggregates/) * [InfluxQL selector functions](/influxdb3/enterprise/reference/influxql/functions/selectors/) [query](/influxdb3/enterprise/tags/query/)[influxql](/influxdb3/enterprise/tags/influxql/) | time | average co | | --- | --- | | time | average co | | 0 | 5.269230769230769 | | time | average temp | | --- | --- | | time | average temp | | 0 | 22.623076923076926 | | time | average temp | | --- | --- | | time | average temp | | 0 | 22.16923076923077 | | time | max temp | min temp | average temp | | --- | --- | --- | --- | | time | max temp | min temp | average temp | | 2022-01-01T08:00:00Z | 23 | 21 | 22 | | 2022-01-01T10:00:00Z | 22.7 | 22.4 | 22.549999999999997 | | 2022-01-01T12:00:00Z | 22.8 | 22.5 | 22.65 | | 2022-01-01T14:00:00Z | 22.8 | 22.7 | 22.75 | | 2022-01-01T16:00:00Z | 22.7 | 22.4 | 22.549999999999997 | | 2022-01-01T18:00:00Z | 23.3 | 23.1 | 23.200000000000003 | | 2022-01-01T20:00:00Z | 22.7 | 22.7 | 22.7 | | time | max temp | min temp | average temp | | --- | --- | --- | --- | | time | max temp | min temp | average temp | | 2022-01-01T08:00:00Z | 21.4 | 21.1 | 21.25 | | 2022-01-01T10:00:00Z | 22.2 | 21.8 | 22 | | 2022-01-01T12:00:00Z | 22.4 | 22.2 | 22.299999999999997 | | 2022-01-01T14:00:00Z | 22.3 | 22.3 | 22.3 | | 2022-01-01T16:00:00Z | 22.6 | 22.4 | 22.5 | | 2022-01-01T18:00:00Z | 22.8 | 22.5 | 22.65 | | 2022-01-01T20:00:00Z | 22.2 | 22.2 | 22.2 | --- # Perform a basic InfluxQL query Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/influxql/basic-query/ InfluxQL (Influx Query Language) is an SQL-like query language used to interact with InfluxDB and work with times series data. > [!Note] > #### InfluxDB v1 to InfluxDB 3 data model > > InfluxQL was designed around the InfluxDB v1 data model, but can still be used > to query data from InfluxDB 3 Enterprise. When using the InfluxDB 3 Enterprise > InfluxQL implementation, the data model is different in the following ways: > > * an InfluxDB v1 **database and retention policy** combination is combined > into a single InfluxDB 3 **database** entity. > * an InfluxDB v1 **measurement** is equivalent to an InfluxDB 3 **table**. A basic InfluxQL query that queries data from InfluxDB most commonly includes the following clauses: \* Required * \* `SELECT`: Specify fields, tags, and calculations to return from a [table](/influxdb3/enterprise/reference/glossary/#table) or use the wildcard alias (`*`) to select all fields and tags from a table. It requires at least one[field key](/influxdb3/enterprise/reference/glossary/#field-key) or the wildcard alias (`*`). For more information, see[Notable SELECT statement behaviors](/influxdb3/enterprise/reference/influxql/select/#notable-select-statement-behaviors). * \* `FROM`: Specify the[table](/influxdb3/enterprise/reference/glossary/#table) to query from. It requires one or more comma-delimited[measurement expressions](/influxdb3/enterprise/reference/influxql/select/#measurement_expression). * `WHERE`: Filter data based on[field values](/influxdb3/enterprise/reference/glossary/#field),[tag values](/influxdb3/enterprise/reference/glossary/#tag), or[timestamps](/influxdb3/enterprise/reference/glossary/#timestamp). Only return data that meets the specified conditions–for example, falls within a time range, contains specific tag values, or contains a field value outside a specified range. ```sql SELECT temp, hum, room FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' ``` ## Result set If at least one row satisfies the query, InfluxDB 3 Enterprise returns row data in the query result set. If a query uses a `GROUP BY` clause, the result set includes the following: * Columns listed in the query’s `SELECT` clause * A `time` column that contains the timestamp for the record or the group * An `iox::measurement` column that contains the record’s[table](/influxdb3/enterprise/reference/glossary/#table) name * Columns listed in the query’s `GROUP BY` clause; each row in the result set contains the values used for grouping ### GROUP BY result columns If a query uses `GROUP BY` and the `WHERE` clause doesn’t filter by time, then groups are based on the[default time range](/influxdb3/enterprise/reference/influxql/group-by/#default-time-range). ## Basic query examples * [Query data within time boundaries](#query-data-within-time-boundaries) * [Query data without time boundaries](#query-data-without-time-boundaries) * [Query specific fields and tags](#query-specific-fields-and-tags) * [Query fields based on tag values](#query-fields-based-on-tag-values) * [Query points based on field values](#query-points-based-on-field-values) * [Alias queried fields and tags](#alias-queried-fields-and-tags) > [!Note] > #### Sample data > > The following examples use the [Home sensor data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-the-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. ### Query data within time boundaries * Use the `SELECT` clause to specify what tags and fields to return. Specify at least one field key. To return all tags and fields, use the wildcard alias (`*`). * Specify the [table](/influxdb3/enterprise/reference/glossary/#table) to query in the `FROM` clause. * Specify time boundaries in the `WHERE` clause. Include time-based predicates that compare the value of the `time` column to a timestamp. Use the `AND` logical operator to chain multiple predicates together. ```sql SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' ``` Query time boundaries can be relative or absolute. [](#query-with-relative-time-boundaries) Query with relative time boundaries To query data from relative time boundaries, compare the value of the `time`column to a timestamp calculated by subtracting an interval from a timestamp. Use `now()` to return the timestamp for the current time (UTC). ##### Query all data from the last month ```sql SELECT * FROM home WHERE time >= now() - 30d ``` ##### Query one day of data from a week ago ```sql SELECT * FROM home WHERE time >= now() - 7d AND time <= now() - 6d ``` [](#query-with-absolute-time-boundaries) Query with absolute time boundaries To query data from absolute time boundaries, compare the value of the `time`column to a timestamp literal. Use the `AND` logical operator to chain together multiple predicates and define both start and stop boundaries for the query. ```sql SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' ``` ### Query data without time boundaries To query data without time boundaries, do not include any time-based predicates in your `WHERE` clause. If a time range is not defined in the `WHERE` clause, the default time range is the Unix epoch (`1970-01-01T00:00:00Z`) to *now*. > [!Warning] > Querying data *without time bounds* can return an unexpected amount of data. > The query may take a long time to complete and results may be truncated. ```sql SELECT * FROM home ``` ### Query specific fields and tags To query specific fields, include them in the `SELECT` clause. If querying multiple fields or tags, comma-delimit each. If a field or tag key includes special characters or spaces or is case-sensitive, wrap the key in *double-quotes*. ```sql SELECT time, room, temp, hum FROM home ``` ### Query fields based on tag values * In the `SELECT` clause, include fields you want to query and tags you want to base conditions on. * In the `WHERE` clause, include predicates that compare the tag identifier to a string literal. Use[logical operators](/influxdb3/enterprise/reference/influxql/where/#logical-operators)to chain multiple predicates together and apply multiple conditions. ```sql SELECT * FROM home WHERE room = 'Kitchen' ``` ### Query points based on field values * In the `SELECT` clause, include fields you want to query. * In the `WHERE` clause, include predicates that compare the field identifier to a value or expression. Use[logical operators](/influxdb3/enterprise/reference/influxql/where/#logical-operators)(`AND`, `OR`) to chain multiple predicates together and apply multiple conditions. ```sql SELECT co, time FROM home WHERE co >= 10 OR co <= -10 ``` ### Alias queried fields and tags To alias or rename fields and tags that you query, use the `AS` clause. After the tag, field, or expression you want to alias, pass `AS` followed by the alias name as an identifier (wrap in double quotes (`"`) if the alias includes spaces or special characters)–for example: ```sql SELECT temp AS temperature, hum AS "humidity (%)" FROM home ``` > [!Note] > When aliasing columns in **InfluxQL**, use the `AS` clause and an[identifier](/influxdb3/enterprise/reference/influxql/#identifiers). When[aliasing columns in **SQL**](/influxdb3/enterprise/query-data/sql/basic-query/#alias-queried-fields-and-tags), > you can use the `AS` clause to define the alias, but it isn’t necessary. [query](/influxdb3/enterprise/tags/query/)[influxql](/influxdb3/enterprise/tags/influxql/) --- # Explore your schema with InfluxQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/influxql/explore-schema/ Use InfluxQL `SHOW` statements to return information about your data schema. > [!Note] > #### InfluxDB v1 to InfluxDB 3 data model > > InfluxQL was designed around the InfluxDB v1 data model, but can still be used > to query data from InfluxDB 3 Enterprise. When using the InfluxDB 3 Enterprise > InfluxQL implementation, the data model is different in the following ways: > > * an InfluxDB v1 **database and retention policy** combination is combined > into a single InfluxDB 3 **database** entity. > * an InfluxDB v1 **measurement** is equivalent to an InfluxDB 3 **table**. * [List measurements in a database](#list-measurements-in-a-database) * [List measurements that contain specific tag key-value pairs](#list-measurements-that-contain-specific-tag-key-value-pairs) * [List measurements that match a regular expression](#list-measurements-that-match-a-regular-expression) * [List field keys in a measurement](#list-field-keys-in-a-measurement) * [List tag keys in a measurement](#list-tag-keys-in-a-measurement) * [List tag keys in measurements that contain a specific tag key-value pair](#list-tag-keys-in-measurements-that-contain-a-specific-tag-key-value-pair) * [List tag values for a specific tag key](#list-tag-values-for-a-specific-tag-key) * [List tag values for multiple tags](#list-tag-values-for-multiple-tags) * [List tag values for tags that match a regular expression](#list-tag-values-for-tags-that-match-a-regular-expression) * [List tag values associated with a specific tag key-value pair](#list-tag-values-associated-with-a-specific-tag-key-value-pair) > [!Note] > #### Sample data > > The following examples use data provided in [sample data sets](/influxdb3/enterprise/reference/sample-data/). > To run the example queries and return identical results, follow the instructions > provided for each sample data set to write the data to your InfluxDB 3 Enterprise > database. ## List measurements in a database Use [`SHOW MEASUREMENTS`](/influxdb3/enterprise/reference/influxql/show/#show-measurements)to list measurements in your InfluxDB database. ```sql SHOW MEASUREMENTS ``` [](#view-example-output) View example output name: measurements | name | |-------------| | bitcoin | | home | |home\_actions| | numbers | | weather | ### List measurements that contain specific tag key-value pairs To return only measurements with specific tag key-value pairs, include a `WHERE`clause with tag key-value pairs to query for. ```sql SHOW MEASUREMENTS WHERE room = 'Kitchen' ``` [](#view-example-output-1) View example output name: measurements | name | |-------------| | home | |home\_actions| ### List measurements that match a regular expression To return only measurements with names that match a[regular expression](/influxdb3/enterprise/reference/influxql/regular-expressions/), include a `WITH` clause that compares the `MEASUREMENT` to a regular expression. ```sql SHOW MEASUREMENTS WITH MEASUREMENT =~ /^home/ ``` [](#view-example-output-2) View example output name: measurements | name | |-------------| | home | |home\_actions| ## List field keys in a measurement Use [`SHOW FIELD KEYS`](/influxdb3/enterprise/reference/influxql/show/#show-field-keys)to return all field keys in a measurement. Include a `FROM` clause to specify the measurement. If no measurement is specified, the query returns all field keys in the database. ```sql SHOW FIELD KEYS FROM home ``` [](#view-example-output-3) View example output name: home |fieldKey|fieldType| |--------|---------| | co | integer | | hum | float | | temp | float | ## List tag keys in a measurement Use [`SHOW TAG KEYS`](/influxdb3/enterprise/reference/influxql/show/#show-tag-keys)to return all tag keys in a measurement. Include a `FROM` clause to specify the measurement. If no measurement is specified, the query returns all tag keys in the database. ```sql SHOW TAG KEYS FROM home_actions ``` [](#view-example-output-4) View example output name: home\_actions |tagKey| |------| |action| |level | | room | ### List tag keys in measurements that contain a specific tag key-value pair To return all tag keys measurements that contain specific tag key-value pairs, include a `WHERE` clause with the tag key-value pairs to query for. ```sql SHOW TAG KEYS WHERE room = 'Kitchen' ``` [](#view-example-output-5) View example output name: home |tagKey| |------| | room | name: home\_actions |tagKey| |------| |action| |level | | room | ## List tag values for a specific tag key Use [`SHOW TAG VALUES`](/influxdb3/enterprise/reference/influxql/show/#show-tag-values)to return all values for specific tags in a measurement. * Include a `FROM` clause to specify one or more measurements to query. * Use the `WITH` clause to compare `KEY` to tag keys to list the values of. * Use the `WHERE` clause to restrict the search to a specific time range (default time range is the last day). ```sql SHOW TAG VALUES FROM weather WITH KEY = location ``` > [!Note] > #### Include a FROM clause > > We strongly recommend including a `FROM` clause with the `SHOW TAG VALUES`statement that specifies 1-50 tables to query. > Without a `FROM` clause, the InfluxDB query engine must read data from all > tables and return unique tag values from each. > > Depending on the number of tables in your database and the number of unique tag > values in each table, excluding a `FROM` clause can result in poor query performance, > query timeouts, or unnecessary resource allocation that may affect other queries. [](#view-example-output-5) View example output name: weather | key | value | |--------|-------------| |location| Concord | |location| Hayward | |location|San Francisco| ### List tag values for multiple tags To return tag values for multiple specific tag keys, use the `IN` operator in the `WITH` clause to compare `KEY` to a list of tag keys. ```sql SHOW TAG VALUES FROM home_actions WITH KEY IN ("level", "action") ``` [](#view-example-output-6) View example output name: home\_actions | key |value| |------|-----| |action|alert| |action|cool | |level | ok | |level |warn | ### List tag values for tags that match a regular expression To return only tag values from tags keys that match a regular expression, use regular expression comparison operators in your `WITH` clause to compare `KEY`to the regular expression. ```sql SHOW TAG VALUES FROM home, home_actions WITH KEY =~ /oo/ ``` [](#view-example-output-7) View example output name: home |key | value | |----|-----------| |room| Kitchen | |room|Living Room| name: home\_actions |key | value | |----|-----------| |room| Kitchen | |room|Living Room| ### List tag values associated with a specific tag key-value pair To list tag values for tags associated with a specific tag key-value pair: * Use the `WITH` clause to identify what tag keys to return values for. * Include a `WHERE` clause that identifies the tag key-value pair to query for. The following example returns tag values for the `action` and `level` tags for points where the `room` tag value is `Kitchen`. ```sql SHOW TAG VALUES FROM home_actions WITH KEY IN ("action", "level") WHERE room = 'Kitchen' ``` [](#view-example-output-8) View example output name: home\_actions | key |value| |------|-----| |action|alert| |action|cool | |level | ok | |level |warn | #### Related * [InfluxQL SHOW statements](/influxdb3/enterprise/reference/influxql/show/) [query](/influxdb3/enterprise/tags/query/)[influxql](/influxdb3/enterprise/tags/influxql/) | name | | --- | | name | | bitcoin | | home | | home_actions | | numbers | | weather | | name | | --- | | name | | home | | home_actions | | name | | --- | | name | | home | | home_actions | | fieldKey | fieldType | | --- | --- | | fieldKey | fieldType | | co | integer | | hum | float | | temp | float | | tagKey | | --- | | tagKey | | action | | level | | room | | tagKey | | --- | | tagKey | | room | | tagKey | | --- | | tagKey | | action | | level | | room | | key | value | | --- | --- | | key | value | | location | Concord | | location | Hayward | | location | San Francisco | | key | value | | --- | --- | | key | value | | action | alert | | action | cool | | level | ok | | level | warn | | key | value | | --- | --- | | key | value | | room | Kitchen | | room | Living Room | | key | value | | --- | --- | | key | value | | room | Kitchen | | room | Living Room | | key | value | | --- | --- | | key | value | | action | alert | | action | cool | | level | ok | | level | warn | --- # Use parameterized queries with InfluxQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/influxql/parameterized-queries/ Parameterized queries in InfluxDB 3 Enterprise let you dynamically and safely change values in a query. If your application code allows user input to customize values or expressions in a query, use a parameterized query to make sure untrusted input is processed strictly as data and not executed as code. Parameterized queries: * help prevent injection attacks, which can occur if input is executed as code * help make queries more reusable > [!Note] > #### Prevent injection attacks > > For more information on security and query parameterization, > see the [OWASP SQL Injection Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/\> SQL\_Injection\_Prevention\_Cheat\_Sheet.html#defense-option-1-prepared-statements-with-parameterized-queries). In InfluxDB 3, a parameterized query is an InfluxQL or SQL query that contains one or more named parameter placeholders–variables that represent input data. * [Use parameters in `WHERE` expressions](#use-parameters-in-where-expressions) * [Parameter data types](#parameter-data-types) * [Data type examples](#data-type-examples) * [Time expressions](#time-expressions) * [Not compatible with parameters](#not-compatible-with-parameters) * [Parameterize an SQL query](#parameterize-an-sql-query) * [Execute parameterized InfluxQL queries](#execute-parameterized-influxql-queries) * [Use the HTTP API](#use-the-http-api) * [Use InfluxDB Flight RPC clients](#use-influxdb-flight-rpc-clients) * [Client support for parameterized queries](#client-support-for-parameterized-queries) * [Not supported](#not-supported) > [!Note] > #### Parameters only supported in `WHERE` expressions > > InfluxDB 3 supports parameters in `WHERE` clause **predicate expressions**. > Parameter values must be one of the [allowed parameter data types](#parameter-data-types). > > If you use parameters in other expressions or clauses, > such as function arguments, `SELECT`, or `GROUP BY`, then your query might not work as you expect. ## Use parameters in `WHERE` expressions You can use parameters in `WHERE` clause **predicate expressions**-–for example, the following query contains a `$temp` parameter: ```sql SELECT * FROM measurement WHERE temp > $temp ``` When executing a query, you specify parameter name-value pairs. The value that you assign to a parameter must be one of the [parameter data types](#parameter-data-types). ```go {"temp": 22.0} ``` The InfluxDB Querier parses the query text with the parameter placeholders, and then generates query plans that replace the placeholders with the values that you provide. This separation of query structure from input data ensures that input is treated as one of the allowed [data types](#parameter-data-types) and not as executable code. ## Parameter data types A parameter value can be one of the following data types: * Null * Boolean * Unsigned integer (`u_int64`) * Integer (`int64`) * Double (`float64`) * String ### Data type examples ```js { "string": "Living Room", "double": 3.14, "unsigned_integer": 1234, "integer": -1234, "boolean": false, "null": Null, } ``` ### Time expressions To parameterize time bounds, substitute a parameter for a timestamp literal–for example: ```sql SELECT * FROM home WHERE time >= $min_time ``` For the parameter value, specify the timestamp literal as a string–for example: ```go // Assign a timestamp string literal to the min_time parameter. parameters := influxdb3.QueryParameters{ "min_time": "2022-01-01 00:00:00.00", } ``` InfluxDB executes the query as the following: ```sql SELECT * FROM home WHERE time >= '2022-01-01 00:00:00.00' ``` ### Not compatible with parameters If you use parameters for the following, your query might not work as you expect: * In clauses other than `WHERE`, such as `SELECT` or `GROUP BY` * As function arguments, such as `avg($temp)` * In place of identifiers, such as column or table names * In place of duration literals, such as `time > now() - $min_duration` ## Parameterize an SQL query > [!Note] > #### Sample data > > The following examples use the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-the-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. To use a parameterized query, do the following: 1. In your query text, use the `$parameter` syntax to reference a parameter name–for example, the following query contains `$room` and `$min_temp` parameter placeholders: ``` SELECT * FROM home WHERE time > now() - 7d AND temp >= $min_temp AND room = $room ``` 2. Provide a value for each parameter name. If you don’t assign a value for a parameter, InfluxDB returns an error. The syntax for providing parameter values depends on the client you use–for example: [Go](#go) ``` // Define a QueryParameters struct--a map of parameters to input values. parameters := influxdb3.QueryParameters{ "room": "Kitchen", "min_temp": 20.0, } ``` After InfluxDB receives your request and parses the query, it executes the query as ```sql SELECT * FROM home WHERE time > now() - 7d AND temp >= 20.0 AND room = 'Kitchen' ``` ## Execute parameterized InfluxQL queries > [!Note] > #### Sample data > > The following examples use the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-the-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. ### Use the HTTP API InfluxDB 3 Enterprise provides the `/api/v3/query_influxql` HTTP API endpoint for executing InfluxQL queries with parameters. ``` POST /api/v3/query_influxql ``` Send a JSON object that contains `db` (database), `q` (query), and `params` (parameter name-value pairs) properties in the request body. The following example sends a parameterized InfluxQL query to the `/api/v3/query_influxql` endpoint: ```bash curl "http://localhost:8181/api/v3/query_influxql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "q": "SELECT * FROM home WHERE time >= $min_time AND temp >= $min_temp AND room = $room", "params": { "min_time": "2022-01-01T08:00:00Z", "min_temp": 22.0, "room": "Kitchen" } }' ``` Replace the following: * `DATABASE_NAME`: the name of the database to query * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to query the specified database The response body contains query results in JSON format: ```json [ {"iox::measurement":"home","time":"2022-01-01T09:00:00","co":0,"hum":36.2,"room":"Kitchen","temp":23.0}, {"iox::measurement":"home","time":"2022-01-01T10:00:00","co":0,"hum":36.1,"room":"Kitchen","temp":22.7}, {"iox::measurement":"home","time":"2022-01-01T11:00:00","co":0,"hum":36.0,"room":"Kitchen","temp":22.4}, {"iox::measurement":"home","time":"2022-01-01T12:00:00","co":0,"hum":36.0,"room":"Kitchen","temp":22.5} ] ``` ### Use InfluxDB Flight RPC clients Using the InfluxDB 3 native Flight RPC protocol and supported clients, you can send a parameterized query and a list of parameter name-value pairs. InfluxDB Flight clients that support parameterized queries pass the parameter name-value pairs in a Flight ticket `params` field. The following examples show how to use client libraries to execute parameterized InfluxQL queries: [Go](#go-2) ```go import ( "context" "fmt" "io" "os" "text/tabwriter" "time" "github.com/InfluxCommunity/influxdb3-go/v2/influxdb3" ) func Query(query string, parameters influxdb3.QueryParameters, options influxdb3.QueryOptions) error { url := os.Getenv("INFLUX_HOST") token := os.Getenv("INFLUX_TOKEN") database := os.Getenv("INFLUX_DATABASE") // Instantiate the influxdb3 client. client, err := influxdb3.New(influxdb3.ClientConfig{ Host: url, Token: token, Database: database, }) if err != nil { panic(err) } // Ensure the client is closed after the Query function finishes. defer func(client *influxdb3.Client) { err := client.Close() if err != nil { panic(err) } }(client) // Call the client's QueryWithParameters function. // Provide the query, parameters, and the InfluxQL QueryType option. iterator, err := client.QueryWithParameters(context.Background(), query, parameters, influxdb3.WithQueryType(options.QueryType)) // Create a buffer for storing rows as you process them. w := tabwriter.NewWriter(io.Discard, 4, 4, 1, ' ', 0) w.Init(os.Stdout, 0, 8, 0, '\t', 0) fmt.Fprintf(w, "time\troom\tco\thum\ttemp\n") // Format and write each row to the buffer. // Process each row as key-value pairs. for iterator.Next() { row := iterator.Value() // Use Go time package to format unix timestamp // as a time with timezone layout (RFC3339 format) time := (row["time"].(time.Time)). Format(time.RFC3339) fmt.Fprintf(w, "%s\t%s\t%d\t%.1f\t%.1f\n", time, row["room"], row["co"], row["hum"], row["temp"]) } w.Flush() return nil } func main() { // Use the $placeholder syntax in a query to reference parameter placeholders // for input data. // The following InfluxQL query contains the placeholders $room and $min_temp. query := ` SELECT * FROM home WHERE time > now() - 7d AND temp >= $min_temp AND room = $room` // Define a QueryParameters struct--a map of placeholder names to input values. parameters := influxdb3.QueryParameters{ "room": "Kitchen", "min_temp": 20.0, } Query(query, parameters, influxdb3.QueryOptions{ QueryType: influxdb3.InfluxQL, }) } ``` ## Client support for parameterized queries * Not all [InfluxDB 3 Flight clients](/influxdb3/enterprise/reference/client-libraries/v3/) support parameterized queries. * InfluxDB doesn’t currently support parameterized queries or DataFusion prepared statements for Flight SQL or Flight SQL clients. * InfluxDB 3 SQL and InfluxQL parameterized queries aren’t supported in InfluxDB v1 and v2 clients. ## Not supported Currently, parameterized queries in InfluxDB 3 Enterprise don’t provide the following: * support for DataFusion prepared statements * query caching, optimization, or performance benefits [query](/influxdb3/enterprise/tags/query/)[security](/influxdb3/enterprise/tags/security/)[influxql](/influxdb3/enterprise/tags/influxql/) --- # Troubleshoot InfluxQL errors Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/influxql/troubleshoot/ Learn how to troubleshoot and fix common InfluxQL errors. > [!Note] > **Disclaimer:** This document does not contain an exhaustive list of all > possible InfluxQL errors. * [error: database name required](#error-database-name-required) * [error parsing query: found …, expected identifier at …](#error-parsing-query-found--expected-identifier-at-) * [error parsing query: mixing aggregate and non-aggregate queries is not supported](#error-parsing-query-mixing-aggregate-and-non-aggregate-queries-is-not-supported) * [invalid operation: time and \*influxql.VarRef are not compatible](#invalid-operation-time-and-influxqlvarref-are-not-compatible) > [!Note] > #### InfluxDB v1 to InfluxDB 3 data model > > InfluxQL was designed around the InfluxDB v1 data model, but can still be used > to query data from InfluxDB 3 Enterprise. When using the InfluxDB 3 Enterprise > InfluxQL implementation, the data model is different in the following ways: > > * an InfluxDB v1 **database and retention policy** combination is combined > into a single InfluxDB 3 **database** entity. > * an InfluxDB v1 **measurement** is equivalent to an InfluxDB 3 **table**. ## error: database name required ``` error: database name required ``` ### Cause The `database name required` error occurs when certain[`SHOW` queries](/influxdb3/enterprise/reference/influxql/show/)do not specify a [database](/influxdb3/enterprise/reference/glossary/#database)in the query or with the query request. For example, the following `SHOW` query doesn’t specify the database and assumes the `db` is not specified in the `/query` API request: ```sql SHOW MEASUREMENTS ``` ### Solution To resolve this error, specify a database with your query request by doing one of the following: * Include an `ON` clause with the `SHOW` statement that specifies the database to query: ``` SHOW MEASUREMENTS ON DATABASE_NAME ``` * If using the [InfluxDB v1 query API](/influxdb3/enterprise/api/query-data/#operation/GetV1ExecuteQuery), Include the `db` query parameter in your request: ``` curl --get http://localhost:8181/query \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-urlencode "db=DATABASE_NAME" \ --data-urlencode "q=SHOW MEASUREMENTS" ``` **Related:**[InfluxQL `SHOW` statements](/influxdb3/enterprise/reference/influxql/show/),[Explore your schema with InfluxQL](/influxdb3/enterprise/query-data/influxql/explore-schema/) ## error parsing query: found …, expected identifier at … ``` error parsing query: found EXAMPLE, expected identifier at line 1, char 14 ``` ### Causes This error occurs when InfluxDB anticipates an identifier in a query but doesn’t find it. Identifiers are tokens that refer to database names, retention policy names, measurement names, field keys, and tag keys. This error is generally caused by one of the following: * [A required identifier is missing](#a-required-identifier-is-missing) * [A string literal is used instead of an identifier](#a-string-literal-is-used-instead-of-an-identifier) * [An InfluxQL keyword is used as an unquoted identifier](#an-influxql-keyword-is-used-as-an-unquoted-identifier) #### A required identifier is missing Some InfluxQL statements and clauses require identifiers to identify databases, measurements, tags, or fields. If the statement is missing a required identifier, the query returns the `expected identifier` error. For example, the following query omits the measurement name from the[`FROM` clause](/influxdb3/enterprise/reference/influxql/select/#from-clause): ```sql SELECT * FROM WHERE color = 'blue' ``` ##### Solution Update the query to include the expected identifier in the `FROM` clause that identifies the measurement to query: ```sql SELECT * FROM measurement_name WHERE color = 'blue' ``` #### A string literal is used instead of an identifier In InfluxQL, string literals are wrapped in single quotes (`''`) while character sequences wrapped in double quotes (`""`) are parsed as identifiers. If you use single quotes to wrap an identifier, the identifier is parsed as a string literal and returns the `expected identifier` error. For example, the following query wraps the measurement name in single quotes: ```sql SELECT * FROM 'measurement-name' WHERE color = 'blue' ``` Results in the following error: ``` error parsing query: found measurement-name, expected identifier at line 1, char 14 ``` ##### Solution Update single-quoted identifiers to use double quotes so they are parsed as identifiers and not as string literals. ```sql SELECT * FROM "measurement-name" WHERE color = 'blue' ``` #### An InfluxQL keyword is used as an unquoted identifier [InfluxQL keyword](/influxdb3/enterprise/reference/influxql/#keywords)are character sequences reserved for specific functionality in the InfluxQL syntax. It is possible to use a keyword as an identifier, but the identifier must be wrapped in double quotes (`""`). > [!Note] > While wrapping identifiers that are InfluxQL keywords in double quotes is an > acceptable workaround, for simplicity, you should avoid using[InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords)as identifiers. ```sql SELECT duration FROM runs ``` Returns the following error: ``` error parsing query: found DURATION, expected identifier, string, number, bool at line 1, char 8 ``` ##### Solution Double quote [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords)when used as identifiers: ```sql SELECT "duration" FROM runs ``` **Related:**[InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords) ## error parsing query: mixing aggregate and non-aggregate queries is not supported ``` error parsing query: mixing aggregate and non-aggregate queries is not supported ``` ### Cause The `mixing aggregate and non-aggregate` error occurs when a `SELECT` statement includes both an [aggregate function](/influxdb3/enterprise/reference/influxql/functions/aggregates/)and a standalone [field key](/influxdb3/enterprise/reference/glossary/#field-key) or[tag key](/influxdb3/enterprise/reference/glossary/#tag-key). Aggregate functions return a single calculated value per group and column and there is no obvious single value to return for any un-aggregated fields or tags. For example, the following example queries two fields from the `home`measurement–`temp` and `hum`. However, it only applies the aggregate function,`MEAN` to the `temp` field. ```sql SELECT MEAN(temp), hum FROM home ``` ### Solution To fix this error, apply an aggregate or selector function to each of the queried fields: ```sql SELECT MEAN(temp), MAX(hum) FROM home ``` **Related:**[InfluxQL functions](/influxdb3/enterprise/reference/influxql/functions/),[Aggregate data with InfluxQL](/influxdb3/enterprise/query-data/influxql/aggregate-select/) ## invalid operation: time and \*influxql.VarRef are not compatible ``` invalid operation: time and *influxql.VarRef are not compatible ``` ### Cause The `time and \*influxql.VarRef are not compatible` error occurs when date-time strings are double-quoted in a query. Date-time strings should be formatted as string literals and wrapped in single quotes (`''`). For example: ```sql {lint="false"} SELECT temp FROM home WHERE time \>= "2022-01-01T08:00:00Z" AND time \<= "2022-01-01T20:00:00Z" ``` Returns the following error: ``` invalid operation: time and *influxql.VarRef are not compatible ``` ### Solution To fix the error, wrap RFC3339 timestamps in single quotes rather than double quotes. ```sql SELECT temp FROM home WHERE time \>= '2022-01-01T08:00:00Z' AND time \<= '2022-01-01T20:00:00Z' ``` **Related:**[Query data within time boundaries](/influxdb3/enterprise/query-data/influxql/basic-query/#query-data-within-time-boundaries),[`WHERE` clause–Time ranges](/influxdb3/enterprise/reference/influxql/where/#time-ranges),[InfluxQL time syntax](/influxdb3/enterprise/reference/influxql/time-and-timezone/#time-syntax) --- # Query data with SQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/ Learn to query data in InfluxDB 3 Enterprise using SQL. * [Explore your schema with SQL](#explore-your-schema-with-sql) * [Perform a basic SQL query](#perform-a-basic-sql-query) * [Aggregate data with SQL](#aggregate-data-with-sql) * [Cast values to different types](#cast-values-to-different-types) * [Compare values in SQL queries](#compare-values-in-sql-queries) * [Fill gaps in data](#fill-gaps-in-data) * [Use parameterized queries with SQL](#use-parameterized-queries-with-sql) ### [Explore your schema with SQL](/influxdb3/enterprise/query-data/sql/explore-schema/) Use SQL to explore your data schema in your InfluxDB 3 Enterprise database. ##### List tables ```sql SHOW TABLES ``` ##### List columns in a table ```sql SHOW COLUMNS IN table ``` [Read more ](/influxdb3/enterprise/query-data/sql/explore-schema/) ### [Perform a basic SQL query](/influxdb3/enterprise/query-data/sql/basic-query/) A basic SQL query that queries data from InfluxDB 3 Enterprise most commonly includes `SELECT`, `FROM`, and `WHERE` clauses. ```sql SELECT temp, room FROM home WHERE time >= now() - INTERVAL '1 day' ``` [Read more ](/influxdb3/enterprise/query-data/sql/basic-query/) ### [Aggregate data with SQL](/influxdb3/enterprise/query-data/sql/aggregate-select/) Use aggregate and selector functions to perform aggregate operations on your time series data. ##### Aggregate fields by groups ```sql SELECT mean(field1) AS mean, selector_first(field2)['value'] as first, tag1 FROM home GROUP BY tag ``` ##### Aggregate by time-based intervals ```sql SELECT DATE_BIN(INTERVAL '1 hour', time, '2022-01-01T00:00:00Z'::TIMESTAMP) AS time, mean(field1), sum(field2), tag1 FROM home GROUP BY 1, tag1 ``` [Read more ](/influxdb3/enterprise/query-data/sql/aggregate-select/) ### [Cast values to different types](/influxdb3/enterprise/query-data/sql/cast-types/) Use the `CAST` function or double-colon `::` casting shorthand syntax to cast a value to a specific type. ```sql -- CAST clause SELECT CAST(1234.5 AS BIGINT) -- Double-colon casting shorthand SELECT 1234.5::BIGINT ``` [Read more ](/influxdb3/enterprise/query-data/sql/cast-types/) ### [Compare values in SQL queries](/influxdb3/enterprise/query-data/sql/compare-values/) Use SQL window functions to compare values across different rows in your time series data. Learn how to calculate differences, percentage changes, and compare values at specific time intervals. ##### Calculate difference from previous value ```sql SELECT time, room, temp, temp - LAG(temp) OVER ( PARTITION BY room ORDER BY time ) AS temp_change FROM home ORDER BY room, time ``` [Read more ](/influxdb3/enterprise/query-data/sql/compare-values/) ### [Fill gaps in data](/influxdb3/enterprise/query-data/sql/fill-gaps/) Use [`date_bin_gapfill`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill) with [`interpolate`](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate) or [`locf`](/influxdb3/enterprise/reference/sql/functions/misc/#locf) to fill gaps of time where no data is returned. ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as time, room, interpolate(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY 1, room ``` [Read more ](/influxdb3/enterprise/query-data/sql/fill-gaps/) ### [Use parameterized queries with SQL](/influxdb3/enterprise/query-data/sql/parameterized-queries/) Use parameterized queries to prevent injection attacks and make queries more reusable. ```sql SELECT * FROM home WHERE time >= $min_time AND temp >= $min_temp AND room = $room ``` [Read more ](/influxdb3/enterprise/query-data/sql/parameterized-queries/) [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/) --- # Aggregate data with SQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/aggregate-select/ An SQL query that aggregates data includes the following clauses: \* Required * \* `SELECT`: Specify fields, tags, and calculations to output from a table or use the wildcard alias (`*`) to select all fields and tags from a table. * \* `FROM`: Specify the table to query data from. * `WHERE`: Only return rows that meets the specified conditions–for example, the time is within a time range, a tag has a specific value, or a field value is above or below a specified threshold. * `GROUP BY`: Group data that have the same values for specified columns and expressions (for example, an aggregate function result). > [!Note] > For simplicity, the verb, **“aggregate,”** in this guide refers to applying > both aggregate and selector functions to a dataset. Learn how to apply aggregate operations to your queried data: * [Aggregate and selector functions](#aggregate-and-selector-functions) * [Aggregate functions](#aggregate-functions) * [Selector functions](#selector-functions) * [Example aggregate queries](#example-aggregate-queries) ## Aggregate and selector functions Both aggregate and selector functions return a single row from each SQL group. For example, if you `GROUP BY room` and perform an aggregate operation in your `SELECT` clause, results include an aggregate value for each unique value of `room`. ### Aggregate functions Use **aggregate functions** to aggregate values in a specified column for each group and return a single row per group containing the aggregate value. [View SQL aggregate functions](/influxdb3/enterprise/reference/sql/functions/aggregate/) ##### Basic aggregate query ```sql SELECT AVG(co) from home ``` ### Selector functions Use **selector functions** to “select” a value from a specified column. The available selector functions are designed to work with time series data. [View SQL selector functions](/influxdb3/enterprise/reference/sql/functions/selector/) Each selector function returns a Rust *struct* (similar to a JSON object) representing a single time and value from the specified column in the each group. What time and value get returned depend on the logic in the selector function. For example, `selector_first` returns the value of specified column in the first row of the group. `selector_max` returns the maximum value of the specified column in the group. #### Selector struct schema The struct returned from a selector function has two properties: * **time**: `time` value in the selected row * **value**: value of the specified column in the selected row ```js {time: 2023-01-01T00:00:00Z, value: 72.1} ``` #### Use selector functions Each selector function has two arguments: * The first is the column to operate on. * The second is the time column to use in the selection logic. In your `SELECT` statement, execute a selector function and use bracket notation to reference properties of the [returned struct](#selector-struct-schema) to populate the column value: ```sql SELECT selector_first(temp, time)['time'] AS time, selector_first(temp, time)['value'] AS temp, room FROM home GROUP BY room ``` ## Example aggregate queries * [Perform an ungrouped aggregation](#perform-an-ungrouped-aggregation) * [Group and aggregate data](#group-and-aggregate-data) * [Downsample data by applying interval-based aggregates](#downsample-data-by-applying-interval-based-aggregates) * [Query rows based on aggregate values](#query-rows-based-on-aggregate-values) > [!Note] > #### Sample data > > The following examples use the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. ### Perform an ungrouped aggregation To aggregate *all* queried values in a specified column: * Use aggregate or selector functions in your `SELECT` statement. * Do not include a `GROUP BY` clause to leave your data ungrouped. ```sql SELECT avg(co) AS 'average co' from home ``` [](#view-example-results) View example results | average co | |-----------------| |5.269230769230769| ### Group and aggregate data To apply aggregate or selector functions to grouped data: * Use aggregate or selector functions in your `SELECT` statement. * Include columns to group by in your `SELECT` statement. * Include a `GROUP BY` clause with a comma-delimited list of columns and expressions to group by. Keep the following in mind when using `GROUP BY`: * `GROUP BY` can use column aliases that are defined in the `SELECT` clause. * `GROUP BY` won’t use an aliased value if the alias is the same as the original column name. `GROUP BY` will use the original value of the column, not the transformed, aliased value. ```sql SELECT room, avg(temp) AS 'average temp' FROM home GROUP BY room ``` [](#view-example-results) View example results | room | average temp | |-----------|------------------| |Living Room|22.16923076923077 | | Kitchen |22.623076923076926| #### Downsample data by applying interval-based aggregates A common use case when querying time series is downsampling data by applying aggregates to time-based groups. To group and aggregate data into time-based groups: * In your `SELECT` clause: * Use the [`DATE_BIN` function](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin)to calculate time intervals and output a column that contains the start of the interval nearest to the `time` timestamp in each row–for example, the following clause calculates two-hour intervals (originating at the Unix epoch) and returns a new `time` column that contains the start of the interval nearest to `home.time`: ``` SELECT DATE_BIN(INTERVAL '2 hours', time) AS time FROM home ... ``` Given a `time` value`2022-01-01T13:00:50.000Z`, the output `time` column contains`2022-01-01T12:00:00.000Z`. * Use [aggregate](/influxdb3/enterprise/reference/sql/functions/aggregate/) or[selector](/influxdb3/enterprise/reference/sql/functions/selector/) functions on specified columns. * In your `GROUP BY` clause: * Specify the `DATE_BIN(...)` column ordinal reference (`1`). This lets you group by the transformed `time` value and maintain the `time`column name. * Specify other columns (for example, `room`) that are specified in the`SELECT` clause and aren’t used in a selector function. ``` SELECT DATE_BIN(INTERVAL '2 hours', time) AS time ... GROUP BY 1, room ... ``` To reference the `DATE_BIN(...)` result column by *name* in the `GROUP BY`clause, assign an alias other than “time” in the `SELECT` clause–for example: ``` SELECT DATE_BIN(INTERVAL '2 hours', time) AS _time FROM home ... GROUP BY _time, room ``` * Include an `ORDER BY` clause with columns to sort by. The following example retrieves unique combinations of time intervals and rooms with their minimum, maximum, and average temperatures: ```sql SELECT DATE_BIN(INTERVAL '2 hours', time) AS time, room, selector_max(temp, time)['value'] AS 'max temp', selector_min(temp, time)['value'] AS 'min temp', avg(temp) AS 'average temp' FROM home GROUP BY 1, room ORDER BY room, 1 ``` [](#view-example-results) View example results | time | room |max temp|min temp| average temp | |--------------------|-----------|--------|--------|------------------| |2022-01-01T08:00:00Z| Kitchen | 23 | 21 | 22 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | 22.4 |22.549999999999997| |2022-01-01T12:00:00Z| Kitchen | 22.8 | 22.5 | 22.65 | |2022-01-01T14:00:00Z| Kitchen | 22.8 | 22.7 | 22.75 | |2022-01-01T16:00:00Z| Kitchen | 22.7 | 22.4 |22.549999999999997| |2022-01-01T18:00:00Z| Kitchen | 23.3 | 23.1 |23.200000000000003| |2022-01-01T20:00:00Z| Kitchen | 22.7 | 22.7 | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.4 | 21.1 | 21.25 | |2022-01-01T10:00:00Z|Living Room| 22.2 | 21.8 | 22 | |2022-01-01T12:00:00Z|Living Room| 22.4 | 22.2 |22.299999999999997| |2022-01-01T14:00:00Z|Living Room| 22.3 | 22.3 | 22.3 | |2022-01-01T16:00:00Z|Living Room| 22.6 | 22.4 | 22.5 | |2022-01-01T18:00:00Z|Living Room| 22.8 | 22.5 | 22.65 | |2022-01-01T20:00:00Z|Living Room| 22.2 | 22.2 | 22.2 | > [!Note] > #### GROUP BY time > > In the `GROUP BY` clause, the name “time” always refers to the `time` column > in the source table. If you want to reference a calculated time column by name, > use an alias different from “time” or use the column ordinal–for example: > > [Column alias](#column-alias)[Column ordinal](#column-ordinal) > > ``` > SELECT > DATE_BIN(INTERVAL '2 hours', time) AS _time, > room, > selector_max(temp, time)['value'] AS 'max temp', > selector_min(temp, time)['value'] AS 'min temp', > avg(temp) AS 'average temp' > FROM home > GROUP BY _time, room > ORDER BY room, _time > ``` > > ``` > SELECT > DATE_BIN(INTERVAL '2 hours', time) AS time, > room, > selector_max(temp, time)['value'] AS 'max temp', > selector_min(temp, time)['value'] AS 'min temp', > avg(temp) AS 'average temp' > FROM home > GROUP BY 1, room > ORDER BY room, 1 > ``` ### Query rows based on aggregate values To query data based on values after an aggregate operation, include a `HAVING`clause with defined predicate conditions such as a value threshold. Predicates in the `WHERE` clause are applied *before* data is aggregated. Predicates in the `HAVING` clause are applied *after* data is aggregated. ```sql SELECT room, avg(co) AS 'average co' FROM home GROUP BY room HAVING "average co" > 5 ``` [](#view-example-results) View example results | room | average co | |-------|------------------| |Kitchen|6.6923076923076925| #### Related * [SQL aggregate functions](/influxdb3/enterprise/reference/sql/functions/aggregate/) * [SQL selector functions](/influxdb3/enterprise/reference/sql/functions/selector/) * [GROUP BY clause](/influxdb3/enterprise/reference/sql/group-by/) [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/) | average co | | --- | | average co | | 5.269230769230769 | | room | average temp | | --- | --- | | room | average temp | | Living Room | 22.16923076923077 | | Kitchen | 22.623076923076926 | | time | room | max temp | min temp | average temp | | --- | --- | --- | --- | --- | | time | room | max temp | min temp | average temp | | 2022-01-01T08:00:00Z | Kitchen | 23 | 21 | 22 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | 22.4 | 22.549999999999997 | | 2022-01-01T12:00:00Z | Kitchen | 22.8 | 22.5 | 22.65 | | 2022-01-01T14:00:00Z | Kitchen | 22.8 | 22.7 | 22.75 | | 2022-01-01T16:00:00Z | Kitchen | 22.7 | 22.4 | 22.549999999999997 | | 2022-01-01T18:00:00Z | Kitchen | 23.3 | 23.1 | 23.200000000000003 | | 2022-01-01T20:00:00Z | Kitchen | 22.7 | 22.7 | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.4 | 21.1 | 21.25 | | 2022-01-01T10:00:00Z | Living Room | 22.2 | 21.8 | 22 | | 2022-01-01T12:00:00Z | Living Room | 22.4 | 22.2 | 22.299999999999997 | | 2022-01-01T14:00:00Z | Living Room | 22.3 | 22.3 | 22.3 | | 2022-01-01T16:00:00Z | Living Room | 22.6 | 22.4 | 22.5 | | 2022-01-01T18:00:00Z | Living Room | 22.8 | 22.5 | 22.65 | | 2022-01-01T20:00:00Z | Living Room | 22.2 | 22.2 | 22.2 | | room | average co | | --- | --- | | room | average co | | Kitchen | 6.6923076923076925 | --- # Perform a basic SQL query Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/basic-query/ The InfluxDB SQL implementation is powered by the [Apache Arrow DataFusion](https://arrow.apache.org/datafusion/)query engine which provides an SQL syntax similar to other relational query languages. A basic SQL query that queries data from InfluxDB 3 Enterprise most commonly includes the following clauses: \* Required * \* `SELECT`: Specify fields, tags, and calculations to output from a table or use the wildcard alias (`*`) to select all fields and tags from a table. * \* `FROM`: Specify the table to query data from. * `WHERE`: Only return rows that meets the specified conditions–for example, the time is within a time range, a tag has a specific value, or a field value is above or below a specified threshold. ```sql SELECT temp, hum, room FROM home WHERE time \>= '2022-01-01T08:00:00Z' AND time \<= '2022-01-01T20:00:00Z' ``` ## Result set If at least one row satisfies the query, InfluxDB 3 Enterprise returns row data in the query result set. An SQL query result set includes columns listed in the query’s `SELECT` statement. ## Basic query examples * [Query data within time boundaries](#query-data-within-time-boundaries) * [Query data without time boundaries](#query-data-without-time-boundaries) * [Query specific fields and tags](#query-specific-fields-and-tags) * [Query fields based on tag values](#query-fields-based-on-tag-values) * [Query points based on field values](#query-points-based-on-field-values) * [Alias queried fields and tags](#alias-queried-fields-and-tags) > [!Note] > #### Sample data > > The following examples use the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. ### Query data within time boundaries * Use the `SELECT` clause to specify what tags and fields to return. To return all tags and fields, use the wildcard alias (`*`). * Specify the table to query in the `FROM` clause. * Specify time boundaries in the `WHERE` clause. Include time-based predicates that compare the value of the `time` column to a timestamp. Use the `AND` logical operator to chain multiple predicates together. ```sql SELECT \* FROM home WHERE time \>= '2022-01-01T08:00:00Z' AND time \<= '2022-01-01T12:00:00Z' ``` Query time boundaries can be relative or absolute. [](#query-with-relative-time-boundaries) Query with relative time boundaries To query data from relative time boundaries, compare the value of the `time`column to a timestamp calculated by subtracting an interval from a timestamp. Use `now()` to return the timestamp for the current time (UTC). ##### Query all data from the last month ```sql SELECT * FROM home WHERE time >= now() - INTERVAL '1 month' ``` ##### Query one day of data data from a week ago ```sql SELECT * FROM home WHERE time >= now() - INTERVAL '7 days' AND time <= now() - INTERVAL '6 days' ``` [](#query-with-absolute-time-boundaries) Query with absolute time boundaries To query data from absolute time boundaries, compare the value of the `time` column to a timestamp literal. Use the `AND` logical operator to chain together multiple predicates and define both start and stop boundaries for the query. ```sql SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' ``` [](#query-data-using-a-time-zone-offset) Query data using a time zone offset To query data using a time zone offset, use the[`AT TIME ZONE` operator](/influxdb3/enterprise/reference/sql/operators/other/#at-time-zone)to apply a time zone offset to timestamps in the `WHERE` clause. > [!Note] > Timestamp types in InfluxDB always represent a UTC time. `AT TIME ZONE` returns > a UTC timestamp adjusted for the offset of the specified time zone. > Timestamps in the `time` column are not updated. > If you need to display the timestamps in your current timezone, this should be > handled client-side. ```sql SELECT * FROM home WHERE time >= '2022-01-01 00:00:00'::TIMESTAMP AT TIME ZONE 'America/Los_Angeles' AND time <= '2022-01-01 12:00:00'::TIMESTAMP AT TIME ZONE 'America/Los_Angeles' ``` ### Query data without time boundaries To query data without time boundaries, do not include any time-based predicates in your `WHERE` clause. > [!Warning] > Querying data *without time bounds* can return an unexpected amount of data. > The query may take a long time to complete and results may be truncated. ```sql SELECT * FROM home ``` ### Query specific fields and tags To query specific fields, include them in the `SELECT` clause. If querying multiple fields or tags, comma-delimit each. If a field or tag key includes special characters or spaces or is case-sensitive, wrap the key in *double-quotes*. ```sql SELECT time, room, temp, hum FROM home ``` ### Query fields based on tag values * Include the fields you want to query and the tags you want to base conditions on in the `SELECT` clause. * Include predicates in the `WHERE` clause that compare the tag identifier to a string literal. Use [logical operators](/influxdb3/enterprise/reference/sql/operators/logical/) to chain multiple predicates together and apply multiple conditions. ```sql SELECT * FROM home WHERE room = 'Kitchen' ``` ### Query points based on field values * In the `SELECT` clause, include fields you want to query. * In the `WHERE` clause, include predicates that compare the field identifier to a value or expression. Use [logical operators](/influxdb3/enterprise/reference/sql/where/#logical-operators) (`AND`, `OR`) to chain multiple predicates together and apply multiple conditions. ```sql SELECT co, time FROM home WHERE co >= 10 OR co <= -10 ``` ### Alias queried fields and tags To alias or rename fields and tags that you query, pass a string literal after the field or tag identifier in the `SELECT` clause. You can use the `AS` clause to define the alias, but it isn’t necessary. The following queries are functionally the same: ```sql SELECT temp 'temperature', hum 'humidity' FROM home SELECT temp AS 'temperature', hum AS 'humidity' FROM home ``` [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/) --- # Cast values to different types Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/cast-types/ Use the `CAST` function or double-colon `::` casting shorthand syntax to cast a value to a specific type. ```sql -- CAST function SELECT CAST(1234.5 AS BIGINT) -- Double-colon casting shorthand SELECT 1234.5::BIGINT ``` * [Cast to a string type](#cast-to-a-string-type) * [Cast to numeric types](#cast-to-numeric-types) * [Float](#cast-to-a-float) * [Integer](#cast-to-an-integer) * [Unsigned integer](#cast-to-an-unsigned-integer) * [Cast to a boolean type](#cast-to-a-boolean-type) * [Cast to a timestamp type](#cast-to-a-timestamp-type) Casting operations can be performed on a column expression or a literal value. For example, the following query uses the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data)and: * Casts all values in the `time` column to integers (Unix nanosecond timestamps). * Casts the literal string value `'1234'` to a 64-bit float for each row. [:: shorthand](#-shorthand)[CAST()](#cast) ```sql SELECT time::BIGINT AS unix_time, '1234'::DOUBLE AS string_to_float FROM home LIMIT 5 ``` ```sql SELECT CAST(time AS BIGINT) AS unix_time, CAST('1234' AS DOUBLE) AS string_to_float FROM home LIMIT 5 ``` | unix\_time |string\_to\_float| |-------------------|-----------------| |1641024000000000000| 1234 | |1641027600000000000| 1234 | |1641031200000000000| 1234 | |1641034800000000000| 1234 | |1641038400000000000| 1234 | ## Cast to a string type Use the `STRING`, `CHAR`, `VARCHAR`, or `TEXT` type in a casting operation to cast a value to a string. [:: shorthand](#-shorthand-2)[CAST()](#cast-2) ```sql value::STRING value::CHAR value::VARCHAR value::TEXT ``` ```sql CAST(value AS STRING) CAST(value AS CHAR) CAST(value AS VARCHAR) CAST(value AS TEXT) ``` SQL supports casting the following to a string value: * **Floats** * **Integers** * **Unsigned integers** * **Booleans** * **Timestamps** ## Cast to numeric types The InfluxDB SQL implementation supports 64-bit floats (`DOUBLE`), integers (`BIGINT`), and unsigned integers (`BIGINT UNSIGNED`). ### Cast to a float Use the `DOUBLE` type in a casting operation to cast a value to a 64-bit float. [:: shorthand](#-shorthand-3)[CAST()](#cast-3) ```sql value::DOUBLE ``` ```sql CAST(value AS DOUBLE) ``` SQL supports casting the following to a float value: * **Strings**: Returns the float equivalent of the numeric string (`[0-9]`). The following string patterns are also supported: * Scientific notation (`'123.4E+10'`) * Infinity (`'±Inf'`) * NaN (`'NaN'`) * **Integers** * **Unsigned integers** ### Cast to an integer Use the `BIGINT` type in a casting operation to cast a value to a 64-bit signed integer. [:: shorthand](#-shorthand-4)[CAST()](#cast-4) ```sql value::BIGINT ``` ```sql CAST(value AS BIGINT) ``` SQL supports casting the following to an integer: * **Strings**: Returns the integer equivalent of the numeric string (`[0-9]`). * **Floats**: Truncates the float value at the decimal. * **Unsigned integers**: Returns the signed integer equivalent of the unsigned integer. * **Booleans**: Returns `1` for `true` and `0` for `false`. * **Timestamps**: Returns the equivalent[nanosecond epoch timestamp](/influxdb3/enterprise/reference/glossary/#unix-timestamp). ### Cast to an unsigned integer Use the `BIGINT UNSIGNED` type in a casting operation to cast a value to a 64-bit unsigned integer. [:: shorthand](#-shorthand-5)[CAST()](#cast-5) ```sql value::BIGINT UNSIGNED ``` ```sql CAST(value AS BIGINT UNSIGNED) ``` SQL supports casting the following to an unsigned integer: * **Strings**: Returns the unsigned integer equivalent of the numeric string (`[0-9]`). * **Floats**: Truncates the float value at the decimal. * **Integers**: Returns the unsigned integer equivalent of the signed integer. * **Booleans**: Returns `1` for `true` and `0` for `false`. * **Timestamps**: Returns the equivalent[nanosecond epoch timestamp](/influxdb3/enterprise/reference/glossary/#unix-timestamp). ## Cast to a boolean type Use the `BOOLEAN` type in a casting operation to cast a value to a boolean. [:: shorthand](#-shorthand-6)[CAST()](#cast-6) ```sql value::BOOLEAN ``` ```sql CAST(value AS BOOLEAN) ``` SQL supports casting the following to a boolean: * **Strings** * Return `true`: * `'true'` *(case-insensitive)* * `'t'`, *(case-insensitive)* * `'1'` * Return `false`: * `'false'` *(case-insensitive)* * `'f'` *(case-insensitive)* * `'0'` * **Integers** * Returns `true`: positive non-zero integer * Returns `false`: `0` * **Unsigned integers** * Returns `true`: non-zero unsigned integer * Returns `false`: `0` ## Cast to a timestamp type Use the `TIMESTAMP` type in a casting operation to cast a value to a timestamp. [:: shorthand](#-shorthand-7)[CAST()](#cast-7) ```sql value::TIMESTAMP ``` ```sql CAST(value AS TIMESTAMP) ``` SQL supports casting the following to a timestamp: * **Strings**: Returns the timestamp equivalent of the string value. The following RFC3339 and RFC339-like string patterns are supported: * `YYYY-MM-DDT00:00:00.000Z` * `YYYY-MM-DDT00:00:00.000-00:00` * `YYYY-MM-DD 00:00:00.000-00:00` * `YYYY-MM-DDT00:00:00Z` * `YYYY-MM-DD 00:00:00.000` * `YYYY-MM-DD 00:00:00` * `YYYY-MM-DD` * **Integers**: Parses the integer as a Unix *second* timestamp and returns the equivalent timestamp. * **Unsigned integers**: Parses the unsigned integer as a Unix nanosecond timestamp and returns the equivalent timestamp. > [!Note] > #### Cast Unix nanosecond timestamps to a timestamp type > > To cast a Unix nanosecond timestamp to a timestamp type, first cast the numeric > value to an unsigned integer (`BIGINT UNSIGNED`) and then a timestamp. > You can also use the [`to_timestamp_nanos`](/influxdb3/enterprise/reference/sql/functions/time-and-date/\> #to\_timestamp\_nanos) > function. > > [:: shorthand](#-shorthand-8)[CAST()](#cast-8)[to\_timestamp\_nanos](#to_timestamp_nanos) > > ``` > 1704067200000000000::BIGINT UNSIGNED::TIMESTAMP > ``` > > ``` > CAST(CAST(1704067200000000000 AS BIGINT UNSIGNED) AS TIMESTAMP) > ``` > > ``` > to_timestamp_nanos(1704067200000000000) > ``` ### Timestamp functions You can also use the following SQL functions to cast a value to a timestamp type: * [`to_timestamp`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#to_timestamp) * [`to_timestamp_millis`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#to_timestamp_millis) * [`to_timestamp_micros`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#to_timestamp_micros) * [`to_timestamp_nanos`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#to_timestamp_nanos) * [`to_timestamp_seconds`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#to_timestamp_seconds) * [to\_unixtime](/influxdb3/enterprise/reference/sql/functions/time-and-date/#to_unixtime) #### Related * [SQL data types](/influxdb3/enterprise/reference/sql/data-types/) [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/) | unix_time | string_to_float | | --- | --- | | unix_time | string_to_float | | 1641024000000000000 | 1234 | | 1641027600000000000 | 1234 | | 1641031200000000000 | 1234 | | 1641034800000000000 | 1234 | | 1641038400000000000 | 1234 | --- # Compare values in SQL queries Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/compare-values/ Use [SQL window functions](/influxdb3/enterprise/reference/sql/functions/window/) to compare values across different rows in your time series data. Window functions like [`LAG`](/influxdb3/enterprise/reference/sql/functions/window/#lag) and [`LEAD`](/influxdb3/enterprise/reference/sql/functions/window/#lead) let you access values from previous or subsequent rows without using self-joins, making it easy to calculate changes over time. Common use cases for comparing values include: * Calculating the difference between the current value and a previous value * Computing rate of change or percentage change * Detecting significant changes or anomalies * Comparing values at specific time intervals * Handling counter metrics that reset to zero **To compare values across rows:** 1. Use a [window function](/influxdb3/enterprise/reference/sql/functions/window/) such as `LAG` or `LEAD` with an `OVER` clause. 2. Include a `PARTITION BY` clause to group data by tags (like `room` or `sensor_id`). 3. Include an `ORDER BY` clause to define the order for comparisons (typically by `time`). 4. Use arithmetic operators to calculate differences, ratios, or percentage changes. ## Examples of comparing values > [!Note] > #### Sample data > > The following examples use the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. * [Calculate the difference from the previous value](#calculate-the-difference-from-the-previous-value) * [Calculate the percentage change](#calculate-the-percentage-change) * [Compare values at regular intervals](#compare-values-at-regular-intervals) * [Compare values with exact time offsets](#compare-values-with-exact-time-offsets) * [Handle counter metrics and resets](#handle-counter-metrics-and-resets) * [Calculate non-negative differences (counter rate)](#calculate-non-negative-differences-counter-rate) * [Calculate cumulative counter increase](#calculate-cumulative-counter-increase) * [Aggregate counter increases by time interval](#aggregate-counter-increases-by-time-interval) ### Calculate the difference from the previous value Use the `LAG` function to access the value from the previous row and calculate the difference. This is useful for detecting changes over time. ```sql SELECT time, room, temp, temp - LAG(temp, 1) OVER ( PARTITION BY room ORDER BY time ) AS temp_change FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|temp\_change| |-------------------|-----------|----|------------| |2022-01-01T08:00:00| Kitchen |21.0| NULL | |2022-01-01T09:00:00| Kitchen |23.0| 2.0 | |2022-01-01T10:00:00| Kitchen |22.7| \-0.3 | |2022-01-01T08:00:00|Living Room|21.1| NULL | |2022-01-01T09:00:00|Living Room|21.4| 0.3 | |2022-01-01T10:00:00|Living Room|21.8| 0.4 | The first row in each partition returns `NULL` for `temp_change` because there’s no previous value. To use a default value instead of `NULL`, provide a third argument to `LAG`: ```sql LAG(temp, 1, 0) -- Returns 0 if no previous value exists ``` ### Calculate the percentage change Calculate the percentage change between the current value and a previous value by dividing the difference by the previous value. ```sql SELECT time, room, temp, ROUND( ((temp - LAG(temp, 1) OVER (PARTITION BY room ORDER BY time)) / LAG(temp, 1) OVER (PARTITION BY room ORDER BY time)) * 100, 2 ) AS percent_change FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|percent\_change| |-------------------|-----------|----|---------------| |2022-01-01T08:00:00| Kitchen |21.0| NULL | |2022-01-01T09:00:00| Kitchen |23.0| 9.52 | |2022-01-01T10:00:00| Kitchen |22.7| \-1.30 | |2022-01-01T08:00:00|Living Room|21.1| NULL | |2022-01-01T09:00:00|Living Room|21.4| 1.42 | |2022-01-01T10:00:00|Living Room|21.8| 1.87 | ### Compare values at regular intervals For regularly spaced time series data (like hourly readings), use `LAG` with an offset parameter to compare values from a specific number of rows back. The following query compares each temperature reading with the reading from one hour earlier (assuming hourly data): ```sql SELECT time, room, temp, LAG(temp, 1) OVER ( PARTITION BY room ORDER BY time ) AS temp_1h_ago, temp - LAG(temp, 1) OVER ( PARTITION BY room ORDER BY time ) AS hourly_change FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T12:00:00Z' ORDER BY room, time ``` | time | room |temp|temp\_1h\_ago|hourly\_change| |-------------------|-----------|----|-------------|--------------| |2022-01-01T08:00:00| Kitchen |21.0| NULL | NULL | |2022-01-01T09:00:00| Kitchen |23.0| 21.0 | 2.0 | |2022-01-01T10:00:00| Kitchen |22.7| 23.0 | \-0.3 | |2022-01-01T11:00:00| Kitchen |22.4| 22.7 | \-0.3 | |2022-01-01T08:00:00|Living Room|21.1| NULL | NULL | |2022-01-01T09:00:00|Living Room|21.4| 21.1 | 0.3 | |2022-01-01T10:00:00|Living Room|21.8| 21.4 | 0.4 | |2022-01-01T11:00:00|Living Room|22.2| 21.8 | 0.4 | ### Compare values with exact time offsets For irregularly spaced time series data or when you need to compare values from an exact time offset (like exactly 1 hour ago, not just the previous row), use a self-join with interval arithmetic. ```sql SELECT current.time, current.room, current.temp AS current_temp, previous.temp AS temp_1h_ago, current.temp - previous.temp AS hourly_diff FROM home AS current LEFT JOIN home AS previous ON current.room = previous.room AND previous.time = current.time - INTERVAL '1 hour' WHERE current.time >= '2022-01-01T08:00:00Z' AND current.time < '2022-01-01T12:00:00Z' ORDER BY current.room, current.time ``` | time | room |current\_temp|temp\_1h\_ago|hourly\_diff| |-------------------|-----------|-------------|-------------|------------| |2022-01-01T08:00:00| Kitchen | 21.0 | NULL | NULL | |2022-01-01T09:00:00| Kitchen | 23.0 | 21.0 | 2.0 | |2022-01-01T10:00:00| Kitchen | 22.7 | 23.0 | \-0.3 | |2022-01-01T11:00:00| Kitchen | 22.4 | 22.7 | \-0.3 | |2022-01-01T08:00:00|Living Room| 21.1 | NULL | NULL | |2022-01-01T09:00:00|Living Room| 21.4 | 21.1 | 0.3 | |2022-01-01T10:00:00|Living Room| 21.8 | 21.4 | 0.4 | |2022-01-01T11:00:00|Living Room| 22.2 | 21.8 | 0.4 | This self-join approach works when: * Your data points don’t fall at regular intervals * You need to compare against a specific time offset regardless of when the previous data point occurred * You want to ensure the comparison is against a value from exactly 1 hour ago (or any other specific interval) ## Handle counter metrics and resets Counter metrics track cumulative values that increase over time, such as total requests, bytes transferred, or errors. Unlike gauge metrics (which can go up or down), counters typically only increase, though they may reset to zero when a service restarts. Use [`GREATEST`](/influxdb3/enterprise/reference/sql/functions/conditional/#greatest) with `LAG` to handle counter resets by treating negative differences as zero. > [!Note] > #### InfluxDB 3 SQL and counter metrics > > InfluxDB 3 SQL doesn’t provide built-in equivalents to Flux’s `increase()`or InfluxQL’s `NON_NEGATIVE_DIFFERENCE()` functions. > Use the patterns shown below to achieve similar results. ### Calculate non-negative differences (counter rate) Calculate the increase between consecutive counter readings, treating negative differences (counter resets) as zero. ```sql SELECT time, host, requests, LAG(requests) OVER (PARTITION BY host ORDER BY time) AS prev_requests, GREATEST( requests - LAG(requests) OVER (PARTITION BY host ORDER BY time), 0 ) AS requests_increase FROM metrics WHERE host = 'server1' ORDER BY time ``` | time | host |requests|prev\_requests|requests\_increase| |-------------------|-------|--------|--------------|------------------| |2024-01-01T00:00:00|server1| 1000 | NULL | 0 | |2024-01-01T01:00:00|server1| 1250 | 1000 | 250 | |2024-01-01T02:00:00|server1| 1600 | 1250 | 350 | |2024-01-01T03:00:00|server1| 50 | 1600 | 0 | |2024-01-01T04:00:00|server1| 300 | 50 | 250 | `LAG(requests)` retrieves the previous counter value, `requests - LAG(requests)` calculates the difference, and `GREATEST(..., 0)` returns 0 for negative differences (counter resets).`PARTITION BY host` ensures comparisons are only within the same host. ### Calculate cumulative counter increase Calculate the total increase in a counter over time, handling resets. Use a Common Table Expression (CTE) to first calculate the differences, then sum them. ```sql WITH counter_diffs AS ( SELECT time, host, requests, GREATEST( requests - LAG(requests) OVER (PARTITION BY host ORDER BY time), 0 ) AS requests_increase FROM metrics WHERE host = 'server1' ) SELECT time, host, requests, SUM(requests_increase) OVER (PARTITION BY host ORDER BY time) AS cumulative_increase FROM counter_diffs ORDER BY time ``` | time | host |requests|cumulative\_increase| |-------------------|-------|--------|--------------------| |2024-01-01T00:00:00|server1| 1000 | 0 | |2024-01-01T01:00:00|server1| 1250 | 250 | |2024-01-01T02:00:00|server1| 1600 | 600 | |2024-01-01T03:00:00|server1| 50 | 600 | |2024-01-01T04:00:00|server1| 300 | 850 | The CTE computes non-negative differences for each row, then `SUM(requests_increase) OVER (...)` creates a running total. The cumulative increase continues to grow despite the counter reset at 03:00. ### Aggregate counter increases by time interval Calculate the total increase in a counter for each time interval (for example, hourly totals). ```sql WITH counter_diffs AS ( SELECT DATE_BIN(INTERVAL '1 hour', time) AS time_bucket, host, requests, GREATEST( requests - LAG(requests) OVER (PARTITION BY host ORDER BY time), 0 ) AS requests_increase FROM metrics ) SELECT time_bucket, host, SUM(requests_increase) AS total_increase FROM counter_diffs WHERE requests_increase > 0 GROUP BY time_bucket, host ORDER BY host, time_bucket ``` | time\_bucket | host |total\_increase| |-------------------|-------|---------------| |2024-01-01T01:00:00|server1| 250 | |2024-01-01T02:00:00|server1| 350 | |2024-01-01T04:00:00|server1| 250 | |2024-01-01T01:00:00|server2| 400 | |2024-01-01T02:00:00|server2| 500 | |2024-01-01T03:00:00|server2| 300 | |2024-01-01T04:00:00|server2| 400 | The CTE calculates differences for each row.`DATE_BIN()` assigns each timestamp to a 1-hour interval, `SUM(requests_increase)` aggregates all increases within each interval, and `WHERE requests_increase > 0` filters out zero increases (first row and counter resets). #### Related * [SQL window functions](/influxdb3/enterprise/reference/sql/functions/window/) * [Aggregate data with SQL](/influxdb3/enterprise/query-data/sql/aggregate-select/) [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/)[window functions](/influxdb3/enterprise/tags/window-functions/) | time | room | temp | temp_change | | --- | --- | --- | --- | | time | room | temp | temp_change | | 2022-01-01T08:00:00 | Kitchen | 21.0 | NULL | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 2.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | -0.3 | | 2022-01-01T08:00:00 | Living Room | 21.1 | NULL | | 2022-01-01T09:00:00 | Living Room | 21.4 | 0.3 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 0.4 | | time | room | temp | percent_change | | --- | --- | --- | --- | | time | room | temp | percent_change | | 2022-01-01T08:00:00 | Kitchen | 21.0 | NULL | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 9.52 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | -1.30 | | 2022-01-01T08:00:00 | Living Room | 21.1 | NULL | | 2022-01-01T09:00:00 | Living Room | 21.4 | 1.42 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 1.87 | | time | room | temp | temp_1h_ago | hourly_change | | --- | --- | --- | --- | --- | | time | room | temp | temp_1h_ago | hourly_change | | 2022-01-01T08:00:00 | Kitchen | 21.0 | NULL | NULL | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 21.0 | 2.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 23.0 | -0.3 | | 2022-01-01T11:00:00 | Kitchen | 22.4 | 22.7 | -0.3 | | 2022-01-01T08:00:00 | Living Room | 21.1 | NULL | NULL | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.1 | 0.3 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 21.4 | 0.4 | | 2022-01-01T11:00:00 | Living Room | 22.2 | 21.8 | 0.4 | | time | room | current_temp | temp_1h_ago | hourly_diff | | --- | --- | --- | --- | --- | | time | room | current_temp | temp_1h_ago | hourly_diff | | 2022-01-01T08:00:00 | Kitchen | 21.0 | NULL | NULL | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 21.0 | 2.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 23.0 | -0.3 | | 2022-01-01T11:00:00 | Kitchen | 22.4 | 22.7 | -0.3 | | 2022-01-01T08:00:00 | Living Room | 21.1 | NULL | NULL | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.1 | 0.3 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 21.4 | 0.4 | | 2022-01-01T11:00:00 | Living Room | 22.2 | 21.8 | 0.4 | | time | host | requests | prev_requests | requests_increase | | --- | --- | --- | --- | --- | | time | host | requests | prev_requests | requests_increase | | 2024-01-01T00:00:00 | server1 | 1000 | NULL | 0 | | 2024-01-01T01:00:00 | server1 | 1250 | 1000 | 250 | | 2024-01-01T02:00:00 | server1 | 1600 | 1250 | 350 | | 2024-01-01T03:00:00 | server1 | 50 | 1600 | 0 | | 2024-01-01T04:00:00 | server1 | 300 | 50 | 250 | | time | host | requests | cumulative_increase | | --- | --- | --- | --- | | time | host | requests | cumulative_increase | | 2024-01-01T00:00:00 | server1 | 1000 | 0 | | 2024-01-01T01:00:00 | server1 | 1250 | 250 | | 2024-01-01T02:00:00 | server1 | 1600 | 600 | | 2024-01-01T03:00:00 | server1 | 50 | 600 | | 2024-01-01T04:00:00 | server1 | 300 | 850 | | time_bucket | host | total_increase | | --- | --- | --- | | time_bucket | host | total_increase | | 2024-01-01T01:00:00 | server1 | 250 | | 2024-01-01T02:00:00 | server1 | 350 | | 2024-01-01T04:00:00 | server1 | 250 | | 2024-01-01T01:00:00 | server2 | 400 | | 2024-01-01T02:00:00 | server2 | 500 | | 2024-01-01T03:00:00 | server2 | 300 | | 2024-01-01T04:00:00 | server2 | 400 | --- # Explore your schema with SQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/explore-schema/ Use SQL to explore your data schema in your InfluxDB 3 Enterprise database. ## List tables in a database Use `SHOW TABLES` to list tables in your InfluxDB database. ```sql SHOW TABLES ``` [](#view-example-output) View example output Tables listed with the `table_schema` of `iox` are tables. Tables with `system` or `information_schema` table schemas are system tables that store internal metadata. |table\_catalog| table\_schema |table\_name |table\_type| |--------------|-------------------|------------|-----------| | public | iox | home |BASE TABLE | | public | iox | noaa |BASE TABLE | | public | system | queries |BASE TABLE | | public |information\_schema| tables | VIEW | | public |information\_schema| views | VIEW | | public |information\_schema| columns | VIEW | | public |information\_schema|df\_settings| VIEW | ## List columns in a table Use the `SHOW COLUMNS` statement to view what columns are in a table. Use the `IN` clause to specify the table. ```sql SHOW COLUMNS IN home ``` [](#view-example-output) View example output |table\_catalog|table\_schema|table\_name|column\_name| data\_type |is\_nullable| |--------------|-------------|-----------|------------|---------------------------|------------| | public | iox | home | co | Int64 | YES | | public | iox | home | hum | Float64 | YES | | public | iox | home | room | Dictionary(Int32, Utf8) | YES | | public | iox | home | temp | Float64 | YES | | public | iox | home | time |Timestamp(Nanosecond, None)| NO | [query](/influxdb3/enterprise/tags/query/)[sql](/influxdb3/enterprise/tags/sql/) | table_catalog | table_schema | table_name | table_type | | --- | --- | --- | --- | | table_catalog | table_schema | table_name | table_type | | public | iox | home | BASE TABLE | | public | iox | noaa | BASE TABLE | | public | system | queries | BASE TABLE | | public | information_schema | tables | VIEW | | public | information_schema | views | VIEW | | public | information_schema | columns | VIEW | | public | information_schema | df_settings | VIEW | | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | | --- | --- | --- | --- | --- | --- | | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | | public | iox | home | co | Int64 | YES | | public | iox | home | hum | Float64 | YES | | public | iox | home | room | Dictionary(Int32, Utf8) | YES | | public | iox | home | temp | Float64 | YES | | public | iox | home | time | Timestamp(Nanosecond, None) | NO | --- # Fill gaps in data Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/fill-gaps/ Use [`date_bin_gapfill`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill)with [`interpolate`](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate)or [`locf`](/influxdb3/enterprise/reference/sql/functions/misc/#locf) to fill gaps of time where no data is returned. Gap-filling SQL queries handle missing data in time series data by filling in gaps with interpolated values or by carrying forward the last available observation. **To fill gaps in data:** 1. Use the `date_bin_gapfill` function to window your data into time-based groups and apply an [aggregate function](/influxdb3/enterprise/reference/sql/functions/aggregate/)to each window. If no data exists in a window, `date_bin_gapfill` inserts a new row with the starting timestamp of the window, all columns in the`GROUP BY` clause populated, and null values for the queried fields. 2. Use either `interpolate` or `locf` to fill the inserted null values in the specified column. * **interpolate**: fills null values by interpolating values between non-null values. * **locf**: fills null values by carrying the last observed value forward. > [!Note] > The expression passed to `interpolate` or `locf` must use an[aggregate function](/influxdb3/enterprise/reference/sql/functions/aggregate/). 3. Include a `WHERE` clause that sets upper and lower time bounds. For example: ```sql WHERE time \>= '2022-01-01T08:00:00Z' AND time \<= '2022-01-01T10:00:00Z' ``` ## Example of filling gaps in data The following examples use the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data)to show how to use `date_bin_gapfill` and the different results of `interplate`and `locf`. [interpolate](#interpolate)[locf](#locf) ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as time, room, interpolate(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY 1, room ``` | time | room |AVG(home.temp)| |--------------------|-----------|--------------| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | 22 | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | 22.85 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.1 | |2022-01-01T08:30:00Z|Living Room| 21.25 | |2022-01-01T09:00:00Z|Living Room| 21.4 | |2022-01-01T09:30:00Z|Living Room| 21.6 | |2022-01-01T10:00:00Z|Living Room| 21.8 | ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as time, room, locf(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY 1, room ``` | time | room |AVG(home.temp)| |--------------------|-----------|--------------| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | 21 | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | 23 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.1 | |2022-01-01T08:30:00Z|Living Room| 21.1 | |2022-01-01T09:00:00Z|Living Room| 21.4 | |2022-01-01T09:30:00Z|Living Room| 21.4 | |2022-01-01T10:00:00Z|Living Room| 21.8 | | time | room | AVG(home.temp) | | --- | --- | --- | | time | room | AVG(home.temp) | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | 22 | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | 22.85 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | 21.25 | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | 21.6 | | 2022-01-01T10:00:00Z | Living Room | 21.8 | | time | room | AVG(home.temp) | | --- | --- | --- | | time | room | AVG(home.temp) | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | 21 | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | 23 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | 21.1 | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | 21.4 | | 2022-01-01T10:00:00Z | Living Room | 21.8 | --- # Use parameterized queries with SQL Source: https://docs.influxdata.com/influxdb3/enterprise/query-data/sql/parameterized-queries/ Parameterized queries in InfluxDB 3 Enterprise let you dynamically and safely change values in a query. If your application code allows user input to customize values or expressions in a query, use a parameterized query to make sure untrusted input is processed strictly as data and not executed as code. Parameterized queries: * help prevent injection attacks, which can occur if input is executed as code * help make queries more reusable > [!Note] > #### Prevent injection attacks > > For more information on security and query parameterization, > see the [OWASP SQL Injection Prevention Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html#defense-option-1-prepared-statements-with-parameterized-queries). In InfluxDB 3, a parameterized query is an InfluxQL or SQL query that contains one or more named parameter placeholders–variables that represent input data. * [Use parameters in `WHERE` expressions](#use-parameters-in-where-expressions) * [Parameter data types](#parameter-data-types) * [Data type examples](#data-type-examples) * [Time expressions](#time-expressions) * [Not compatible with parameters](#not-compatible-with-parameters) * [Parameterize an SQL query](#parameterize-an-sql-query) * [Execute parameterized SQL queries](#execute-parameterized-sql-queries) * [Use the HTTP API](#use-the-http-api) * [Use InfluxDB Flight RPC clients](#use-influxdb-flight-rpc-clients) * [Client support for parameterized queries](#client-support-for-parameterized-queries) * [Not supported](#not-supported) > [!Note] > #### Parameters only supported in `WHERE` expressions > > InfluxDB 3 supports parameters in `WHERE` clause **predicate expressions**. > Parameter values must be one of the [allowed parameter data types](#parameter-data-types). > > If you use parameters in other expressions or clauses, > such as function arguments, `SELECT`, or `GROUP BY`, then your query might not work as you expect. ## Use parameters in `WHERE` expressions You can use parameters in `WHERE` clause **predicate expressions**-–for example, the following query contains a `$temp` parameter: ```sql SELECT * FROM measurement WHERE temp > $temp ``` When executing a query, you specify parameter name-value pairs. The value that you assign to a parameter must be one of the [parameter data types](#parameter-data-types). ```go {"temp": 22.0} ``` The InfluxDB Querier parses the query text with the parameter placeholders, and then generates query plans that replace the placeholders with the values that you provide. This separation of query structure from input data ensures that input is treated as one of the allowed [data types](#parameter-data-types) and not as executable code. ## Parameter data types A parameter value can be one of the following data types: * Null * Boolean * Unsigned integer (`u_int64`) * Integer (`int64`) * Double (`float64`) * String ### Data type examples ```js { "string": "Living Room", "double": 3.14, "unsigned_integer": 1234, "integer": -1234, "boolean": false, "null": Null, } ``` ### Time expressions To parameterize time bounds, substitute a parameter for a timestamp literal–for example: ```sql SELECT * FROM home WHERE time >= $min_time ``` For the parameter value, specify the timestamp literal as a string–for example: ```go // Assign a timestamp string literal to the min_time parameter. parameters := influxdb3.QueryParameters{ "min_time": "2022-01-01 00:00:00.00", } ``` InfluxDB executes the query as the following: ```sql SELECT * FROM home WHERE time >= '2022-01-01 00:00:00.00' ``` ### Not compatible with parameters If you use parameters for the following, your query might not work as you expect: * In clauses other than `WHERE`, such as `SELECT` or `GROUP BY` * As function arguments, such as `avg($temp)` * In place of identifiers, such as column or table names * In place of duration literals, such as `INTERVAL $minutes` ## Parameterize an SQL query > [!Note] > #### Sample data > > The following examples use the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-the-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. To use a parameterized query, do the following: 1. In your query text, use the `$parameter` syntax to reference a parameter name–for example, the following query contains `$room` and `$min_temp` parameter placeholders: ``` SELECT * FROM home WHERE time > now() - INTERVAL '7 days' AND temp >= $min_temp AND room = $room ``` 2. Provide a value for each parameter name. If you don’t assign a value for a parameter, InfluxDB returns an error. The syntax for providing parameter values depends on the client you use–for example: [Go](#go) ``` // Define a QueryParameters struct--a map of parameters to input values. parameters := influxdb3.QueryParameters{ "room": "Kitchen", "min_temp": 20.0, } ``` After InfluxDB receives your request and parses the query, it executes the query as ```sql SELECT * FROM home WHERE time > now() - INTERVAL '7 days' AND temp >= 20.0 AND room = 'Kitchen' ``` ## Execute parameterized SQL queries > [!Note] > #### Sample data > > The following examples use the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). > To run the example queries and return results,[write the sample data](/influxdb3/enterprise/reference/sample-data/#write-the-home-sensor-data-to-influxdb)to your InfluxDB 3 Enterprise database before running the example queries. ### Use the HTTP API InfluxDB 3 Enterprise provides the `/api/v3/query_sql` HTTP API endpoint for executing SQL queries with parameters. ``` POST /api/v3/query_sql ``` Send a JSON object that contains `db` (database), `q` (query), and `params` (parameter name-value pairs) properties in the request body. The following example sends a parameterized SQL query to the `/api/v3/query_sql` endpoint: ```bash curl "http://localhost:8181/api/v3/query_sql" \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: application/json" \ --data '{ "db": "DATABASE_NAME", "q": "SELECT * FROM home WHERE time >= $min_time AND temp >= $min_temp AND room = $room", "params": { "min_time": "2022-01-01T08:00:00Z", "min_temp": 22.0, "room": "Kitchen" } }' ``` Replace the following: * `DATABASE_NAME`: the name of the database to query * `AUTH_TOKEN`: your [database token](/influxdb3/enterprise/admin/tokens/)with permission to query the specified database The response body contains query results in JSON format: ```json [ {"co":0,"hum":36.2,"room":"Kitchen","temp":23.0,"time":"2022-01-01T09:00:00"}, {"co":0,"hum":36.1,"room":"Kitchen","temp":22.7,"time":"2022-01-01T10:00:00"}, {"co":0,"hum":36.0,"room":"Kitchen","temp":22.4,"time":"2022-01-01T11:00:00"}, {"co":0,"hum":36.0,"room":"Kitchen","temp":22.5,"time":"2022-01-01T12:00:00"} ] ``` ### Use InfluxDB Flight RPC clients Using the InfluxDB 3 native Flight RPC protocol and supported clients, you can send a parameterized query and a list of parameter name-value pairs. InfluxDB Flight clients that support parameterized queries pass the parameter name-value pairs in a Flight ticket `params` field. The following examples show how to use client libraries to execute parameterized SQL queries: [Go](#go-2) ```go import ( "context" "fmt" "io" "os" "text/tabwriter" "time" "github.com/InfluxCommunity/influxdb3-go/v2/influxdb3" ) func Query(query string, parameters influxdb3.QueryParameters) error { url := os.Getenv("INFLUX_HOST") token := os.Getenv("INFLUX_TOKEN") database := os.Getenv("INFLUX_DATABASE") // Instantiate the influxdb3 client. client, err := influxdb3.New(influxdb3.ClientConfig{ Host: url, Token: token, Database: database, }) if err != nil { panic(err) } // Ensure the client is closed after the Query function finishes. defer func(client *influxdb3.Client) { err := client.Close() if err != nil { panic(err) } }(client) // Call the client's QueryWithParameters function. // Provide the query and parameters. The default QueryType is SQL. iterator, err := client.QueryWithParameters(context.Background(), query, parameters) // Create a buffer for storing rows as you process them. w := tabwriter.NewWriter(io.Discard, 4, 4, 1, ' ', 0) w.Init(os.Stdout, 0, 8, 0, '\t', 0) fmt.Fprintf(w, "time\troom\tco\thum\ttemp\n") // Format and write each row to the buffer. // Process each row as key-value pairs. for iterator.Next() { row := iterator.Value() // Use Go time package to format unix timestamp // as a time with timezone layout (RFC3339 format) time := (row["time"].(time.Time)). Format(time.RFC3339) fmt.Fprintf(w, "%s\t%s\t%d\t%.1f\t%.1f\n", time, row["room"], row["co"], row["hum"], row["temp"]) } w.Flush() return nil } func main() { // Use the $placeholder syntax in a query to reference parameter placeholders // for input data. // The following SQL query contains the placeholders $room and $min_temp. query := ` SELECT * FROM home WHERE time > now() - INTERVAL '7 days' AND temp >= $min_temp AND room = $room` // Define a QueryParameters struct--a map of placeholder names to input values. parameters := influxdb3.QueryParameters{ "room": "Kitchen", "min_temp": 20.0, } } ``` ## Client support for parameterized queries * Not all [InfluxDB 3 Flight clients](/influxdb3/enterprise/reference/client-libraries/v3/) support parameterized queries. * InfluxDB doesn’t currently support parameterized queries or DataFusion prepared statements for Flight SQL or Flight SQL clients. * InfluxDB 3 SQL and InfluxQL parameterized queries aren’t supported in InfluxDB v1 and v2 clients. ## Not supported Currently, parameterized queries in InfluxDB 3 Enterprise don’t provide the following: * support for DataFusion prepared statements * query caching, optimization, or performance benefits [query](/influxdb3/enterprise/tags/query/)[security](/influxdb3/enterprise/tags/security/)[sql](/influxdb3/enterprise/tags/sql/) --- # InfluxDB 3 Enterprise reference documentation Source: https://docs.influxdata.com/influxdb3/enterprise/reference/ ### [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) Configure InfluxDB 3 Enterprise with influxdb3 serve options or environment variables. ### [Command line tools](/influxdb3/enterprise/reference/cli/) View command line tools used to manage and interact with InfluxDB 3 Enterprise. ### [Line protocol reference](/influxdb3/enterprise/reference/line-protocol/) InfluxDB 3 Enterprise uses line protocol to write data points. It is a text-based format that provides the table, tag set, field set, and timestamp of a data point. ### [Processing engine reference](/influxdb3/enterprise/reference/processing-engine/) The InfluxDB 3 Processing engine is an embedded Python virtual machine that runs inside InfluxDB 3 Enterprise to execute Python code in response to triggers you define without requiring external application servers or middleware. ### [SQL reference documentation](/influxdb3/enterprise/reference/sql/) Learn the SQL syntax and structure used to query InfluxDB. ### [InfluxQL reference documentation](/influxdb3/enterprise/reference/influxql/) InfluxQL is an SQL-like query language for interacting with data in InfluxDB. ### [Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/) Complete reference for all configuration options available with the upgraded InfluxDB 3 Enterprise storage engine, including WAL, snapshot, compaction, caching, and replication settings. ### [API client libraries](/influxdb3/enterprise/reference/client-libraries/) InfluxDB client libraries are language-specific tools that integrate with InfluxDB APIs. View the list of available client libraries. ### [InfluxDB 3 Enterprise internals](/influxdb3/enterprise/reference/internals/) Learn about InfluxDB 3 Enterprise internal systems and mechanisms. ### [Naming restrictions and conventions](/influxdb3/enterprise/reference/naming-restrictions/) Learn about naming restrictions and conventions for databases, tables, tags, fields, and other identifiers in InfluxDB 3 Enterprise. ### [Usage telemetry](/influxdb3/enterprise/reference/telemetry/) InfluxData collects telemetry data to help improve the InfluxDB 3 Enterprise. Learn what data InfluxDB 3 Enterprise collects and sends to InfluxData, how it’s used, and how you can opt out. ### [Glossary](/influxdb3/enterprise/reference/glossary/) Terms related to InfluxData products and platforms. ### [Sample data](/influxdb3/enterprise/reference/sample-data/) Sample datasets are used throughout the the InfluxDB 3 Enterprise documentation to demonstrate functionality. Use the following sample datasets to replicate provided examples. ### [InfluxDB HTTP API](/influxdb3/enterprise/reference/api/) The InfluxDB HTTP API for InfluxDB 3 Enterprise provides a programmatic interface for interactions with InfluxDB. --- # InfluxDB HTTP API Source: https://docs.influxdata.com/influxdb3/enterprise/reference/api/ This page has moved to [InfluxDB HTTP API](/influxdb3/enterprise/api/). --- # Command line tools Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/ View command line tools used to run, manage, and interact with InfluxDB 3 Enterprise: ### [influxdb3](/influxdb3/enterprise/reference/cli/influxdb3/) The `influxdb3` CLI runs and interacts with the InfluxDB 3 Enterprise server. --- # influxdb3 CLI Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/ The `influxdb3` CLI runs and interacts with the InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 [GLOBAL-OPTIONS] [COMMAND] ``` ## Commands | Command | Description | |-----------------------------------------------------------------|--------------------------------------| | [create](/influxdb3/enterprise/reference/cli/influxdb3/create/) | Create resources | | [debug](/influxdb3/enterprise/reference/cli/influxdb3/debug/) | Diagnostic tools | | [delete](/influxdb3/enterprise/reference/cli/influxdb3/delete/) | Delete resources | |[disable](/influxdb3/enterprise/reference/cli/influxdb3/disable/)| Disable resources | | [enable](/influxdb3/enterprise/reference/cli/influxdb3/enable/) | Enable resources | | [import](/influxdb3/enterprise/reference/cli/influxdb3/import/) | Manage bulk Parquet imports | |[install](/influxdb3/enterprise/reference/cli/influxdb3/install/)| Install plugins | |[loadcap](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/)| Capture and manage workload profiles | | [manage](/influxdb3/enterprise/reference/cli/influxdb3/manage/) |Perform cluster maintenance operations| | [query](/influxdb3/enterprise/reference/cli/influxdb3/query/) | Query InfluxDB 3 Enterprise | | [remove](/influxdb3/enterprise/reference/cli/influxdb3/remove/) | Remove stopped nodes | | [serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) | Run the InfluxDB 3 Enterprise server | | [show](/influxdb3/enterprise/reference/cli/influxdb3/show/) | List resources | | [stop](/influxdb3/enterprise/reference/cli/influxdb3/stop/) | Gracefully stop nodes | | [test](/influxdb3/enterprise/reference/cli/influxdb3/test/) | Test plugins | | [update](/influxdb3/enterprise/reference/cli/influxdb3/update/) | Update resources | | [write](/influxdb3/enterprise/reference/cli/influxdb3/write/) | Write to InfluxDB 3 Enterprise | ## Global options |Option| | Description | |------|------------|-----------------------------------------------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information including runtime configuration options| | `-V` |`--version` | Print version | For advanced global configuration options (including `--num-io-threads` and other runtime settings), see [Configuration options](/influxdb3/enterprise/reference/config-options/#global-configuration-options). ## Quick-Start Mode For development, testing, and home use, you can start InfluxDB 3 Enterprise by running `influxdb3` without the `serve` subcommand or any configuration parameters. The system automatically generates required values: * **`node-id`**: `{hostname}-node` (fallback: `primary-node`) * **`cluster-id`**: `{hostname}-cluster` (fallback: `primary-cluster`) * **`object-store`**: `file` * **`data-dir`**: `~/.influxdb` The system displays warning messages showing the auto-generated identifiers: ``` Using auto-generated node id: mylaptop-node. For production deployments, explicitly set --node-id Using auto-generated cluster id: mylaptop-cluster. For production deployments, explicitly set --cluster-id ``` > [!Important] > #### Production deployments > > Quick-start mode is designed for development and testing environments. > For production deployments, use explicit configuration with the `serve` subcommand > and specify all required parameters as shown in the [Examples](#examples) below. **Configuration precedence**: CLI flags \> environment variables \> auto-generated defaults For more information about quick-start mode, see [Get started](/influxdb3/enterprise/get-started/setup/#quick-start-mode-development). ## Examples In the examples below, replace the following: * `my-host-01`: a unique identifier for your InfluxDB 3 Enterprise server. * `my-cluster-01`: a unique identifier for your InfluxDB 3 Enterprise cluster. The value you use must be different from `--node-id` values in the cluster. ### Quick-start influxdb3 server ```bash # Zero-config startup influxdb3 # Override specific defaults influxdb3 --object-store memory # Use environment variables to override defaults INFLUXDB3_NODE_ID=my-node influxdb3 ``` ### Run the InfluxDB 3 server ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 ``` ### Run the InfluxDB 3 server with custom IO threads ```bash influxdb3 --num-io-threads=8 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 ``` ### Display short-form help for all commands ```bash influxdb3 -h ``` ### Display long-form help for all commands ```bash influxdb3 --help ``` ### Run the InfluxDB 3 Enterprise server with extra verbose logging ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 \ --verbose ``` ### Run InfluxDB 3 Enterprise with debug logging using LOG\_FILTER ```bash LOG_FILTER=debug influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 ``` | Command | Description | | --- | --- | | Command | Description | | create | Create resources | | debug | Diagnostic tools | | delete | Delete resources | | disable | Disable resources | | enable | Enable resources | | import | Manage bulk Parquet imports | | install | Install plugins | | loadcap | Capture and manage workload profiles | | manage | Perform cluster maintenance operations | | query | Query InfluxDB 3 Enterprise | | remove | Remove stopped nodes | | serve | Run the InfluxDB 3 Enterprise server | | show | List resources | | stop | Gracefully stop nodes | | test | Test plugins | | update | Update resources | | write | Write to InfluxDB 3 Enterprise | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information including runtime configuration options | | -V | --version | Print version | --- # influxdb3 create Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/ The `influxdb3 create` command creates a resource such as a database or authentication token. ## Usage ```bash influxdb3 create ``` ## Subcommands | Subcommand | Description | |---------------------------------------------------------------------------------------|-----------------------------------------------| | [database](/influxdb3/enterprise/reference/cli/influxdb3/create/database/) | Create a new database | | [file\_index](/influxdb3/enterprise/reference/cli/influxdb3/create/file_index/) |Create a new file index for a database or table| | [last\_cache](/influxdb3/enterprise/reference/cli/influxdb3/create/last_cache/) | Create a new last value cache | |[distinct\_cache](/influxdb3/enterprise/reference/cli/influxdb3/create/distinct_cache/)| Create a new distinct value cache | | [table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) | Create a new table in a database | | [token](/influxdb3/enterprise/reference/cli/influxdb3/create/token/) | Create a new authentication token | | [trigger](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/) |Create a new trigger for the processing engine | | query\_group | Create a new distributed query group (not yet operational) | | help | Print command help or the help of a subcommand | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | database | Create a new database | | file_index | Create a new file index for a database or table | | last_cache | Create a new last value cache | | distinct_cache | Create a new distinct value cache | | table | Create a new table in a database | | token | Create a new authentication token | | trigger | Create a new trigger for the processing engine | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 create database Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/database/ The `influxdb3 create database` command creates a new database in your InfluxDB 3 Enterprise instance. Provide a database name and, optionally, specify connection settings and authentication credentials using flags or environment variables. ## Usage ```bash # Syntax influxdb3 create database [OPTIONS] ``` ## Arguments * **`DATABASE_NAME`**: The name of the database to create. Valid database names are alphanumeric and start with a letter or number. Dashes (-) and underscores (\_) are allowed. You can also set the database name using the `INFLUXDB3_DATABASE_NAME` environment variable. ## Options |Option| | Description | |------|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | |`--retention-period`|Database [retention period](/influxdb3/enterprise/reference/glossary/#retention-period) ([duration](/influxdb3/enterprise/reference/glossary/#duration) value, for example: `30d`, `24h`, `1h`)| | | `--schema-mode` | [Schema mode](#schema-mode) for tables in the database: `implicit` (default) or `explicit`. Fixed when the database is created | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | `--tls-no-verify` | Disable TLS certificate verification. **Not recommended in production.** Useful for testing with self-signed certificates | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables instead of providing CLI options directly: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples The following examples show how to create a database. In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token ### Create a database (default) Creates a database using settings from environment variables and defaults. ```bash influxdb3 create database DATABASE_NAME ``` ### Create a database with an authentication token Creates a database using the specified arguments. Flags override their associated environment variables. ```bash influxdb3 create database --token AUTH_TOKEN DATABASE_NAME ``` ### Create a database with a retention period Creates a database with a 30-day retention period. Data older than 30 days will not be queryable. ```bash influxdb3 create database --retention-period 30d DATABASE_NAME ``` ### Create a database with infinite retention Creates a database with no retention period (data never expires). ```bash influxdb3 create database --retention-period none DATABASE_NAME ``` ### Create a database with a 90-day retention period Creates a database with a 90-day retention period using an authentication token. ```bash influxdb3 create database \ --retention-period 90d \ --token AUTH_TOKEN \ DATABASE_NAME ``` ### Create a database with a 1-year retention period Creates a database with a 1-year retention period. ```bash influxdb3 create database --retention-period 1y DATABASE_NAME ``` ### Create a database with a combined duration Creates a database with a retention period of 30 days and 12 hours. ```bash influxdb3 create database --retention-period 30d12h DATABASE_NAME ``` ### Create a database with explicit schema mode Creates a database that rejects writes to tables or columns that you haven’t declared with the `influxdb3 create table` command or the`/api/v3/configure/table` API. ```bash influxdb3 create database --schema-mode explicit DATABASE_NAME ``` ## Retention period duration formats Retention periods are specified as [duration](/influxdb3/enterprise/reference/glossary/#duration)values using a numeric value plus a duration unit. ### Valid duration units |Unit| Description | |----|---------------| |`h` | hour | |`d` | day | |`w` | week | |`mo`|month (30 days)| |`y` |year (365 days)| > [!Note] > Minute (`m`) and second (`s`) units are not supported for retention periods. > [!Warning] > #### Retention period constraints > > * **Minimum for data retention**: The practical minimum retention period is 1 hour (`1h`). > * **Zero-duration periods**: Setting a retention period to `0` (for example,`0d` or `0h`) is allowed but marks all data for immediate deletion at query time.*This differs from InfluxDB 1.x and 2.x where `0d` meant infinite retention.* > * **Infinite retention**: Use `none` to set an infinite retention period. ### Example duration values * `1h` - 1 hour * `24h` - 24 hours * `7d` - 7 days * `4w` - 4 weeks * `30d` - 30 days * `1mo` - 1 month (30 days) * `90d` - 90 days * `1y` - 1 year (365 days) * `none` - infinite (data never expires) You can combine units: `30d12h` (30.5 days), `1y6mo` (545 days) For complete details about retention periods, see[Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/). ## Schema mode The `--schema-mode` option controls where table and column definitions come from: * **`implicit`** *(default)*: Tables and columns are created automatically from the line protocol you write. * **`explicit`**: Tables and columns must be declared with the[`influxdb3 create table` command](/influxdb3/enterprise/reference/cli/influxdb3/create/table/)or the `/api/v3/configure/table` API before you can write to them. A write that names an undeclared table or column is rejected. Schema mode applies to every table in the database and is fixed when you create the database. You can’t change a database’s schema mode afterward. > [!Note] > Explicit schema mode is available in InfluxDB 3 Enterprise only. > > For more information, see[Enforce a schema](/influxdb3/enterprise/admin/databases/enforce-schema/). #### Related * [Create a database](/influxdb3/enterprise/admin/databases/create/) * [Create database API](/influxdb3/enterprise/api/database/#operation/PostConfigureDatabase) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --retention-period | Database retention period ( duration value, for example: 30d , 24h , 1h ) | | | --schema-mode | Schema mode for tables in the database: implicit (default) or explicit . Fixed when the database is created | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | --tls-no-verify | Disable TLS certificate verification. Not recommended in production. Useful for testing with self-signed certificates | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | | Unit | Description | | --- | --- | | Unit | Description | | h | hour | | d | day | | w | week | | mo | month (30 days) | | y | year (365 days) | --- # influxdb3 create distinct_cache Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/distinct_cache/ The `influxdb3 create distinct_cache` command creates a new distinct value cache for a specific table and column set in your InfluxDB 3 Enterprise instance. Use this command to configure a cache that tracks unique values in specified columns. You must provide the database, token, table, and columns. Optionally, you can specify a name for the cache. ## Usage ```bash influxdb3 create distinct_cache [OPTIONS] \ --database \ --token --table
\ --columns \ [CACHE_NAME] ``` ## Arguments * **`CACHE_NAME`**: *(Optional)* A name to assign to the cache. If omitted, the CLI generates a name automatically. ## Options |Option| | Description | |------|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-t` | `--table` | *(Required)* Table to create the cache for | | | `--columns` |*(Required)* Comma-separated list of columns to cache distinct values for–for example: `col1,col2,col3` (see [Metadata cache hierarchy](#metadata-cache-hierarchy))| | |`--max-cardinality`| Maximum number of distinct value combinations to hold in the cache | | | `--max-age` | Maximum age of an entry in the cache entered as a human-readable duration–for example: `30d`, `24h` | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | `--tls-no-verify` | Disable TLS certificate verification. **Not recommended in production.** Useful for testing with self-signed certificates | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | Additional InfluxDB 3 Enterprise option: |Option| | Description | |------|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-n` |`--node-spec`|Which node(s) load historical data into the cache. Two value formats are supported: `all` (default) - all query- and processing-capable nodes load historical data, or `nodes:[,..]` - only the listed nodes load historical data, reducing the initial cache warm-up load at the cost of temporarily incomplete query results from excluded nodes| > [!Important] > #### Metadata cache hierarchy > > The distinct value cache has a hierarchical structure with a level for each specified column. > The order specified in the `--columns` option determines the order of levels, > from top-to-bottom, of the cache hierarchy. ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Prerequisites Before creating a distinct value cache, make sure you: 1. [Create a database](/influxdb3/enterprise/reference/cli/influxdb3/create/database/) 2. [Create a table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) that includes the columns you want to cache 3. Have a valid authentication token ## Examples Before running the following commands, replace the placeholder values with your own: * `DATABASE_NAME`: The database name * `TABLE_NAME`: The name of the table to cache values from * `CACHE_NAME`: The name of the distinct value cache to create * `COLUMN_NAME`: The column to cache distinct values from You can also set environment variables (such as `INFLUXDB3_AUTH_TOKEN`) instead of passing options inline. ### Create a distinct cache for one column Track unique values from a single column. This setup is useful for testing or simple use cases. ```bash influxdb3 create distinct_cache \ --database DATABASE_NAME \ --table TABLE_NAME \ --column COLUMN_NAME \ CACHE_NAME ``` ### Create a hierarchical cache with constraints Create a distinct value cache for multiple columns. The following example tracks unique combinations of `room` and `sensor_id`, and sets limits on the number of entries and their maximum age. ```bash influxdb3 create distinct_cache \ --database my_test_db \ --table my_sensor_table \ --columns room,sensor_id \ --max-cardinality 1000 \ --max-age 30d \ my_sensor_distinct_cache ``` ## Common pitfalls * `--column` is not valid. Use `--columns`. * Tokens must be included explicitly unless set via `INFLUXDB3_AUTH_TOKEN` * Table and column names must already exist or be recognized by the engine | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -t | --table | ( Required ) Table to create the cache for | | | --columns | ( Required ) Comma-separated list of columns to cache distinct values for–for example: col1,col2,col3 (see Metadata cache hierarchy ) | | | --max-cardinality | Maximum number of distinct value combinations to hold in the cache | | | --max-age | Maximum age of an entry in the cache entered as a human-readable duration–for example: 30d , 24h | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification. Not recommended in production. Useful for testing with self-signed certificates | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Option | | Description | | --- | --- | --- | | Option | | Description | | -n | --node-spec | Which node(s) load historical data into the cache. Two value formats are supported: all (default) - all query- and processing-capable nodes load historical data, or nodes:[,..] - only the listed nodes load historical data, reducing the initial cache warm-up load at the cost of temporarily incomplete query results from excluded nodes | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 create file_index Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/file_index/ The `influxdb3 create file_index` command creates a new file index for a database or table. ## Usage ```bash influxdb3 create file_index [OPTIONS] \ --database \ --token \ ... ``` ## Arguments * **COLUMNS**: The columns to use for the file index. ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-t` | `--table` | Table to apply the file index too | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification. **Not recommended in production.** Useful for testing with self-signed certificates| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Create a new file index for a database](#create-a-new-file-index-for-a-database) * [Create a new file index for a specific table](#create-a-new-file-index-for-a-specific-table) In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TABLE_NAME`: Table name ### Create a new file index for a database ```bash influxdb3 create file_index \ --database DATABASE_NAME \ --token AUTH_TOKEN \ column1 column2 column3 ``` ### Create a new file index for a specific table ```bash influxdb3 create file_index \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ column1 column2 column3 ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -t | --table | Table to apply the file index too | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification. Not recommended in production. Useful for testing with self-signed certificates | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 create last_cache Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/last_cache/ The `influxdb3 create last_cache` command creates a last value cache, which stores the most recent values for specified columns in a table. Use this to efficiently retrieve the latest values based on key column combinations. ## Usage ```bash influxdb3 create last_cache [OPTIONS] \ --database DATABASE_NAME \ --table TABLE_NAME \ --token AUTH_TOKEN \ CACHE_NAME ``` ## Arguments * **CACHE\_NAME**: *(Optional)* Name for the cache. If omitted, InfluxDB automatically generates one. ## Options |Option| | Description | |------|-----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-t` | `--table` | *(Required)* Table to create the cache for | | | `--key-columns` | Comma-separated list of columns to use as keys in the cache–for example: `foo,bar,baz` | | |`--value-columns`| Comma-separated list of columns to store as values in the cache–for example: `foo,bar,baz` | | | `--count` | Number of entries per unique key column combination to store in the cache (default `1`) | | | `--ttl` |Cache entries’ time-to-live (TTL) in [Humantime form](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)–for example: `10s`, `1min 30sec`, `3 hours` (default `4 hours`)| | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`| Disable TLS certificate verification. **Not recommended in production.** Useful for testing with self-signed certificates | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | Additional InfluxDB 3 Enterprise option: |Option| | Description | |------|-------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-n` |`--node-spec`|Which node(s) load historical data into the cache. Two value formats are supported: `all` (default) - all query- and processing-capable nodes load historical data, or `nodes:[,..]` - only the listed nodes load historical data, reducing the initial cache warm-up load at the cost of temporarily incomplete query results from excluded nodes| ### Option environment variables You can use the following environment variables as substitutes for CLI options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Prerequisites Before creating a last value cache, ensure you’ve done the following: * Create a [database](/influxdb3/enterprise/reference/cli/influxdb3/create/database/). * Create a [table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) with the columns you want to cache. * Have a valid authentication token. ## Examples A last value cache stores the most recent values from specified columns in a table. ### Create a basic last value cache for one column The following example shows how to track the most recent value for a single key (the last temperature for each room): ```bash influxdb3 create last_cache \ --database DATABASE_NAME \ --table my_sensor_table \ --token AUTH_TOKEN \ --key-columns room \ --value-columns temp \ my_temp_cache ``` ### Create a last value cache with multiple keys and values The following example shows how to: * Use multiple columns as a composite key * Track several values per key combination * Set a cache entry limit with `--count` * Configure automatic expiry with `--ttl` ```bash influxdb3 create last_cache \ --database DATABASE_NAME \ --table my_sensor_table \ --token AUTH_TOKEN \ --key-columns room,sensor_id \ --value-columns temp,hum \ --count 10 \ --ttl 1h \ my_sensor_cache ``` ## Usage notes * Define the table schema to include all specified key and value columns. * Pass tokens using `--token`, unless you’ve set one through an environment variable. * Specify `--count` and `--ttl` to override the defaults; otherwise, the system uses default values. | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -t | --table | ( Required ) Table to create the cache for | | | --key-columns | Comma-separated list of columns to use as keys in the cache–for example: foo,bar,baz | | | --value-columns | Comma-separated list of columns to store as values in the cache–for example: foo,bar,baz | | | --count | Number of entries per unique key column combination to store in the cache (default 1 ) | | | --ttl | Cache entries’ time-to-live (TTL) in Humantime form –for example: 10s , 1min 30sec , 3 hours (default 4 hours ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification. Not recommended in production. Useful for testing with self-signed certificates | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Option | | Description | | --- | --- | --- | | Option | | Description | | -n | --node-spec | Which node(s) load historical data into the cache. Two value formats are supported: all (default) - all query- and processing-capable nodes load historical data, or nodes:[,..] - only the listed nodes load historical data, reducing the initial cache warm-up load at the cost of temporarily incomplete query results from excluded nodes | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 create table Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/table/ The `influxdb3 create table` command creates a new table in a specified database. Tables must include at least one tag column and can optionally include field columns with defined data types. > [!Note] > InfluxDB automatically creates tables when you write line protocol data. Use this command > only if you need to define a custom schema or apply a custom partition template before > writing data. ## Usage ```bash # Syntax influxdb3 create table [OPTIONS] \ --tags [...] \ --database \ --token \ ``` ## Arguments * **TABLE\_NAME**: The name of the table to create. ## Options |Option| | Description | |------|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | |`--retention-period`|[Retention period](/influxdb3/enterprise/reference/glossary/#retention-period) ([duration](/influxdb3/enterprise/reference/glossary/#duration) value, for example: `30d`, `24h`, `1h`) for data in the table| | | `--token` | *(Required)* Authentication token | | | `--tags` | *(Required)* Comma-separated list of tag columns to include in the table | | | `--fields` | Comma-separated list of field columns and their types to include in the table | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | `--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | > [!Important] > #### Tag and field naming requirements > > Tag and field keys are alphanumeric and must start with a letter or number. > They can contain dashes (`-`) and underscores (`_`). ### Field data types Use the following data types when defining field columns with `--fields`: | Type | Description | |---------|----------------------------| | `int64` | 64-bit signed integer | |`uint64` | 64-bit unsigned integer | |`float64`|64-bit floating point number| | `utf8` | UTF-8 string | | `bool` |Boolean (`true` or `false`) | Provide each field as `NAME:TYPE`–for example, `temp:float64`. Separate multiple fields with commas: `--fields temp:float64,hum:float64,co:int64`. ### Option environment variables You can use the following environment variables to set options instead of passing them via CLI flags: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples In the examples below, replace the following: * `DATABASE_NAME`: The database name * `AUTH_TOKEN`: The authentication token * `TABLE_NAME`: A name for the new table ### Create an empty table ```bash influxdb3 create table \ --tags tag1,tag2,tag3 \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` > [!Important] > In InfluxDB 3 Enterprise 3.12 with the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/), > a query on a table that has no fields returns HTTP `500`. > Include at least one field with `--fields`. ### Create a table with tag and field columns ```bash influxdb3 create table \ --tags room,sensor_id \ --fields temp:float64,hum:float64,co:int64 \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` ### Create a table with a retention period ```bash influxdb3 create table \ --tags room,sensor_id \ --fields temp:float64,hum:float64 \ --retention-period 7d \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` ### Verification Use the `SHOW TABLES` query to verify that the table was created successfully: ```bash influxdb3 query \ --database my_test_db \ --token AUTH_TOKEN \ "SHOW TABLES" Example output: +---------------+--------------------+----------------------------+------------+ | table_catalog | table_schema | table_name | table_type | +---------------+--------------------+----------------------------+------------+ | public | iox | my_sensor_table | BASE TABLE | | public | system | distinct_caches | BASE TABLE | | public | system | last_caches | BASE TABLE | | public | system | parquet_files | BASE TABLE | +---------------+--------------------+----------------------------+------------+ ``` > [!Note] > `SHOW TABLES` is an SQL query. It isn’t supported in InfluxQL. ## Relation to schema mode InfluxDB 3 Enterprise databases use one of two[schema modes](/influxdb3/enterprise/reference/cli/influxdb3/create/database/#schema-mode), set when you create the database: * **`implicit`** *(default)*: `influxdb3 create table` is optional. InfluxDB 3 Enterprise creates tables and columns automatically from the line protocol you write. * **`explicit`** *(InfluxDB 3 Enterprise only)*:`influxdb3 create table` is how you declare a table’s tag and field columns before writing to it. A write that names a table or column you haven’t declared is rejected. Add columns to a declared table later with the[`influxdb3 update table` command](/influxdb3/enterprise/reference/cli/influxdb3/update/table/). In both modes, `influxdb3 create table` accepts the same options and defines the table the same way. The schema mode only changes what happens when a write references a table or column that `create table` hasn’t defined. #### Related * [Create a table](/influxdb3/enterprise/admin/tables/create/) * [Create table API](/influxdb3/enterprise/api/table/#operation/PostConfigureTable) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --retention-period | Retention period ( duration value, for example: 30d , 24h , 1h ) for data in the table | | | --token | ( Required ) Authentication token | | | --tags | ( Required ) Comma-separated list of tag columns to include in the table | | | --fields | Comma-separated list of field columns and their types to include in the table | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Type | Description | | --- | --- | | Type | Description | | int64 | 64-bit signed integer | | uint64 | 64-bit unsigned integer | | float64 | 64-bit floating point number | | utf8 | UTF-8 string | | bool | Boolean ( true or false ) | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 create token Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/token/ The `influxdb3 create token` command creates a new authentication token. This returns the raw token string. Use it to authenticate future CLI commands and API requests. > [!Important] > InfluxDB displays the raw token string only once. Be sure to copy and securely store it. ## Usage ```bash influxdb3 create token ``` ## Commands | Command | Description | | |------------------------------------------------------------------------------|-----------------------------------------------------------------------------|---| |[`--admin`](/influxdb3/enterprise/reference/cli/influxdb3/create/token/admin/)|Create an operator or named admin token for the InfluxDB 3 Enterprise server.| | Additional InfluxDB 3 Enterprise command: | Command | Description | |----------------------------------------------------------------------------------------|-------------------------------------------------------------| |[`--permission`](/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/)|Create a resource token with fine-grained access permissions.| ## Options |Option| | Description | |------|------------|-------------------------------| | | `--admin` | Create an admin token | | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Command | Description | | | --- | --- | --- | | Command | Description | | | --admin | Create an operator or named admin token for the InfluxDB 3 Enterprise server. | | | Command | Description | | --- | --- | | Command | Description | | --permission | Create a resource token with fine-grained access permissions. | | Option | | Description | | --- | --- | --- | | Option | | Description | | | --admin | Create an admin token | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 create token --admin Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/token/admin/ Create an operator token or named admin token. ## Usage ``` influxdb3 create token --admin [OPTIONS] ``` ## Options | Option | Description | |---------------|-------------------------------------------------------------------------------------------------------------| |`--regenerate` | Regenerates the operator token. Requires `--token` and the current operator token | | `--name` | Name of the token | | `--expiry` | Expires in `duration`–for example, 10d for 10 days 1y for 1 year | | `--host` |The host URL of the running InfluxDB 3 server [env: `INFLUXDB3_HOST_URL=`] [default: `http://127.0.0.1:8181`]| | `--token` | An existing admin token for the InfluxDB 3 server | | `--tls-ca` | An optional arg to use a custom ca for useful for testing with self signed certs | | `--format` | Output format for token [possible values: `json`, `text`] | | `--offline` | Generate token without connecting to server (for automation) | |`--output-file`| File path to save the token (required with `--offline`) | |`-h`, `--help` | Print help information | | `--help-all` | Print more detailed help information | ## Examples * [Create an operator token](#create-an-operator-token) * [Use the operator token to create a named admin token](#use-the-operator-token-to-create-a-named-admin-token) * [Use the token to create a database](#use-the-token-to-create-a-database) * [Generate an offline admin token](#generate-an-offline-admin-token) ### Create an operator token The operator token is a special token that has full administrative privileges on the InfluxDB server and doesn’t expire. The first admin token you create becomes the operator token for the instance. You can create an operator token using the `--admin` flag without any additional options. ```bash influxdb3 create token --admin ``` The output is the raw token string you can use to authenticate future CLI commands and API requests. For CLI commands, use the `--token` option or the `INFLUXDB3_AUTH_TOKEN` environment variable to pass the token string. ### Use the operator token to create a named admin token ```bash influxdb3 create token \ --admin \ --token OPERATOR_TOKEN \ --name TOKEN_NAME \ --expiry DURATION ``` Replace the following: * `OPERATOR_TOKEN`: Your operator token for the server * `TOKEN_NAME`: Name for your new admin token * `DURATION`: Duration for the token to remain valid, in [humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) format (for example, `10d` for 10 days or `1y` for 1 year). ### Use the token to create a database ```bash influxdb3 create database \ --token ADMIN_TOKEN \ DATABASE_NAME ``` Replace the following: * `ADMIN_TOKEN`: Your InfluxDB admin token * `DATABASE_NAME`: Name for your new database > [!Note] > #### Use CLI environment variables > > Set the token as an environment variable to simplify repeated CLI commands: > > ``` > export INFLUXDB3_AUTH_TOKEN=ADMIN_TOKEN > ``` ### Generate an offline admin token Generate an offline admin token file to use if no tokens exist when the server starts. Once started, you can interact with the server using the provided token. Offline admin tokens are designed to help with automated deployments. Include the following options: * `--offline` *(Required)* * `--output-file` *(Required)* * `--name` *(default is `_admin`)* * `--expiry` *(Optional)* ```bash influxdb3 create token --admin \ --name TOKEN_NAME \ --expiry DURATION \ --offline \ --output-file path/to/admin-token.json ``` Replace the following: * `TOKEN_NAME`: Name for your offline admin token * `DURATION`: Duration for the token to remain valid, in [humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html) format (for example, `10d` for 10 days or `1y` for 1 year). * `path/to/admin-token.json`: File path to use for the generated token file [](#view-example-offline-admin-token-file) View example offline admin token file ```json { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "example-admin-token", "expiry_millis": 1756400061529 } ``` | Option | Description | | --- | --- | | Option | Description | | --regenerate | Regenerates the operator token. Requires --token and the current operator token | | --name | Name of the token | | --expiry | Expires in duration –for example, 10d for 10 days 1y for 1 year | | --host | The host URL of the running InfluxDB 3 server [env: INFLUXDB3_HOST_URL= ] [default: http://127.0.0.1:8181 ] | | --token | An existing admin token for the InfluxDB 3 server | | --tls-ca | An optional arg to use a custom ca for useful for testing with self signed certs | | --format | Output format for token [possible values: json , text ] | | --offline | Generate token without connecting to server (for automation) | | --output-file | File path to save the token (required with --offline ) | | -h , --help | Print help information | | --help-all | Print more detailed help information | --- # influxdb3 create token --permission Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/token/permission/ The `influxdb3 create token` command with the `--permission` option creates a new authentication token with fine-grained access permissions for specific resources in InfluxDB 3 Enterprise. Fine-grained access permissions allow you to specify the exact actions, such as`read` and `write` that a token can perform on a specific resource, such as a database or a system information endpoint. ## Usage ```bash influxdb3 create token --permission --name [OPTIONS] ``` ## Options | Option | | Description | |------------------------------|---------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| | |`--permission `|Permissions in `RESOURCE_TYPE:RESOURCE_NAMES:ACTIONS` format–for example, `db:*:read,write`, `system:*:read`. `--permission` may be specified multiple times| | | `--name ` | Name of the token | | `-H` | `--host ` | The host URL of the running InfluxDB 3 Enterprise server [env: INFLUXDB3\_HOST\_URL=] [default: http://127.0.0.1:8181] | | | `--token ` | The [enterprise token](/influxdb3/enterprise/admin/tokens/admin) | |[env: INFLUXDB3\_AUTH\_TOKEN=]| | | | | `--expiry ` | The token expiration time as a duration (for example, 1h, 7d, 1y). If not set, the token does not expire until revoked | | | `--tls-ca ` | An optional arg to use a custom CA for testing with self-signed certs [env: INFLUXDB3\_TLS\_CA=] | | | `--format ` | Output format (`json` or `text` *(default)*) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ## Permission format The `--permission` option takes a value in the format `RESOURCE_TYPE:RESOURCE_NAMES:ACTIONS`. * `RESOURCE_TYPE`: Available resource types include: * `db` for databases * `system` for system information endpoints. * `RESOURCE_NAMES`: Can be a specific resource name, such as a database name, a comma-separated list of names, or `*` to grant access to all resources of the type. * `ACTIONS`: A list of actions. Available actions depend on the resource type. ## Examples * [Create a token with read and write access to a database](#create-a-token-with-read-and-write-access-to-a-database) * [Create a token with read-only access to a database](#create-a-token-with-read-only-access-to-a-database) * [Create a token with access to multiple databases](#create-a-token-with-access-to-multiple-databases) * [Create a token with access to all databases](#create-a-token-with-access-to-all-databases) * [Create a token that expires in seven days](#create-a-token-that-expires-in-seven-days) * [Create a system token for health information](#create-a-system-token-for-health-information) * [Create a token with access to all system information](#create-a-token-with-access-to-all-system-information) * [Create a token with multiple permissions](#create-a-token-with-multiple-permissions) * [Generate an offline permission (resource) tokens file](#generate-an-offline-permission-resource-tokens-file) ### Create a token with read and write access to a database ```bash influxdb3 create token \ --permission "db:my_database:read,write" \ --name "Read/write token for my_database" ``` ### Create a token with read-only access to a database ```bash influxdb3 create token \ --permission "db:my_database:read" \ --name "Read-only token for my_database" ``` ### Create a token with access to multiple databases ```bash influxdb3 create token \ --permission "db:database1,database2:read,write" \ --name "Multi-database token" ``` ### Create a token with access to all databases ```bash influxdb3 create token \ --permission "db:*:read,write" \ --name "All databases token" ``` ### Create a token that expires in seven days ```bash influxdb3 create token \ --permission "db:my_database:read,write" \ --name "Expiring token" \ --expiry 7d ``` ### Create a system token for health information ```bash influxdb3 create token \ --permission "system:health:read" \ --name "System health token" ``` ### Create a token with access to all system information ```bash influxdb3 create token \ --permission "system:*:read" \ --name "All system information" ``` ### Create a token with multiple permissions ```bash influxdb3 create token \ --permission "db:database1:read,write" \ --permission "system:health:read" \ --name "Multi-permission token" ``` ### Generate an offline permission (resource) tokens file Generate an offline permission (resource) tokens file to use if no resource tokens exist when the server starts. Once started, you can interact with the server using the provided tokens. Offline permission tokens are designed to help with automated deployments. Include the following options: * `--name` *(Required)* * `--permissions` *(Required)* * `--offline` *(Required)* * `--output-file` *(Required)* * `--create-databases` *(Optional)* * `--expiry` *(Optional)* ```bash influxdb3 create token \ --name TOKEN_NAME \ --permission "TOKEN_PERMISSIONS" \ --expiry DURATION \ --offline \ --create-databases DATABASE_LIST \ --output-file path/to/tokens.json ``` Replace the following: * `TOKEN_NAME`: Name for your offline permission token * `TOKEN_PERMISSIONS`:[Token permissions](#permission-format). * `DURATION`: Duration for the token to remain valid, in[humantime](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html)format (for example, `10d` for 10 days or `1y` for 1 year). * `DATABASE_LIST`: Comma-separated list of database names to create when starting the InfluxDB 3 Enterprise server using the generated tokens file * `path/to/tokens.json`: File path to use for the generated tokens file [](#view-example-offline-permission-tokens-file) View example offline permission tokens file ```json { "create_databases": [ "db1", "db2", "db3" ], "tokens": [ { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "example-token", "expiry_millis": 1756400061529, "permissions": [ "db:db1,db2:read,write", "db:db3:read" ] } ] } ``` > [!Note] > If you write a new offline permission token to an existing permission token file, > the command appends the new token to the existing output file. | Option | | Description | | --- | --- | --- | | Option | | Description | | | --permission | Permissions in RESOURCE_TYPE:RESOURCE_NAMES:ACTIONS format–for example, db:*:read,write , system:*:read . --permission may be specified multiple times | | | --name | Name of the token | | -H | --host | The host URL of the running InfluxDB 3 Enterprise server [env: INFLUXDB3_HOST_URL=] [default: http://127.0.0.1:8181] | | | --token | The enterprise token | | [env: INFLUXDB3_AUTH_TOKEN=] | | | | | --expiry | The token expiration time as a duration (for example, 1h, 7d, 1y). If not set, the token does not expire until revoked | | | --tls-ca | An optional arg to use a custom CA for testing with self-signed certs [env: INFLUXDB3_TLS_CA=] | | | --format | Output format ( json or text (default) ) | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 create trigger Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/ The `influxdb3 create trigger` command creates a new trigger for the processing engine. ## Usage ```bash # Syntax influxdb3 create trigger [OPTIONS] \ --database \ --token \ --plugin-filename \ --trigger-spec \ ``` ## Arguments * **TRIGGER\_NAME**: A name for the new trigger. ## Options |Option| | Description | |------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-p` | `--path` |Path to plugin file or directory (single `.py` file or directory containing `__init__.py` for multifile plugins). Can be local path (with `--upload`) or server path. Replaces `--plugin-filename`.| | | `--upload` | Upload local plugin files to the server. Requires admin token. Use with `--path` to specify local files. | | | `--plugin-filename` | *(Deprecated: use `--path` instead)* Name of the file, stored in the server’s `plugin-dir`, that contains the Python plugin code to run | | | `--trigger-spec` | Trigger specification: `table:`, `all_tables`, `every:`, `cron:`, or `request:` | | |`--trigger-arguments`| Additional arguments for the trigger, in the format `key=value`, separated by commas (for example, `arg1=val1,arg2=val2`) | | | `--disabled` | Create the trigger in disabled state (default: enabled) | | | `--error-behavior` | Error handling behavior: `log`, `retry`, or `disable` (default: `log`) | | |`--run-asynchronous` | Run the trigger asynchronously, allowing multiple triggers to run simultaneously (default is synchronous) | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | `--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | Additional InfluxDB 3 Enterprise option: |Option| | Description | |------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | |`--node-spec`|Which nodes own the trigger. Two value formats are supported: `all` (default) - every node that runs the Processing Engine (a node with `--plugin-dir` configured) owns the trigger, or `nodes:[,...]` - only the listed nodes own the trigger. In a cluster, set this to your process node, see [Pin a trigger to the process node](#pin-a-trigger-to-the-process-node)| ### Reference a plugin from GitHub To fetch a plugin remotely instead of downloading it locally, prefix the plugin path with `gh:`. The path after the prefix is appended to the configured `--plugin-repo`–for example: ``` gh:examples/wal_plugin/wal_plugin.py ``` Despite the name, `gh:` doesn’t require GitHub or a Git repository–`--plugin-repo` accepts any HTTP/HTTPS URL that serves raw plugin files. By default, `gh:`-prefixed plugins resolve against the [`influxdata/influxdb3_plugins`](https://github.com/influxdata/influxdb3_plugins) repository at `https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/`. For example, `gh:examples/wal_plugin/wal_plugin.py` resolves to `https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/examples/wal_plugin/wal_plugin.py`. To reference plugins from a different repository, set the `--plugin-repo` server option (or the `INFLUXDB3_PLUGIN_REPO` environment variable) to a raw content URL base path. For more information, see the [`plugin-repo` config option](/influxdb3/enterprise/reference/config-options/#plugin-repo). InfluxDB fetches the plugin over HTTP when the trigger is created (to validate it), and again each time the trigger starts–for example, on server startup or when re-enabled. If the fetch fails, `influxdb3 create trigger` returns an error with the HTTP status code and URL: ``` error fetching plugin from repository: 404 Not Found https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/not_found.py ``` Unlike local plugins, `gh:`-prefixed plugins aren’t automatically reloaded when the source changes–disable and re-enable the trigger to fetch updates. Only single-file plugins are supported with the `gh:` prefix; multi-file plugin directories must be uploaded locally with `--upload`. ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples The following examples show how to use the `influxdb3 create trigger` command to create triggers in different scenarios. * [Create a trigger for a specific table](#create-a-trigger-for-a-specific-table) * [Create a trigger for all tables](#create-a-trigger-for-all-tables) * [Create a trigger with a schedule](#create-a-trigger-with-a-schedule) * [Create a trigger for HTTP requests](#create-a-trigger-for-http-requests) * [Create a trigger with a multifile plugin](#create-a-trigger-with-a-multifile-plugin) * [Upload and create a trigger with a local plugin](#upload-and-create-a-trigger-with-a-local-plugin) * [Create a trigger with additional arguments](#create-a-trigger-with-additional-arguments) * [Create a disabled trigger](#create-a-disabled-trigger) * [Create a trigger with error handling](#create-a-trigger-with-error-handling) Replace the following placeholders with your values: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `PLUGIN_FILENAME`: Python plugin filename * `TRIGGER_NAME`: Name of the trigger to create * `TABLE_NAME`: Name of the table to trigger on ### Create a trigger for a specific table Create a trigger that processes data from a specific table. ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename PLUGIN_FILENAME \ --trigger-spec table:TABLE_NAME \ TRIGGER_NAME ``` `PLUGIN_FILENAME` must implement the [data write plugin](/influxdb3/enterprise/plugins/#create-a-data-write-plugin) interface. ### Create a trigger for all tables Create a trigger that applies to all tables in the specified database. ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec all_tables \ TRIGGER_NAME ``` `PLUGIN_FILENAME` must implement the [data write plugin](/influxdb3/enterprise/plugins/#create-a-data-write-plugin) interface. This is useful when you want a trigger to apply to any table in the database, regardless of name. ### Create a trigger with a schedule Create a trigger that runs at a specific interval using a duration. Supported duration units: `s` (seconds), `m` (minutes), `h` (hours), `d` (days), `w` (weeks), `M` (months), `y` (years). Maximum interval is 1 year. ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec every:5m \ TRIGGER_NAME ``` Create a trigger that runs based on a cron schedule using extended 6-field cron format. The cron expression follows the format: ``` second minute hour day_of_month month day_of_week ``` Fields: * **second**: 0-59 * **minute**: 0-59 * **hour**: 0-23 * **day\_of\_month**: 1-31 * **month**: 1-12 or JAN-DEC * **day\_of\_week**: 0-7 (0 or 7 is Sunday) or SUN-SAT Example: Run at 6:00 AM every weekday (Monday-Friday): ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec "cron:0 0 6 * * 1-5" \ TRIGGER_NAME ``` `PLUGIN_FILENAME` must implement the [scheduled plugin](/influxdb3/enterprise/plugins/#create-a-scheduled-plugin) interface. ### Create a trigger for HTTP requests Create a trigger that provides an API endpoint and processes HTTP requests. ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename PLUGIN_FILENAME \ --trigger-spec request:REQUEST_PATH \ TRIGGER_NAME ``` `PLUGIN_FILENAME` must implement the [HTTP request plugin](/influxdb3/enterprise/plugins/#create-an-http-request-plugin) interface. ### Create a trigger with a multifile plugin Create a trigger using a plugin organized in multiple files. The plugin directory must contain an `__init__.py` file. ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --path "my_complex_plugin" \ --trigger-spec "every:5m" \ TRIGGER_NAME ``` The `--path` points to a directory in the server’s `plugin-dir` with the following structure: ``` my_complex_plugin/ ├── __init__.py # Required entry point ├── processors.py # Supporting modules └── utils.py ``` For more information about multifile plugins, see [Create your plugin file](/influxdb3/enterprise/plugins/#create-your-plugin-file). ### Upload and create a trigger with a local plugin Upload plugin files from your local machine and create a trigger in a single command. Requires admin token. ```bash # Upload single-file plugin influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --path "/local/path/to/plugin.py" \ --upload \ --trigger-spec "every:1m" \ TRIGGER_NAME # Upload multifile plugin directory influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --path "/local/path/to/plugin-dir" \ --upload \ --trigger-spec "table:TABLE_NAME" \ TRIGGER_NAME ``` The `--upload` flag transfers local files to the server’s plugin directory. This is useful for: * Local plugin development and testing * Deploying plugins without SSH access * Automating plugin deployment For more information, see [Upload plugins from local machine](/influxdb3/enterprise/plugins/#upload-plugins-from-local-machine). ### Create a trigger with additional arguments ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec table:TABLE_NAME \ --trigger-arguments arg1=value1,arg2=value2 \ TRIGGER_NAME ``` ### Create a disabled trigger Create a trigger in a disabled state. ```bash influxdb3 create trigger \ --disabled \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec table:TABLE_NAME \ TRIGGER_NAME ``` Creating a trigger in a disabled state prevents it from running immediately. You can enable it later when you’re ready to activate it. ### Create a trigger with error handling Log the error to the service output and the `system.processing_engine_logs` table: ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec table:TABLE_NAME \ --error-behavior log \ TRIGGER_NAME ``` Rerun the trigger if it fails: ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec table:TABLE_NAME \ --error-behavior retry \ TRIGGER_NAME ``` Disable the trigger if it fails: ```bash influxdb3 create trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --plugin-filename \ --trigger-spec table:TABLE_NAME \ --error-behavior disable \ TRIGGER_NAME ``` ### Pin a trigger to the process node In a cluster, we recommend running the Processing Engine on a standalone process node. Create the trigger on the process node, and set `--node-spec` to the process node’s ID: ```bash influxdb3 create trigger \ --host http://PROCESS_HOST:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --path schedule_rollup.py \ --trigger-spec "every:5s" \ --node-spec "nodes:PROCESS_NODE_ID" \ hourly_rollup ``` The default, `--node-spec all`, makes every node that runs the Processing Engine own the trigger. The node in `--node-spec` must exist and run `process` mode: * An unknown node ID returns `invalid node name ()`. * A node that doesn’t run `process` mode returns `Node doesn't have proper mode, expect process`. You can’t change `--node-spec` after you create the trigger. To move a trigger to another node, delete the trigger and create it again. For request triggers, send `/api/v3/engine/` requests to the node in `--node-spec`. For the complete setup, see [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -p | --path | Path to plugin file or directory (single .py file or directory containing __init__.py for multifile plugins). Can be local path (with --upload ) or server path. Replaces --plugin-filename . | | | --upload | Upload local plugin files to the server. Requires admin token. Use with --path to specify local files. | | | --plugin-filename | (Deprecated: use --path instead) Name of the file, stored in the server’s plugin-dir , that contains the Python plugin code to run | | | --trigger-spec | Trigger specification: table: , all_tables , every: , cron: , or request: | | | --trigger-arguments | Additional arguments for the trigger, in the format key=value , separated by commas (for example, arg1=val1,arg2=val2 ) | | | --disabled | Create the trigger in disabled state (default: enabled) | | | --error-behavior | Error handling behavior: log , retry , or disable (default: log ) | | | --run-asynchronous | Run the trigger asynchronously, allowing multiple triggers to run simultaneously (default is synchronous) | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Option | | Description | | --- | --- | --- | | Option | | Description | | | --node-spec | Which nodes own the trigger. Two value formats are supported: all (default) - every node that runs the Processing Engine (a node with --plugin-dir configured) owns the trigger, or nodes:[,...] - only the listed nodes own the trigger. In a cluster, set this to your process node, see Pin a trigger to the process node | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 debug Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/debug/ The `influxdb3 debug` command includes diagnostic tools for troubleshooting your InfluxDB 3 Enterprise deployment. ## Usage ```bash influxdb3 debug ``` ## Subcommands | Subcommand | Description | |---------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| | catalog | Inspect catalog files | | help | Print command help or the help of a subcommand | |[object-store-check](/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/)| Validate object store compatibility | | [sweep](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/) |Read back [orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/) audit trails as reports| | [verify-references](/influxdb3/enterprise/reference/cli/influxdb3/debug/verify-references/) | Check that every object the newest compactor checkpoint references exists | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | catalog | Inspect catalog files | | help | Print command help or the help of a subcommand | | object-store-check | Validate object store compatibility | | sweep | Read back orphaned file cleanup audit trails as reports | | verify-references | Check that every object the newest compactor checkpoint references exists | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 debug object-store-check Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/ The `influxdb3 debug object-store-check` command validates that an object store correctly implements the semantic operations that InfluxDB 3 Enterprise relies on. It writes synthetic test objects to `/oscheck-/` and reports any compatibility violations. This command connects directly to the object store and does not require a running InfluxDB 3 Enterprise server. The command is available in InfluxDB 3 Enterprise 3.10.0 and later. ## Usage ```bash influxdb3 debug object-store-check [OPTIONS] ``` ## Options |Option| | Description |Required| |------|-------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| | | `--object-store ` | Object store type. Valid values: `s3`, `google`, `azure`, `file`, `memory` | Yes | | | `--bucket ` | Object store bucket name. Required for `s3`, `google`, and `azure` object store types. | Varies | | | `--aws-endpoint ` | Custom AWS or S3-compatible endpoint URL | No | | | `--aws-access-key-id ` | AWS access key ID | No | | |`--aws-secret-access-key `| AWS secret access key | No | | | `--aws-allow-http` | Allow non-HTTPS connections to the object store | No | | | `--aws-default-region ` | AWS region | No | | | `--check-prefix ` | Prefix for synthetic test objects. The command writes to `/oscheck-/`. Environment variable: `INFLUXDB3_OSCHECK_PREFIX` | Yes | | | `--probe-prefix ` |Prefix of an existing catalog to also replay the catalog loader’s startup operations against, to diagnose failures without writing new data. The probe itself is read-only, but the command still runs the synthetic write and delete checks under `--check-prefix` first.| No | | `-h` | `--help` | Print help information | No | | | `--help-all` | Print detailed help information | No | ## Examples ### Check a MinIO (S3-compatible) object store Replace the following: * `MINIO_USERNAME`: your MinIO username * `MINIO_PASSWORD`: your MinIO password ```bash influxdb3 debug object-store-check \ --object-store s3 \ --bucket influxdb3 \ --aws-endpoint http://localhost:9000 \ --aws-access-key-id MINIO_USERNAME \ --aws-secret-access-key MINIO_PASSWORD \ --aws-allow-http \ --check-prefix oscheck ``` ### Also probe an existing catalog prefix Add `--probe-prefix` with your actual cluster ID to also diagnose catalog load failures against your real catalog. The probe itself is read-only, but the command still runs the synthetic write and delete checks under `--check-prefix` first, so that prefix must remain writable and the credentials need write access. ```bash influxdb3 debug object-store-check \ --object-store s3 \ --bucket influxdb3 \ --aws-endpoint http://localhost:9000 \ --aws-access-key-id MINIO_USERNAME \ --aws-secret-access-key MINIO_PASSWORD \ --aws-allow-http \ --check-prefix oscheck \ --probe-prefix my-cluster-id ``` | Option | | Description | Required | | --- | --- | --- | --- | | Option | | Description | Required | | | --object-store | Object store type. Valid values: s3 , google , azure , file , memory | Yes | | | --bucket | Object store bucket name. Required for s3 , google , and azure object store types. | Varies | | | --aws-endpoint | Custom AWS or S3-compatible endpoint URL | No | | | --aws-access-key-id | AWS access key ID | No | | | --aws-secret-access-key | AWS secret access key | No | | | --aws-allow-http | Allow non-HTTPS connections to the object store | No | | | --aws-default-region | AWS region | No | | | --check-prefix | Prefix for synthetic test objects. The command writes to /oscheck-/ . Environment variable: INFLUXDB3_OSCHECK_PREFIX | Yes | | | --probe-prefix | Prefix of an existing catalog to also replay the catalog loader’s startup operations against, to diagnose failures without writing new data. The probe itself is read-only, but the command still runs the synthetic write and delete checks under --check-prefix first. | No | | -h | --help | Print help information | No | | | --help-all | Print detailed help information | No | --- # influxdb3 debug sweep Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/ The `influxdb3 debug sweep` command reads back the audit trail that[orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/)writes each time it runs. Both subcommands work directly against object storage. They take the same object store connection options as `influxdb3 serve`and don’t need a running InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 debug sweep ``` ## Subcommands | Subcommand | Description | |---------------------------------------------------------------------------|----------------------------------------------| | [list](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/list/) |List recorded cleanup passes, oldest to newest| |[report](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/report/)| Read one pass’s audit trail back as a report | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | list | List recorded cleanup passes, oldest to newest | | report | Read one pass’s audit trail back as a report | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 debug sweep list Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/list/ The `influxdb3 debug sweep list` command names the[orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/)passes recorded for a cluster, oldest to newest. It reads the audit trail directly from object storage and doesn’t need a running InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 debug sweep list [OPTIONS] --cluster-id ``` ## Options | Option | Description |Required| |-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------|--------| | `--object-store ` |Object store type. Valid values: `s3`, `google`, `azure`, `file`, `memory`. Default: `file`. Environment variable: `INFLUXDB3_OBJECT_STORE`| No | | `--bucket ` | Object store bucket name. Required for `s3`, `google`, and `azure` object store types | Varies | | `--cluster-id ` | Cluster whose cleanup audit trail to read. Environment variable: `INFLUXDB3_CLUSTER_ID` | Yes | |`--engine-path-prefix `| Engine path prefix the cluster runs with, if any. Environment variable: `INFLUXDB3_ENGINE_PATH_PREFIX` | No | | `-h`, `--help` | Print help information | No | | `--help-all` | Print detailed help information | No | `influxdb3 debug sweep list` accepts the same object store connection options as `influxdb3 serve`. For example, `--aws-access-key-id` and`--aws-secret-access-key` for S3-compatible stores. For the complete object store flag reference, see[`influxdb3 debug object-store-check`](/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/#options). ## Examples ### List recorded cleanup passes Replace the following: * `CLUSTER_ID`: your cluster identifier * `S3_BUCKET`: your object store bucket name ```bash influxdb3 debug sweep list \ --object-store s3 \ --bucket S3_BUCKET \ --cluster-id CLUSTER_ID ``` If no passes are recorded yet, the command prints `no passes recorded` to standard error and exits successfully. | Option | Description | Required | | --- | --- | --- | | Option | Description | Required | | --object-store | Object store type. Valid values: s3 , google , azure , file , memory . Default: file . Environment variable: INFLUXDB3_OBJECT_STORE | No | | --bucket | Object store bucket name. Required for s3 , google , and azure object store types | Varies | | --cluster-id | Cluster whose cleanup audit trail to read. Environment variable: INFLUXDB3_CLUSTER_ID | Yes | | --engine-path-prefix | Engine path prefix the cluster runs with, if any. Environment variable: INFLUXDB3_ENGINE_PATH_PREFIX | No | | -h , --help | Print help information | No | | --help-all | Print detailed help information | No | --- # influxdb3 debug sweep report Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/report/ The `influxdb3 debug sweep report` command reads one[orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/)pass’s audit trail back and summarizes it. It reads directly from object storage and doesn’t need a running InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 debug sweep report [OPTIONS] --cluster-id ``` ## Options | Option | Description |Required| |-------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------| | `--object-store ` | Object store type. Valid values: `s3`, `google`, `azure`, `file`, `memory`. Default: `file`. Environment variable: `INFLUXDB3_OBJECT_STORE` | No | | `--bucket ` | Object store bucket name. Required for `s3`, `google`, and `azure` object store types | Varies | | `--cluster-id ` | Cluster whose cleanup audit trail to read. Environment variable: `INFLUXDB3_CLUSTER_ID` | Yes | |`--engine-path-prefix `| Engine path prefix the cluster runs with, if any. Environment variable: `INFLUXDB3_ENGINE_PATH_PREFIX` | No | | `--pass ` | Pass to summarize. Defaults to the newest pass that has at least one readable segment | No | | `--candidates` |Print every recorded candidate path, marked `queued` (a segment enqueued it for deletion) or `would-be` (found but not enqueued, for example, by a dry-run segment)| No | | `--json` | Emit the summary as JSON, including full segment detail | No | | `-h`, `--help` | Print help information | No | | `--help-all` | Print detailed help information | No | `influxdb3 debug sweep report` accepts the same object store connection options as `influxdb3 serve`. For example, `--aws-access-key-id` and`--aws-secret-access-key` for S3-compatible stores. For the complete object store flag reference, see[`influxdb3 debug object-store-check`](/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/#options). ## Examples * [Summarize the newest pass](#summarize-the-newest-pass) * [Summarize a specific pass](#summarize-a-specific-pass) * [Compare a dry-run pass against an armed pass](#compare-a-dry-run-pass-against-an-armed-pass) Replace the following in the examples below: * `CLUSTER_ID`: your cluster identifier * `S3_BUCKET`: your object store bucket name * `PASS_NAME`: a pass name returned by[`influxdb3 debug sweep list`](/influxdb3/enterprise/reference/cli/influxdb3/debug/sweep/list/) ### Summarize the newest pass ```bash influxdb3 debug sweep report \ --object-store s3 \ --bucket S3_BUCKET \ --cluster-id CLUSTER_ID ``` The summary includes whether the pass is complete, how many segments it wrote, how many objects it scanned, the candidate count and reclaimable bytes, and a breakdown of protected objects by reason. ### Summarize a specific pass ```bash influxdb3 debug sweep report \ --object-store s3 \ --bucket S3_BUCKET \ --cluster-id CLUSTER_ID \ --pass PASS_NAME \ --candidates ``` ### Compare a dry-run pass against an armed pass Use `--json` to get the full segment detail and diff two passes, for example, to confirm a `dry-run` pass and a later `armed` pass found the same candidates: ```bash influxdb3 debug sweep report \ --object-store s3 \ --bucket S3_BUCKET \ --cluster-id CLUSTER_ID \ --pass PASS_NAME \ --json > pass-summary.json ``` | Option | Description | Required | | --- | --- | --- | | Option | Description | Required | | --object-store | Object store type. Valid values: s3 , google , azure , file , memory . Default: file . Environment variable: INFLUXDB3_OBJECT_STORE | No | | --bucket | Object store bucket name. Required for s3 , google , and azure object store types | Varies | | --cluster-id | Cluster whose cleanup audit trail to read. Environment variable: INFLUXDB3_CLUSTER_ID | Yes | | --engine-path-prefix | Engine path prefix the cluster runs with, if any. Environment variable: INFLUXDB3_ENGINE_PATH_PREFIX | No | | --pass | Pass to summarize. Defaults to the newest pass that has at least one readable segment | No | | --candidates | Print every recorded candidate path, marked queued (a segment enqueued it for deletion) or would-be (found but not enqueued, for example, by a dry-run segment) | No | | --json | Emit the summary as JSON, including full segment detail | No | | -h , --help | Print help information | No | | --help-all | Print detailed help information | No | --- # influxdb3 debug verify-references Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/debug/verify-references/ The `influxdb3 debug verify-references` command checks that every object the cluster’s newest compactor checkpoint references still exists in object storage. It’s the inverse of[orphaned file cleanup](/influxdb3/enterprise/admin/orphaned-file-cleanup/): the cleanup process proves nothing extra is kept, and this command proves nothing referenced is gone. Use it for general store-integrity audits, for example after an incident, a migration, or suspected data loss. The command reads directly from object storage and doesn’t need a running InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 debug verify-references [OPTIONS] --cluster-id ``` ## Options | Option | Description | Required | |-------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------| | `--object-store ` | Object store type. Valid values: `s3`, `google`, `azure`, `file`, `memory`. Default: `file`. Environment variable: `INFLUXDB3_OBJECT_STORE` | No | | `--bucket ` | Object store bucket name. Required for `s3`, `google`, and `azure` object store types | Varies | | `--cluster-id ` | Cluster whose references to verify. Environment variable: `INFLUXDB3_CLUSTER_ID` | Yes | |`--engine-path-prefix `| Engine path prefix the cluster runs with, if any. Environment variable: `INFLUXDB3_ENGINE_PATH_PREFIX` | No | | `--concurrency ` | Number of concurrent run-set probes. Must be at least `1` |No, default `32`| | `--json` | Emit the result as JSON | No | | `--require-checkpoint` |Fail instead of verifying vacuously clean when no checkpoint is recorded for the cluster. Use this to catch a mistyped cluster ID or bucket, which otherwise looks the same as a fresh cluster| No | | `-h`, `--help` | Print help information | No | | `--help-all` | Print detailed help information | No | `influxdb3 debug verify-references` accepts the same object store connection options as `influxdb3 serve`. For example,`--aws-access-key-id` and `--aws-secret-access-key` for S3-compatible stores. For the complete object store flag reference, see[`influxdb3 debug object-store-check`](/influxdb3/enterprise/reference/cli/influxdb3/debug/object-store-check/#options). ## Exit codes Automation can gate on the exit code instead of parsing output: |Exit code| Meaning | |---------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `0` | Every referenced object exists. This includes a cluster with no checkpoint yet, unless `--require-checkpoint` is set | | `1` |The audit itself couldn’t run (object store errors, an invalid object store configuration, or no checkpoint with `--require-checkpoint` set). Command-line parse errors, such as an unknown flag or a missing `--cluster-id`, exit with `2` instead, so check the output before treating `2` as missing data| | `2` | One or more referenced objects are missing or unreadable | ## Examples ### Verify a cluster’s compacted-data references Replace the following: * `CLUSTER_ID`: your cluster identifier * `S3_BUCKET`: your object store bucket name ```bash influxdb3 debug verify-references \ --object-store s3 \ --bucket S3_BUCKET \ --cluster-id CLUSTER_ID ``` A clean run prints a line similar to: ``` OK: every compacted-data reference exists ``` ### Gate automation on a required checkpoint Use `--require-checkpoint` in a script so a mistyped cluster ID or bucket fails loudly instead of reporting a vacuously clean result: ```bash influxdb3 debug verify-references \ --object-store s3 \ --bucket S3_BUCKET \ --cluster-id CLUSTER_ID \ --require-checkpoint \ --json ``` | Option | Description | Required | | --- | --- | --- | | Option | Description | Required | | --object-store | Object store type. Valid values: s3 , google , azure , file , memory . Default: file . Environment variable: INFLUXDB3_OBJECT_STORE | No | | --bucket | Object store bucket name. Required for s3 , google , and azure object store types | Varies | | --cluster-id | Cluster whose references to verify. Environment variable: INFLUXDB3_CLUSTER_ID | Yes | | --engine-path-prefix | Engine path prefix the cluster runs with, if any. Environment variable: INFLUXDB3_ENGINE_PATH_PREFIX | No | | --concurrency | Number of concurrent run-set probes. Must be at least 1 | No, default 32 | | --json | Emit the result as JSON | No | | --require-checkpoint | Fail instead of verifying vacuously clean when no checkpoint is recorded for the cluster. Use this to catch a mistyped cluster ID or bucket, which otherwise looks the same as a fresh cluster | No | | -h , --help | Print help information | No | | --help-all | Print detailed help information | No | | Exit code | Meaning | | --- | --- | | Exit code | Meaning | | 0 | Every referenced object exists. This includes a cluster with no checkpoint yet, unless --require-checkpoint is set | | 1 | The audit itself couldn’t run (object store errors, an invalid object store configuration, or no checkpoint with --require-checkpoint set). Command-line parse errors, such as an unknown flag or a missing --cluster-id , exit with 2 instead, so check the output before treating 2 as missing data | | 2 | One or more referenced objects are missing or unreadable | --- # influxdb3 delete Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/ The `influxdb3 delete` command deletes a resource such as a cache, a database, or a table. ## Usage ```bash influxdb3 delete ``` ## Subcommands | Subcommand | Description | |---------------------------------------------------------------------------------------|---------------------------------------------| | [database](/influxdb3/enterprise/reference/cli/influxdb3/delete/database/) | Delete a database | | [file\_index](/influxdb3/enterprise/reference/cli/influxdb3/delete/file_index/) | Delete a file index for a database or table | | [last\_cache](/influxdb3/enterprise/reference/cli/influxdb3/delete/last_cache/) | Delete a last value cache | |[distinct\_cache](/influxdb3/enterprise/reference/cli/influxdb3/delete/distinct_cache/)| Delete a metadata cache | | [table](/influxdb3/enterprise/reference/cli/influxdb3/delete/table/) | Delete a table from a database | | [token](/influxdb3/enterprise/reference/cli/influxdb3/delete/token/) |Delete an authorization token from the server| | [trigger](/influxdb3/enterprise/reference/cli/influxdb3/delete/trigger/) | Delete a trigger for the processing engine | | query\_group | Delete a distributed query group (not yet operational) | | help | Print command help or the help of a subcommand | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | database | Delete a database | | file_index | Delete a file index for a database or table | | last_cache | Delete a last value cache | | distinct_cache | Delete a metadata cache | | table | Delete a table from a database | | token | Delete an authorization token from the server | | trigger | Delete a trigger for the processing engine | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 delete database Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/database/ The `influxdb3 delete database` command deletes a database. ## Usage ```bash # Syntax influxdb3 delete database [OPTIONS] ``` ## Arguments * **DATABASE\_NAME**: The name of the database to delete. Valid database names are alphanumeric and start with a letter or number. Dashes (`-`) and underscores (`_`) are allowed. Environment variable: `INFLUXDB3_DATABASE_NAME` ## Options |Option| | Description | |------|-----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--data-only` | Delete only data while preserving schemas and all associated resources (tokens, triggers, caches, etc.). Default behavior deletes everything | | |`--remove-tables`|Used with `--data-only` to remove table resources (caches) while preserving database-level resources (tokens, triggers, processing engine configurations)| | | `--hard-delete` | When to hard delete data (never/now/default/timestamp). Default behavior is a soft delete that allows recovery | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | |`--tls-no-verify`| Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Delete a database](#delete-a-database) * [Delete a database while specifying the token inline](#delete-a-database-while-specifying-the-token-inline) * [Delete database data only (preserve schema and resources)](#delete-database-data-only-preserve-schema-and-resources) * [Delete database data and tables (preserve database resources)](#delete-database-data-and-tables-preserve-database-resources) * [Hard delete a database immediately](#hard-delete-a-database-immediately) * [Hard delete a database at a specific time](#hard-delete-a-database-at-a-specific-time) In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token ### Delete a database ```bash influxdb3 delete database DATABASE_NAME ``` ### Delete a database while specifying the token inline ```bash influxdb3 delete database --token AUTH_TOKEN DATABASE_NAME ``` ### Delete database data only (preserve schema and resources) Delete all data from a database while preserving: * Database schema (tables and columns) * Authentication tokens * Processing engine configurations and triggers * Last value caches (LVC) and distinct value caches (DVC) This is useful when you want to clear old data and re-write new data to the same schema without recreating resources. ```bash influxdb3 delete database --data-only DATABASE_NAME ``` ### Delete database data and tables (preserve database resources) Delete all data and table resources (caches) while preserving database-level resources: * Authentication tokens * Processing engine triggers * Processing engine configurations This is useful when you want to start fresh with a new schema but keep existing authentication and trigger configurations. ```bash influxdb3 delete database --data-only --remove-tables DATABASE_NAME ``` ### Hard delete a database immediately Permanently delete a database and all its data immediately without the ability to recover. ```bash influxdb3 delete database --hard-delete now DATABASE_NAME ``` ### Hard delete a database at a specific time Schedule a database for permanent deletion at a specific timestamp. ```bash influxdb3 delete database --hard-delete "2024-01-01T00:00:00Z" DATABASE_NAME ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --data-only | Delete only data while preserving schemas and all associated resources (tokens, triggers, caches, etc.). Default behavior deletes everything | | | --remove-tables | Used with --data-only to remove table resources (caches) while preserving database-level resources (tokens, triggers, processing engine configurations) | | | --hard-delete | When to hard delete data (never/now/default/timestamp). Default behavior is a soft delete that allows recovery | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 delete distinct_cache Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/distinct_cache/ The `influxdb3 delete distinct_cache` command deletes a distinct value cache. ## Usage ```bash influxdb3 delete distinct_cache [OPTIONS] \ --database \ --table
\ [CACHE_NAME] ``` ## Arguments * **CACHE\_NAME**: *(Optional)* Name of the cache to delete. ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-t` | `--table` | *(Required)* Table to delete the cache for | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples ### Delete a distinct value cache ```bash influxdb3 delete distinct_cache \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ CACHE_NAME ``` In the example above, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TABLE_NAME`: Table name * `CACHE_NAME`: Name of the distinct value cache to delete | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -t | --table | ( Required ) Table to delete the cache for | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 delete file_index Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/file_index/ The `influxdb3 delete file_index` command deletes a file index for a database or table. ## Usage ```bash influxdb3 delete file_index [OPTIONS] --database ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-t` | `--table` | Table to delete the file index from | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Delete a file index from a database](#delete-a-file-index-from-a-database) * [Delete a file index from a specific table](#delete-a-file-index-from-a-specific-table) In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TABLE_NAME`: Table name ### Delete a file index from a database ```bash influxdb3 delete file_index \ --database DATABASE_NAME \ --token AUTH_TOKEN ``` ### Delete a file index from a specific table ```bash influxdb3 delete file_index \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -t | --table | Table to delete the file index from | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 delete last_cache Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/last_cache/ The `influxdb3 delete last_cache` command deletes a last value cache. ## Usage ```bash influxdb3 delete last_cache [OPTIONS] --database --table
[CACHE_NAME] ``` ## Arguments * **CACHE\_NAME**: *(Optional)* Name of the cache to delete. ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-t` | `--table` | *(Required)* Table to delete the cache from | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples ### Delete a last value cache ```bash influxdb3 delete last_cache \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --table TABLE_NAME \ CACHE_NAME ``` In the example above, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TABLE_NAME`: Table name * `CACHE_NAME`: Name of the last value cache to delete | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -t | --table | ( Required ) Table to delete the cache from | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 delete table Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/table/ The `influxdb3 delete table` command deletes a table from a database. ## Usage ```bash # Syntax influxdb3 delete table [OPTIONS] --database ``` ## Arguments * **TABLE\_NAME**: The name of the table to delete. ## Options |Option| | Description | |------|-----------------|-----------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--data-only` |Delete only data while preserving the table schema and all associated resources (caches, etc.). Default behavior deletes everything| | | `--hard-delete` | When to hard delete data (never/now/default/timestamp). Default behavior is a soft delete that allows recovery | | | `--token` | *(Required)* Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | |`--tls-no-verify`| Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TABLE_NAME`: Name of the table to delete ### Delete a table ```bash influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` ### Delete table data only (preserve schema and resources) Delete all data from a table while preserving: * Table schema (column definitions) * Last value caches (LVC) and distinct value caches (DVC) associated with the table This is useful when you want to clear old data and re-write new data to the same schema without recreating the table structure. ```bash influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --data-only \ TABLE_NAME ``` ### Hard delete a table immediately Permanently delete a table and all its data immediately without the ability to recover. ```bash influxdb3 delete table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --hard-delete now \ TABLE_NAME ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --data-only | Delete only data while preserving the table schema and all associated resources (caches, etc.). Default behavior deletes everything | | | --hard-delete | When to hard delete data (never/now/default/timestamp). Default behavior is a soft delete that allows recovery | | | --token | ( Required ) Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 delete token Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/token/ The `influxdb3 delete token` command deletes an authorization token from the InfluxDB 3 Enterprise server. ## Usage ```bash # Syntax influxdb3 delete token [OPTIONS] ``` ## Options | Option | Description |Default| Environment | |-----------------|-------------------------------------------------------------------------------------------------------------|-------|-------------------------| | `--token` | *(Required)* The token for authentication with the InfluxDB 3 Enterprise server | | `INFLUXDB3_AUTH_TOKEN` | | `--token-name` | *(Required)* The name of the token to be deleted | | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | `INFLUXDB3_TLS_CA` | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| |`INFLUXDB3_TLS_NO_VERIFY`| | `-h`, `--help` | Print help information | | | | `--help-all` | Print detailed help information | | | ## Examples In the examples below, replace the following: * `AUTH_TOKEN`: Authentication token with permission to delete tokens * `TOKEN_TO_DELETE`: Name of the token to delete ### Delete a token by name ```bash influxdb3 delete token --token-name TOKEN_TO_DELETE --token AUTH_TOKEN ``` ### Show help for the command ```bash influxdb3 delete token --help ``` #### Related * [Manage tokens](/influxdb3/enterprise/admin/tokens/) * [InfluxDB /api/v3 Token API reference](/influxdb3/enterprise/api/v3/#tag/Token) [cli](/influxdb3/enterprise/tags/cli/) | Option | Description | Default | Environment | | --- | --- | --- | --- | | Option | Description | Default | Environment | | --token | ( Required ) The token for authentication with the InfluxDB 3 Enterprise server | | INFLUXDB3_AUTH_TOKEN | | --token-name | ( Required ) The name of the token to be deleted | | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | INFLUXDB3_TLS_CA | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | INFLUXDB3_TLS_NO_VERIFY | | -h , --help | Print help information | | | | --help-all | Print detailed help information | | | --- # influxdb3 delete trigger Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/delete/trigger/ The `influxdb3 delete trigger` command deletes a processing engine trigger. ## Usage ```bash # Syntax influxdb3 delete trigger [OPTIONS] --database ``` ## Arguments * **TRIGGER\_NAME**: The name of the trigger to delete. ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | | `--force` | Force delete even if the trigger is active | | | `--tls-ca` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Delete a trigger](#delete-a-trigger) * [Force delete an active trigger](#force-delete-an-active-trigger) In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TRIGGER_NAME`: Name of the trigger to delete ### Delete a trigger ```bash influxdb3 delete trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TRIGGER_NAME ``` ### Force delete an active trigger ```bash influxdb3 delete trigger \ --force \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TRIGGER_NAME ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | | --force | Force delete even if the trigger is active | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 disable Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/disable/ The `influxdb3 disable` command disables resources such as a trigger. ## Usage ```bash influxdb3 disable ``` ## Subcommands | Subcommand | Description | |-------------------------------------------------------------------------|----------------------------------------------| |[trigger](/influxdb3/enterprise/reference/cli/influxdb3/disable/trigger/)| Disables a plugin trigger | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | trigger | Disables a plugin trigger | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 disable trigger Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/disable/trigger/ The `influxdb3 disable trigger` command disables a plugin trigger. ## Usage ```bash influxdb3 disable trigger [OPTIONS] --database ``` ## Arguments: * **TRIGGER\_NAME**: Name of the trigger to disable ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 enable Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/enable/ The `influxdb3 enable` command enables resources such as a trigger. ## Usage ```bash influxdb3 enable ``` ## Subcommands | Subcommand | Description | |------------------------------------------------------------------------|----------------------------------------------| |[trigger](/influxdb3/enterprise/reference/cli/influxdb3/enable/trigger/)| Enable a trigger to enable plugin execution | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | trigger | Enable a trigger to enable plugin execution | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 enable trigger Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/enable/trigger/ The `influxdb3 enable trigger` command enables a trigger to enable plugin execution. ## Usage ```bash influxdb3 enable trigger [OPTIONS] --database ``` ## Arguments: * **TRIGGER\_NAME**: Name of the trigger to enable ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 import Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/import/ The `influxdb3 import` command manages bulk imports of Parquet data into InfluxDB 3 Enterprise databases and tables. > [!Important] > Bulk import requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > The target database and table must exist before importing. ## Usage ```bash influxdb3 import ``` ## Subcommands | Subcommand | Description | |--------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------| | [upload](/influxdb3/enterprise/reference/cli/influxdb3/import/upload/) | Upload Parquet files into a database and table | |[from-object-store](/influxdb3/enterprise/reference/cli/influxdb3/import/from-object-store/)|Import Parquet files directly from object storage, without streaming file bytes through the client| | [list](/influxdb3/enterprise/reference/cli/influxdb3/import/list/) | List import jobs | | help | Print command help or the help of a subcommand | ## Permissions * `upload` and `from-object-store` need the `write` action on the target database. * `list` needs the `describe` action on at least one database, and only returns import jobs for databases your token can describe. For details, see [Permissions](/influxdb3/enterprise/admin/import-data/#permissions)in the bulk import guide. ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| #### Related * [Import data](/influxdb3/enterprise/admin/import-data/) | Subcommand | Description | | --- | --- | | Subcommand | Description | | upload | Upload Parquet files into a database and table | | from-object-store | Import Parquet files directly from object storage, without streaming file bytes through the client | | list | List import jobs | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 import from-object-store Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/import/from-object-store/ The `influxdb3 import from-object-store` command imports every Parquet file found recursively under a prefix into an existing InfluxDB 3 Enterprise database and table. InfluxDB 3 Enterprise reads the source files itself, using its own object store credentials, so file bytes never stream through the `influxdb3` client or the machine that runs it. This command needs the `write` action on the target database, for example,`db:DATABASE_NAME:write`. ## Usage ```bash influxdb3 import from-object-store [OPTIONS] ``` ## Arguments | Argument | Description | |----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |``|Where to read the Parquet files from: a prefix in InfluxDB 3 Enterprise’s own object store (for example, `some/prefix`), or an `s3://bucket/prefix` URL. Every other URL scheme is rejected with an HTTP 400 error. Every `.parquet` file found recursively under the prefix is imported as its own job.| ## Options |Option| | Description | Default | Environment variable | |------|-----------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------|-------------------------| | `-H` | `--host ` | Host URL of the InfluxDB 3 Enterprise server |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | |`--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | `-d` |`--database `| Target database name | | | | `-t` | `--table
` | Target table name | | | | | `--column ` |Map a Parquet column to an InfluxDB type: `=`. See [`import upload` column type mapping](/influxdb3/enterprise/reference/cli/influxdb3/import/upload/#column-type-mapping) for supported types. Can be specified multiple times.| | | | | `--tls-ca ` | Path to a custom TLS certificate authority | | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification (not recommended in production) | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | | | `--help-all` | Print detailed help information | | | ## How the server reads the source InfluxDB 3 Enterprise uses its own configured object store credentials to read the source. You don’t send credentials in the command. * When `` names a prefix in InfluxDB 3 Enterprise’s own object store, staging each file is a copy within that same store. * When `` names an external S3 bucket, InfluxDB 3 Enterprise’s own object store must also be S3 (or an S3-compatible endpoint set with`--aws-endpoint`). If it isn’t, InfluxDB 3 Enterprise has no credentials to reach another bucket and rejects the request with an HTTP 400 error. InfluxDB 3 Enterprise’s credentials must also be able to read the source bucket. This is typically automatic within one AWS account, and otherwise possible only when a bucket policy on the source bucket grants that access. For an external S3 bucket, InfluxDB 3 Enterprise first tries a true S3 server-side copy: a single request that moves the file directly between buckets without streaming its bytes through InfluxDB 3 Enterprise. Server-side copy applies only when all of the following hold: * Both the source and destination object stores are S3 (or an S3-compatible endpoint set with `--aws-endpoint`). * InfluxDB 3 Enterprise’s credentials can read the source bucket and write the destination bucket. * The source file is smaller than 5 GiB. When any of these doesn’t hold, InfluxDB 3 Enterprise falls back automatically to streaming the file through itself (never through the`influxdb3` client) and logs a warning. Control this behavior with the[`--import-attempt-server-side-copy`](/influxdb3/enterprise/reference/config-options/#import-attempt-server-side-copy)server option (default `true`). Set it to `false` to always use the streamed fallback. > [!Note] > You can’t use your object store’s root, or the reserved`uploaded_imports/` prefix, as an import source. > Choose a dedicated prefix for your source files. ## Examples * [Import from the server’s own object store](#import-from-the-servers-own-object-store) * [Import from an S3 bucket](#import-from-an-s3-bucket) * [Import with column type mapping](#import-with-column-type-mapping) In the examples below, replace the following: * `DATABASE_NAME`: the name of the target database * `TABLE_NAME`: the name of the target table * `AUTH_TOKEN`: your authentication token ### Import from the server’s own object store ```bash influxdb3 import from-object-store \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ parquet-imports/2025-01/ ``` ### Import from an S3 bucket ```bash influxdb3 import from-object-store \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ s3://my-bucket/parquet-exports/ ``` ### Import with column type mapping ```bash influxdb3 import from-object-store \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ --column timestamp=time \ --column host=tag \ --column cpu_usage=f64 \ s3://my-bucket/parquet-exports/ ``` #### Related * [Import data](/influxdb3/enterprise/admin/import-data/) | Argument | Description | | --- | --- | | Argument | Description | | | Where to read the Parquet files from: a prefix in InfluxDB 3 Enterprise’s own object store (for example, some/prefix ), or an s3://bucket/prefix URL. Every other URL scheme is rejected with an HTTP 400 error. Every .parquet file found recursively under the prefix is imported as its own job. | | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | -H | --host | Host URL of the InfluxDB 3 Enterprise server | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | -d | --database | Target database name | | | | -t | --table
| Target table name | | | | | --column | Map a Parquet column to an InfluxDB type: = . See import upload column type mapping for supported types. Can be specified multiple times. | | | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification (not recommended in production) | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | | | --help-all | Print detailed help information | | | --- # influxdb3 import list Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/import/list/ The `influxdb3 import list` command lists bulk import jobs. InfluxDB 3 Enterprise returns only the import jobs for databases that your token can describe. For example, a token with `db:*:describe` sees import jobs for every database, and a token with `db:DATABASE_NAME:describe` sees only import jobs for `DATABASE_NAME`. This command needs the `describe` action on at least one database, and returns an HTTP 403 error if your token can’t describe any database. ## Usage ```bash influxdb3 import list [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|----------------------|--------------------------------------------------------------------|-----------------------|-------------------------| | `-H` | `--host ` | Host URL of the InfluxDB 3 Enterprise server |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | |`--token `| Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` | Path to a custom TLS certificate authority | | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` |Disable TLS certificate verification (not recommended in production)| |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | | | `--help-all` | Print detailed help information | | | ## Examples Replace the following: * `AUTH_TOKEN`: your authentication token ```bash influxdb3 import list \ --host http://localhost:8181 \ --token AUTH_TOKEN ``` #### Related * [Import data](/influxdb3/enterprise/admin/import-data/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | -H | --host | Host URL of the InfluxDB 3 Enterprise server | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification (not recommended in production) | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | | | --help-all | Print detailed help information | | | --- # influxdb3 import upload Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/import/upload/ The `influxdb3 import upload` command uploads one or more Parquet files into an existing InfluxDB 3 Enterprise database and table. Each file becomes a separate import job. Imported data is written to your object storage and becomes queryable after the compactor processes it. This command needs the `write` action on the target database, for example,`db:DATABASE_NAME:write`. For files that already sit in InfluxDB 3 Enterprise’s own object store, or in an S3 bucket that the server can reach directly, use[`influxdb3 import from-object-store`](/influxdb3/enterprise/reference/cli/influxdb3/import/from-object-store/)instead: the server reads the files itself, so file bytes never stream through the `influxdb3` client. ## Usage ```bash influxdb3 import upload [OPTIONS] --database --table
``` ## Arguments |Argument| Description | |--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |``|*Required*. Path to a Parquet file or a directory, or an object store URL (`s3://bucket/prefix`, `gs://bucket/prefix`, `az://container/prefix`, `file:///path`). If a directory or a URL prefix, all `*.parquet` files are processed recursively and one import job is created per file. URL sources read credentials from the environment; use `--source-opt` to set or override store options.| ## Options |Option| | Description | Default | Environment variable | |------|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------|-------------------------| | `-H` | `--host ` | Host URL of the InfluxDB 3 Enterprise server |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | | `--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | `-d` | `--database ` | Target database name | | | | `-t` | `--table
` | Target table name | | | | | `--column ` |Map a Parquet column to an InfluxDB type: `=`. Supported types: `i64`, `u64`, `f64`, `bool`, `string`, `time`, `tag`. Can be specified multiple times. Unmapped columns default to field values.| | | | | `--source-opt ` | Object store option for a URL source, for example `aws_region=us-west-2`. Overrides environment variables. Can be specified multiple times. Ignored for local paths. | | | | |`--concurrency `| Maximum number of files to upload concurrently | `8` | | | | `--tls-ca ` | Path to a custom TLS certificate authority | | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification (not recommended in production) | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | | | `--help-all` | Print detailed help information | | | ## Column type mapping The `--column` option maps Parquet columns to InfluxDB data types. Specify the option multiple times to map multiple columns. Unmapped columns default to field values. | Type | Description | |--------|-----------------------------| | `i64` | Signed 64-bit integer field | | `u64` |Unsigned 64-bit integer field| | `f64` | 64-bit float field | | `bool` | Boolean field | |`string`| String field | | `time` | Timestamp column | | `tag` | Tag column | ## Examples In the examples below, replace the following: * `DATABASE_NAME`: the name of the target database * `TABLE_NAME`: the name of the target table * `AUTH_TOKEN`: your authentication token ### Upload a single Parquet file ```bash influxdb3 import upload \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ /path/to/data.parquet ``` ### Upload all Parquet files in a directory ```bash influxdb3 import upload \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ /path/to/parquet-directory/ ``` ### Upload with column type mapping ```bash influxdb3 import upload \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ --column timestamp=time \ --column host=tag \ --column cpu_usage=f64 \ /path/to/metrics.parquet ``` ### Upload Parquet files from an S3 bucket Pass an `s3://` URL as the source to upload files that your machine can reach directly. File bytes still stream through the `influxdb3` client. For files that already sit in object storage InfluxDB 3 Enterprise can reach itself, use[`influxdb3 import from-object-store`](/influxdb3/enterprise/reference/cli/influxdb3/import/from-object-store/)instead. ```bash influxdb3 import upload \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database DATABASE_NAME \ --table TABLE_NAME \ --source-opt aws_region=us-west-2 \ s3://my-bucket/parquet-exports/ ``` #### Related * [Import data](/influxdb3/enterprise/admin/import-data/) | Argument | Description | | --- | --- | | Argument | Description | | | Required . Path to a Parquet file or a directory, or an object store URL ( s3://bucket/prefix , gs://bucket/prefix , az://container/prefix , file:///path ). If a directory or a URL prefix, all *.parquet files are processed recursively and one import job is created per file. URL sources read credentials from the environment; use --source-opt to set or override store options. | | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | -H | --host | Host URL of the InfluxDB 3 Enterprise server | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | -d | --database | Target database name | | | | -t | --table
| Target table name | | | | | --column | Map a Parquet column to an InfluxDB type: = . Supported types: i64 , u64 , f64 , bool , string , time , tag . Can be specified multiple times. Unmapped columns default to field values. | | | | | --source-opt | Object store option for a URL source, for example aws_region=us-west-2 . Overrides environment variables. Can be specified multiple times. Ignored for local paths. | | | | | --concurrency | Maximum number of files to upload concurrently | 8 | | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification (not recommended in production) | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | | | --help-all | Print detailed help information | | | | Type | Description | | --- | --- | | Type | Description | | i64 | Signed 64-bit integer field | | u64 | Unsigned 64-bit integer field | | f64 | 64-bit float field | | bool | Boolean field | | string | String field | | time | Timestamp column | | tag | Tag column | --- # influxdb3 install Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/install/ The `influxdb3 install` command and its subcommands manage Python package installations for the InfluxDB 3 Enterprise processing engine. ## Usage ```bash influxdb3 install ``` ## Subcommands | Subcommand | Description | |---------------------------------------------------------------------------|-----------------------------------------------------| |[`package`](/influxdb3/enterprise/reference/cli/influxdb3/install/package/)|Install Python packages for processing engine plugins| | `help` | Print command help or the help of a subcommand | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | package | Install Python packages for processing engine plugins | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 install package Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/install/package/ The `influxdb3 install package` command installs Python packages within the plugin environment for use in [InfluxDB 3 Enterprise processing engine plugins](/influxdb3/enterprise/process/). Use this command to add external dependencies that your plugins require, such as data processing libraries, notification tools, or forecasting packages. ## Usage ```bash # Syntax influxdb3 install package [OPTIONS] [PACKAGES]... ``` ## Arguments * **`[PACKAGES]...`**: One or more package names to install (space-separated) ## Options | Option | Description | Default | Environment Variable | |-----------------------------------------------|-------------------------------------------------------------------------------------------------------------|-----------------------|---------------------------| | `-H`, `--host ` | The host URL of the running InfluxDB 3 Enterprise server |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | `--token ` | The token for authentication with the InfluxDB 3 Enterprise server | | `INFLUXDB3_AUTH_TOKEN` | | `--plugin-dir ` | Location of the plugins directory | `/plugins` | `INFLUXDB3_PLUGIN_DIR` | |`--virtual-env-location `| Custom virtual environment location | | `VIRTUAL_ENV` | | `--package-manager ` | Package manager to use: `discover`, `pip`, `uv`, or `disabled` | `discover` |`INFLUXDB3_PACKAGE_MANAGER`| | `--plugin-repo ` | Plugin repository URL | | `INFLUXDB3_PLUGIN_REPO` | | `-r`, `--requirements ` | Path to a `requirements.txt` file | | | | `--tls-ca ` | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | `INFLUXDB3_TLS_CA` | | `--tls-no-verify` |Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `INFLUXDB3_TLS_NO_VERIFY` | | `-h`, `--help` | Print help information | | | | `--help-all` | Print detailed help information | | | ## Examples ### Install a single package ```bash influxdb3 install package pandas ``` ### Install multiple packages ```bash influxdb3 install package pint pandas requests ``` ### Install packages from a requirements file ```bash influxdb3 install package -r requirements.txt ``` ### Install packages with custom host and authentication ```bash influxdb3 install package \ --host http://localhost:8181 \ --token AUTH_TOKEN \ pint pandas ``` Replace `AUTH_TOKEN` with your [admin token](/influxdb3/enterprise/admin/tokens/)for your InfluxDB 3 Enterprise instance. ### Install packages with a specific package manager ```bash influxdb3 install package \ --package-manager uv \ prophet adtk ``` ### Install packages with a custom CA certificate ```bash influxdb3 install package \ --tls-ca /path/to/ca-cert.pem \ requests ``` ## Package management ### Package manager selection By default (`--package-manager discover`), the CLI automatically detects and uses the best available package manager: 1. **uv** (preferred): Faster package installation 2. **pip** (fallback): Standard Python package manager ### Virtual environment The CLI manages a virtual environment for plugin packages to avoid conflicts with system Python packages. You can customize the virtual environment location with `--virtual-env-location` or the `VIRTUAL_ENV` environment variable. ### Security mode If your InfluxDB 3 Enterprise server was started with [`--package-manager disabled`](/influxdb3/enterprise/reference/config-options/#package-manager), the `influxdb3 install package` command is blocked for security and compliance requirements. When attempting to install packages with this command while the server has package installation disabled, the command fails with a `403 Forbidden` error: ``` Package installation has been disabled. Contact your administrator for more information. ``` The server’s `--package-manager disabled` setting is designed for: * **Enterprise security requirements**: Prevent arbitrary package installation * **Compliance environments**: Control exactly which packages are available * **Air-gapped deployments**: Pre-install all dependencies before deployment * **Multi-tenant scenarios**: Prevent tenants from installing potentially malicious packages In these environments, administrators must pre-install all required Python packages into the server’s virtual environment before starting InfluxDB 3 Enterprise. For more information, see the [`package-manager`](/influxdb3/enterprise/reference/config-options/#package-manager) configuration option. ### Troubleshooting If package installation fails: * **Check if package installation is disabled**: If you receive a `403 Forbidden` error, contact your administrator. Package installation may be disabled on your InfluxDB 3 Enterprise instance. * **Verify network connectivity**: Ensure your InfluxDB 3 Enterprise instance can reach PyPI or your custom package repository * **Check package names**: Verify package names are correct and available in the package repository * **Review logs**: Check InfluxDB 3 Enterprise server logs for detailed error messages * **Test with pip**: Try installing the package directly with `pip` to verify it’s available * **Use requirements file**: For complex dependencies, use a `requirements.txt` file with version pinning * **Check Docker disk space** (Docker environments only): If running InfluxDB 3 Enterprise in Docker and seeing “No space left on device” errors, free up disk space: ``` # Check Docker disk usage docker system df # Remove unused images and build cache docker image prune -af docker buildx prune -af ``` When an install fails, for example for a package that doesn’t exist, InfluxDB 3 Enterprise returns HTTP status `500` with pip’s error message. #### Related * [Processing engine and Python plugins](/influxdb3/enterprise/plugins/) [cli](/influxdb3/enterprise/tags/cli/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[plugins](/influxdb3/enterprise/tags/plugins/) | Option | Description | Default | Environment Variable | | --- | --- | --- | --- | | Option | Description | Default | Environment Variable | | -H , --host | The host URL of the running InfluxDB 3 Enterprise server | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | --token | The token for authentication with the InfluxDB 3 Enterprise server | | INFLUXDB3_AUTH_TOKEN | | --plugin-dir | Location of the plugins directory | /plugins | INFLUXDB3_PLUGIN_DIR | | --virtual-env-location | Custom virtual environment location | | VIRTUAL_ENV | | --package-manager | Package manager to use: discover , pip , uv , or disabled | discover | INFLUXDB3_PACKAGE_MANAGER | | --plugin-repo | Plugin repository URL | | INFLUXDB3_PLUGIN_REPO | | -r , --requirements | Path to a requirements.txt file | | | | --tls-ca | Path to a custom TLS certificate authority (for self-signed or internal certificates) | | INFLUXDB3_TLS_CA | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | INFLUXDB3_TLS_NO_VERIFY | | -h , --help | Print help information | | | | --help-all | Print detailed help information | | | --- # influxdb3 loadcap Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap` to capture and manage anonymized workload profiles on a query-capable InfluxDB 3 Enterprise node. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap ``` ## Subcommands | Subcommand | Description | |---------------------------------------------------------------------------|----------------------------------------------| | [start](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/start/) | Start a timed capture | | [list](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/list/) | List capture profiles | | [files](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/files/) | List files in a profile | | [preview](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/preview/) | Preview a profile | |[download](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/download/)| Download a profile archive | | [delete](/influxdb3/enterprise/reference/cli/influxdb3/loadcap/delete/) | Delete a profile | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Subcommand | Description | | --- | --- | | Subcommand | Description | | start | Start a timed capture | | list | List capture profiles | | files | List files in a profile | | preview | Preview a profile | | download | Download a profile archive | | delete | Delete a profile | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 loadcap delete Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/delete/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap delete` to delete a workload capture profile. You can’t delete a profile while it’s still capturing. Wait for the capture to finish, then retry. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap delete --profile-id [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|---------------------------|------------------------------------------|-----------------------|-------------------------| | |`--profile-id `| Capture profile identifier | | | | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | | `--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` |Path to a custom TLS certificate authority| | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | | --profile-id | Capture profile identifier | | | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 loadcap download Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/download/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap download` to download a workload capture profile. If you omit `--output`, the command writes `.tar.gz` to the current directory. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap download --profile-id [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|---------------------------|------------------------------------------|-----------------------|-------------------------| | |`--profile-id `| Capture profile identifier | | | | `-o` | `--output ` | Output archive path | `.tar.gz` | | | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | | `--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` |Path to a custom TLS certificate authority| | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | | --profile-id | Capture profile identifier | | | | -o | --output | Output archive path | .tar.gz | | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 loadcap files Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/files/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap files` to list files in a workload capture profile. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap files --profile-id [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|---------------------------|------------------------------------------|-----------------------|-------------------------| | |`--profile-id `| Capture profile identifier | | | | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | | `--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` |Path to a custom TLS certificate authority| | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | | --profile-id | Capture profile identifier | | | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 loadcap list Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/list/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap list` to list capture profiles stored by a query-capable InfluxDB 3 Enterprise node. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap list [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|----------------------|------------------------------------------|-----------------------|-------------------------| | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | |`--token `| Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` |Path to a custom TLS certificate authority| | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 loadcap preview Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/preview/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap preview` to inspect a workload capture profile without downloading it. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap preview --profile-id [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|---------------------------|------------------------------------------|-----------------------|-------------------------| | |`--profile-id `| Capture profile identifier | | | | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | | `--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` |Path to a custom TLS certificate authority| | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | | --profile-id | Capture profile identifier | | | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 loadcap start Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/loadcap/start/ * InfluxDB 3 Enterprise v3.10.0+ Use `influxdb3 loadcap start` to capture write requests, query requests, or both on a query-capable InfluxDB 3 Enterprise node. Only one capture can run on a node at a time. > [!Note] > Load capture requires the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/)—the default for new clusters. > On clusters that started on 3.10 or earlier, first run the [storage engine upgrade](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree) (`--upgrade-pacha-tree`). > Send requests to a node with an explicit `--mode` setting that includes `query`, for example, `--mode query` or `--mode ingest --mode query --mode compact`. > Load capture isn’t available on a node that uses the default `--mode all` configuration. ## Usage ```bash influxdb3 loadcap start --type --duration [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|-----------------------|-----------------------------------------------------------------------------|-----------------------|-------------------------| | | `--type ` | Request types to capture: `write`, `query`, or `both` | | | | |`--duration `|Capture duration, for example, `30s`, `5m`, or `1h`. The maximum is one hour.| | | | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | |`--token ` | Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` | Path to a custom TLS certificate authority | | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | ## Example ```bash influxdb3 loadcap start \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --type both \ --duration 5m ``` #### Related * [Capture workload data](/influxdb3/enterprise/admin/load-capture/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | | --type | Request types to capture: write , query , or both | | | | | --duration | Capture duration, for example, 30s , 5m , or 1h . The maximum is one hour. | | | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 manage Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/manage/ * InfluxDB 3 Enterprise v3.11.1+ Use `influxdb3 manage` to perform maintenance operations on an InfluxDB 3 Enterprise cluster. > [!Note] > These commands act on the [upgraded storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) (PachaTree) migration. > They have no effect on a cluster that has never run the storage engine upgrade. ## Usage ```bash influxdb3 manage ``` ## Subcommands | Subcommand | Description | |----------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------| | [cleanup-parquet](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/) |Remove pre-upgrade Parquet data after a storage engine upgrade| |[retry-upgrade-to-pacha-tree](/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/)| Retry a failed storage engine upgrade | | help | Print command help or the help of a subcommand | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| #### Related * [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) | Subcommand | Description | | --- | --- | | Subcommand | Description | | cleanup-parquet | Remove pre-upgrade Parquet data after a storage engine upgrade | | retry-upgrade-to-pacha-tree | Retry a failed storage engine upgrade | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 manage cleanup-parquet Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/ * InfluxDB 3 Enterprise v3.11.1+ Use `influxdb3 manage cleanup-parquet` to permanently remove Parquet-era data that a completed [storage engine upgrade](/influxdb3/enterprise/reference/internals/storage-engine/) (Parquet-to-PachaTree) left behind. Any node in the cluster accepts the request, but the compactor node executes the cleanup. > [!Important] > #### This deletes data permanently > > * Parquet files superseded by the PachaTree migration are deleted. > * The operation is irreversible. > * After cleanup, `influxdb3 manage downgrade-to-parquet` is no longer possible. > > Run with `--dry-run` first to see what a real cleanup would delete. ## Usage ```bash influxdb3 manage cleanup-parquet [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|----------------------|-------------------------------------------------------------------------------------------------------------|-----------------------|-------------------------| | | `--dry-run` | Report what would be deleted without deleting anything | | | | | `--yes` | Skip the confirmation prompt. Conflicts with `--dry-run` | | | | | `--wait` | Poll every 5 seconds until the cleanup completes or fails. Exits non-zero on failure | | | | | `--status-only` |Print the status of the current or last cleanup, then exit. Conflicts with `--dry-run`, `--yes`, and `--wait`| | | | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | |`--token `| Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` | Path to a custom TLS certificate authority | | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | ## Examples ### Preview what a cleanup would delete ```bash influxdb3 manage cleanup-parquet \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --dry-run ``` ### Run a cleanup and wait for it to finish ```bash influxdb3 manage cleanup-parquet \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --yes \ --wait ``` ### Check the status of the current or last cleanup ```bash influxdb3 manage cleanup-parquet \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --status-only ``` #### Related * [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | | --dry-run | Report what would be deleted without deleting anything | | | | | --yes | Skip the confirmation prompt. Conflicts with --dry-run | | | | | --wait | Poll every 5 seconds until the cleanup completes or fails. Exits non-zero on failure | | | | | --status-only | Print the status of the current or last cleanup, then exit. Conflicts with --dry-run , --yes , and --wait | | | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 manage retry-upgrade-to-pacha-tree Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/ * InfluxDB 3 Enterprise v3.11.2+ Use `influxdb3 manage retry-upgrade-to-pacha-tree` to reset a [storage engine upgrade](/influxdb3/enterprise/reference/internals/storage-engine/) (Parquet-to-PachaTree) that latched to a `failed` state. The command re-checks every previously failed upgrade source against the catalog and object store: * Sources that are still readable are requeued. * Sources whose table or database was dropped are recorded as skipped. * Sources that can’t be read back are left unresolved for the next retry. The reset itself only changes durable state. The migration resumes the next time a compactor node starts with `--upgrade-pacha-tree`. Starting in v3.11.8 and v3.12.2, the command refuses to run while any ingest node in the upgrade lists Parquet files it couldn’t convert. Follow the [retry procedure](#retry-after-an-ingest-conversion-failure) to resolve these failures before you run the command. ## Usage ```bash influxdb3 manage retry-upgrade-to-pacha-tree [OPTIONS] ``` ## Options |Option| | Description | Default | Environment variable | |------|----------------------|------------------------------------------|-----------------------|-------------------------| | `-H` | `--host ` | InfluxDB 3 Enterprise server URL |`http://127.0.0.1:8181`| `INFLUXDB3_HOST_URL` | | |`--token `| Authentication token | | `INFLUXDB3_AUTH_TOKEN` | | | `--tls-ca ` |Path to a custom TLS certificate authority| | `INFLUXDB3_TLS_CA` | | | `--tls-no-verify` | Disable TLS certificate verification | |`INFLUXDB3_TLS_NO_VERIFY`| | `-h` | `--help` | Print help information | | | ## Example ```bash influxdb3 manage retry-upgrade-to-pacha-tree \ --host http://localhost:8181 \ --token AUTH_TOKEN ``` If any sources are left unresolved, run the command again to re-attempt them. ## Retry after an ingest conversion failure Starting in v3.11.8 and v3.12.2, an ingest node reports exhausted conversion attempts in `system.upgrade_parquet_node`. The cluster upgrade remains `failed` until you reset it with this command. To reset the failed upgrade and resume the migration, follow these steps: 1. [Query `system.upgrade_parquet_node`](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables)and check `last_message` for the file and error. 2. Fix the cause of the conversion failure. 3. Restart each affected ingest node to retry its files. 4. Run `influxdb3 manage retry-upgrade-to-pacha-tree` as shown in the [example](#example). If the command reports that an ingest node still lists unconverted files, resolve that node’s failure and retry. 5. Restart a compactor node with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet)to resume the migration. #### Related * [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) * [influxdb3 manage cleanup-parquet](/influxdb3/enterprise/reference/cli/influxdb3/manage/cleanup-parquet/) | Option | | Description | Default | Environment variable | | --- | --- | --- | --- | --- | | Option | | Description | Default | Environment variable | | -H | --host | InfluxDB 3 Enterprise server URL | http://127.0.0.1:8181 | INFLUXDB3_HOST_URL | | | --token | Authentication token | | INFLUXDB3_AUTH_TOKEN | | | --tls-ca | Path to a custom TLS certificate authority | | INFLUXDB3_TLS_CA | | | --tls-no-verify | Disable TLS certificate verification | | INFLUXDB3_TLS_NO_VERIFY | | -h | --help | Print help information | | | --- # influxdb3 query Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/query/ The `influxdb3 query` command executes a query against a running InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 query [OPTIONS] --database [QUERY]... ``` ##### Aliases `query`, `q` ## Arguments * **QUERY**: The query to execute. Provide the query in one of the following ways: * a string * a path to a file that contains the query using the `--file` option * from stdin ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-l` | `--language` | Query language of the query string (`sql` *(default)* or `influxql`) | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, `parquet`) | | `-o` | `--output` | Output query results to the specified file | | `-f` | `--file` | A file that contains the query to execute | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Query data using SQL](#query-data-using-sql) * [Query data using InfluxQL](#query-data-using-influxql) * [Query data and return JSON-formatted results](#query-data-and-return-json-formatted-results) * [Query data and write results to a file](#query-data-and-write-results-to-a-file) In the examples below, replace`DATABASE_NAME`: with the name of the database to query. ### Query data using SQL [string](#string)[file](#file)[stdin](#stdin) ```bash influxdb3 query \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'SELECT * FROM home' ``` ```bash influxdb3 query \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./query.sql ``` ```bash cat ./query.sql | influxdb3 query \ --database DATABASE_NAME \ --token AUTH_TOKEN \ ``` ### Query data using InfluxQL [string](#string-2)[file](#file-2)[stdin](#stdin-2) ```bash influxdb3 query \ --language influxql \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'SELECT * FROM home' ``` ```bash influxdb3 query \ --language influxql \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./query.influxql ``` ```bash cat ./query.influxql | influxdb3 query \ --language influxql \ --database DATABASE_NAME \ --token AUTH_TOKEN ``` ### Query data and return JSON-formatted results [string](#string-3)[file](#file-3)[stdin](#stdin-3) ```bash influxdb3 query \ --format json \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'SELECT * FROM home' ``` ```bash influxdb3 query \ --format json \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./query.sql ``` ```bash cat ./query.sql | influxdb3 query \ --format json \ --database DATABASE_NAME \ --token AUTH_TOKEN \ ``` ### Query data and write results to a file [string](#string-4)[file](#file-4)[stdin](#stdin-4) ```bash influxdb3 query \ --output /path/to/results.txt \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'SELECT * FROM home' ``` ```bash influxdb3 query \ --output /path/to/results.txt \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./query.sql ``` ```bash cat ./query.sql | influxdb3 query \ --output /path/to/results.txt \ --database DATABASE_NAME \ --token AUTH_TOKEN ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -l | --language | Query language of the query string ( sql (default) or influxql ) | | | --format | Output format ( pretty (default) , json , jsonl , csv , parquet ) | | -o | --output | Output query results to the specified file | | -f | --file | A file that contains the query to execute | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 remove Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/remove/ The `influxdb3 remove` command removes resources from your InfluxDB 3 Enterprise cluster. ## Usage ```bash influxdb3 remove ``` ## Subcommands | Subcommand | Description | |------------------------------------------------------------------|----------------------------------------------| |[node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)| Remove a stopped node from the cluster | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | node | Remove a stopped node from the cluster | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 remove node Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/remove/node/ The `influxdb3 remove node` command permanently removes a **stopped** node from your InfluxDB 3 Enterprise cluster. The cluster drains the node’s data (up to its final snapshot) before the catalog entry and the node’s object-store file paths are purged. > [!Important] > #### Stop the node gracefully before removing it > > A node must be `stopped` before it can be removed. > Use [`influxdb3 stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)against the live node and wait for it to reach `stopped`—the graceful stop > is what drains the node’s[write-ahead log (WAL) tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail). > A node stuck in `stopping` (for example, because the process was killed > mid-stop or was already dead) cannot be removed normally. > To recover such a node safely, follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/). ## Usage ```bash influxdb3 remove node [OPTIONS] --node-id ``` ## Options |Option| | Description | |------|------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------| | | `--node-id` | *(Required)* The node ID to remove | | | `--no-confirm` | Skip the confirmation prompt | | |`--force-finalize`|Force removal of a node that did not shut down cleanly–**this can lose data** (see [Force removal](#force-removal-of-a-node-that-did-not-shut-down-cleanly))| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Behavior When you remove a node: 1. The command verifies the node is `stopped` and marks it `removing` in the catalog. 2. The cluster drains the node’s data up to the node’s final snapshot. 3. The node’s catalog entry and object-store file paths are purged. Removal permanently deletes the node’s object-store file paths, including its WAL files. Any acknowledged writes that were **not** covered by the node’s final snapshot are deleted with them. That is why the node must complete a graceful stop first. > [!Important] > #### On the Parquet engine, removal completes after the storage engine upgrade > > On the Parquet engine, the node stays `removing` until the upgraded > storage engine’s compactor starts. > While the node is `removing`, it can’t register again under the same node ID. > The upgraded compactor removes the catalog entry, but leaves the node’s > Parquet files in object storage. > After catalog removal, `influxdb3 manage cleanup-parquet` also leaves > those files in place. > See[Node removal on the Parquet engine](/influxdb3/enterprise/admin/node-lifecycle/#removal-on-the-parquet-engine-completes-after-the-storage-engine-upgrade). ### Removal is refused if unsnapshotted WAL remains (upgraded engine) On clusters that have fully adopted the upgraded storage engine (the default for new clusters), InfluxDB 3 Enterprise **refuses** (HTTP 409) to remove a`stopped` node that still has an unsnapshotted[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail): ```text node 'NODE_ID' has unsnapshotted WAL (wal file N, snapshotted through M); restart the node so it snapshots the tail, or force the removal (set force_finalize=true, or pass --force-finalize on the CLI) to remove it and lose the unsnapshotted writes ``` To resolve this safely, follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/): restart the node with the same `--node-id`, stop it gracefully so it snapshots the tail, and then remove it. > [!Note] > This safeguard applies only to clusters that have fully adopted the > upgraded storage engine (new clusters, or after the storage engine upgrade > completes). > Parquet clusters and clusters still mid-upgrade are **not** guarded. > On those clusters, a graceful[`stop node`](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) before > removal is the only protection against losing the[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail). > > To determine which engine your cluster runs, start with how it was created: > new clusters on InfluxDB 3 Enterprise 3.11+ default to the upgraded storage > engine, and clusters that started on 3.10 or earlier keep the Parquet engine > until you restart them with `--upgrade-pacha-tree`. > If you started a storage engine upgrade, confirm it finished—query`system.upgrade_parquet_node` and check that every node reports `completed`. > See [Query system data](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables). For more information about the upgraded storage engine, see[Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). ## Force removal of a node that did not shut down cleanly If `remove node` fails without `--force-finalize`, the node never finished stopping—its process is gone and never confirmed a clean stop.`--force-finalize` removes it anyway. **This can lose data**: recently acknowledged writes the node had not yet captured in a snapshot are deleted with it. If the node had finished snapshotting and only failed to report a clean stop, nothing is lost. Prefer the safe path first: restart the node with the same `--node-id`, let it shut down cleanly, and remove it again. See [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/). Use `--force-finalize` only when the node cannot be brought back and you accept the possible loss. ## Examples * [Remove a stopped node](#remove-a-stopped-node) * [Remove a node without confirmation](#remove-a-node-without-confirmation) * [Force removal of a node that cannot be recovered](#force-removal-of-a-node-that-cannot-be-recovered) In the examples below, replace the following: * `NODE_ID`: The node identifier of the node to remove * `AUTH_TOKEN`: Authentication token with sufficient privileges ### Remove a stopped node First stop the node gracefully, then remove it: ```bash # Stop the node and wait for it to reach the stopped state influxdb3 stop node --node-id NODE_ID # Remove the stopped node from the cluster influxdb3 remove node --node-id NODE_ID ``` The command prompts for confirmation. ### Remove a node without confirmation ```bash influxdb3 remove node --node-id NODE_ID --no-confirm ``` ### Force removal of a node that cannot be recovered Only use `--force-finalize` after attempting[crash recovery](/influxdb3/enterprise/admin/recover-node/), when the node cannot be brought back and you accept the possible loss of unsnapshotted writes: ```bash influxdb3 remove node --node-id NODE_ID --force-finalize ``` #### Related * [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) * [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) * [influxdb3 show nodes](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/) | Option | | Description | | --- | --- | --- | | Option | | Description | | | --node-id | ( Required ) The node ID to remove | | | --no-confirm | Skip the confirmation prompt | | | --force-finalize | Force removal of a node that did not shut down cleanly– this can lose data (see Force removal ) | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 serve Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/serve/ The `influxdb3 serve` command starts the InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 serve [OPTIONS] ``` ## Required Parameters * **node-id**: A unique identifier for your server instance. Must be unique for any hosts sharing the same object store. * **cluster-id**: A unique identifier for your cluster. Must be different from any node-id in your cluster. * **object-store**: Determines where time series data is stored. * Other object store parameters depending on the selected `object-store` type. > [!Note] > `--node-id` and `--cluster-id` support alphanumeric strings with optional hyphens. > [!Important] > #### Global configuration options > > Some configuration options (like [`--num-io-threads`](/influxdb3/enterprise/reference/config-options/#num-io-threads)) are **global** and must be specified **before** the `serve` command: > > ``` > influxdb3 --num-io-threads=8 serve --node-id=node0 --cluster-id=cluster0 --verbose > ``` > > See [Global configuration options](/influxdb3/enterprise/reference/config-options/#global-configuration-options) for the complete list. ## Options |Option| | Description | |------|-----------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | | `--admin-token-recovery-http-bind` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#admin-token-recovery-http-bind)* | | | `--admin-token-recovery-tcp-listener-file-path` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#admin-token-recovery-tcp-listener-file-path)* | | | `--admin-token-file` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#admin-token-file)* | | | `--aws-access-key-id` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-access-key-id)* | | | `--aws-allow-http` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-allow-http)* | | | `--aws-credentials-file` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-credentials-file)* | | | `--aws-default-region` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-default-region)* | | | `--aws-endpoint` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-endpoint)* | | | `--aws-secret-access-key` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-secret-access-key)* | | | `--aws-session-token` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-session-token)* | | | `--aws-skip-signature` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#aws-skip-signature)* | | | `--azure-allow-http` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#azure-allow-http)* | | | `--azure-endpoint` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#azure-endpoint)* | | | `--azure-storage-access-key` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#azure-storage-access-key)* | | | `--azure-storage-account` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#azure-storage-account)* | | | `--bucket` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#bucket)* | | | `--catalog-sync-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#catalog-sync-interval)* | | | `--cluster-id` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#cluster-id)* | | | `--compaction-check-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compaction-check-interval)* | | | `--compaction-cleanup-wait` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compaction-cleanup-wait)* | | | `--compaction-gen2-duration` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compaction-gen2-duration)* | | | `--compaction-max-num-files-per-plan` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compaction-max-num-files-per-plan)* | | | `--compaction-multipliers` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compaction-multipliers)* | | | `--compaction-row-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compaction-row-limit)* | | | `--compactor-dispatch-ack-timeout` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-dispatch-ack-timeout)* | | | `--compactor-dispatch-target` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-dispatch-target)* | | | `--compactor-sweep-audit-retention` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-sweep-audit-retention)* | | | `--compactor-sweep-auto-resume` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-sweep-auto-resume)* | | | `--compactor-sweep-grace` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-sweep-grace)* | | | `--compactor-sweep-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-sweep-interval)* | | | `--compactor-sweep-mode` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#compactor-sweep-mode)* | | | `--conn-info` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#conn-info)* | | | `--data-dir` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#data-dir)* | | | `--datafusion-config` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#datafusion-config)* | | | `--datafusion-max-parquet-fanout` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#datafusion-max-parquet-fanout)* | | | `--datafusion-num-threads` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#datafusion-num-threads)* | | | `--datafusion-runtime-disable-lifo-slot` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-event-interval` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-global-queue-interval` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-max-blocking-threads` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-max-io-events-per-tick` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-thread-keep-alive` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-thread-priority` | Development-only Tokio runtime configuration | | | `--datafusion-runtime-type` | Development-only Tokio runtime configuration | | | `--datafusion-use-cached-parquet-loader` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#datafusion-use-cached-parquet-loader)* | | | `--delete-grace-period` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#delete-grace-period)* | | | `--disable-authz` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#disable-authz)* | | | `--disable-file-cache` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#disable-file-cache)* | | | `--distinct-cache-eviction-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#distinct-cache-eviction-interval)* | | | `--distinct-value-cache-disable-from-history` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#distinct-value-cache-disable-from-history)* | | | `--exec-mem-pool-size` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#exec-mem-pool-size)* | | | `--force-snapshot-mem-size` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#force-snapshot-mem-size)* | | | `--gen1-duration` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#gen1-duration)* | | | `--gen1-lookback-duration` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#gen1-lookback-duration)* | | | `--google-service-account` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#google-service-account)* | | | `--hard-delete-default-duration` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#hard-delete-default-duration)* | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | | | `--http-bind` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#http-bind)* | | | `--import-attempt-server-side-copy` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#import-attempt-server-side-copy)* | | | `--jwt-default-ttl-seconds` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#jwt-default-ttl-seconds)* | | | `--jwt-issuer` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#jwt-issuer)* | | | `--jwt-key-id` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#jwt-key-id)* | | | `--jwt-private-key` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#jwt-private-key)* | | | `--last-cache-eviction-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#last-cache-eviction-interval)* | | | `--last-value-cache-disable-from-history` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#last-value-cache-disable-from-history)* | | | `--license-email` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#license-email)* | | | `--license-file` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#license-file)* | | | `--log-destination` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#log-destination)* | | | `--log-filter` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#log-filter)* | | | `--log-format` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#log-format)* | | | `--max-concurrent-queries` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#max-concurrent-queries)* | | | `--max-http-request-size` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#max-http-request-size)* | | | `--mode` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#mode)* | | | `--node-id` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#node-id)* | | | `--node-id-from-env` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#node-id-from-env)* | | | `--num-cores` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#num-cores)* | | | `--num-datafusion-threads` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#datafusion-num-threads)* | | | `--num-database-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#num-database-limit)* | | | `--num-table-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#num-table-limit)* | | | `--num-total-columns-per-table-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#num-total-columns-per-table-limit)* | | | `--oauth-audience` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#oauth-audience)* | | | `--oauth-client-id` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#oauth-client-id)* | | | `--oauth-issuer` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#oauth-issuer)* | | | `--oauth-scopes` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#oauth-scopes)* | | | `--object-store` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#object-store)* | | | `--object-store-connection-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#object-store-connection-limit)* | | | `--object-store-http2-max-frame-size` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#object-store-http2-max-frame-size)* | | | `--object-store-http2-only` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#object-store-http2-only)* | | | `--object-store-max-retries` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#object-store-max-retries)* | | | `--object-store-retry-timeout` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#object-store-retry-timeout)* | | | `--package-manager` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#package-manager)* | | | `--parquet-mem-cache-prune-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#parquet-mem-cache-prune-interval)* | | | `--parquet-mem-cache-prune-percentage` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#parquet-mem-cache-prune-percentage)* | | | `--file-cache-recency` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#file-cache-recency)* | | | `--file-cache-size` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#file-cache-size)* | | | `--permission-tokens-file` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#permission-tokens-file)* | | | `--plugin-dir` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#plugin-dir)* | | | `--plugin-dir-only` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#plugin-dir-only)* | | | `--plugin-repo` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#plugin-repo)* | | | `--preemptive-cache-age` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#preemptive-cache-age)* | | |`--processing-engine-restart-state-snapshot-interval`| *See [configuration options](/influxdb3/enterprise/reference/config-options/#processing-engine-restart-state-snapshot-interval)* | | | `--query-file-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#query-file-limit)* | | | `--query-log-max-entries` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#query-log-max-entries)* | | | `--rbac-authoring-disabled` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#rbac-authoring-disabled)* | | | `--replication-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#replication-interval)* | | | `--restrict-plugin-triggers-to` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#restrict-plugin-triggers-to)* | | | `--retention-check-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#retention-check-interval)* | | | `--snapshotted-wal-files-to-keep` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#snapshotted-wal-files-to-keep)* | | | `--table-index-cache-concurrency-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#table-index-cache-concurrency-limit)* | | | `--table-index-cache-max-entries` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#table-index-cache-max-entries)* | | | `--tcp-listener-file-path` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#tcp-listener-file-path)* | | | `--disable-telemetry-upload` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#disable-telemetry-upload)* | | | `--telemetry-endpoint` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#telemetry-endpoint)* | | | `--tls-cert` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#tls-cert)* | | | `--tls-key` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#tls-key)* | | | `--tls-minimum-version` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#tls-minimum-version)* | | | `--traces-exporter` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-exporter)* | | | `--traces-exporter-jaeger-agent-host` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-exporter-jaeger-agent-host)* | | | `--traces-exporter-jaeger-agent-port` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-exporter-jaeger-agent-port)* | | | `--traces-exporter-jaeger-service-name` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-exporter-jaeger-service-name)* | | |`--traces-exporter-jaeger-trace-context-header-name` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-exporter-jaeger-trace-context-header-name)* | | | `--traces-jaeger-debug-name` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-jaeger-debug-name)* | | | `--traces-jaeger-max-msgs-per-second` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-jaeger-max-msgs-per-second)* | | | `--traces-jaeger-tags` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#traces-jaeger-tags)* | | | `--trigger-retry-max-attempts` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#trigger-retry-max-attempts)* | | | `--trigger-work-silence-timeout` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#trigger-work-silence-timeout)* | | | `--upgrade-pacha-tree` |Migrate existing Parquet data to the upgraded storage engine (the default for new clusters). Replaces the deprecated `--use-pacha-tree`. *See [configuration options](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree).*| | | `--user-auth-type` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#user-auth-type)* | | | `--virtual-env-location` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#virtual-env-location)* | | | `--wait-for-running-ingester` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#wait-for-running-ingester)* | | | `--wal-flush-interval` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#wal-flush-interval)* | | | `--wal-max-buffered-writes` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#wal-max-buffered-writes)* | | | `--wal-replay-concurrency-limit` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#wal-replay-concurrency-limit)* | | | `--wal-replay-fail-on-error` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#wal-replay-fail-on-error)* | | | `--wal-files-per-snapshot` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#wal-files-per-snapshot)* | | | `--webui-public-uri` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#webui-public-uri)* | | | `--webui-session-secret` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#webui-session-secret)* | | | `--without-auth` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#without-auth)* | | | `--without-user-auth` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#without-user-auth)* | | | `--write-timestamp-max-future` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#write-timestamp-max-future)* | | | `--write-timestamp-max-past` | *See [configuration options](/influxdb3/enterprise/reference/config-options/#write-timestamp-max-past)* | ### Option environment variables You can use environment variables to define most `influxdb3 serve` options. For more information, see[Configuration options](/influxdb3/enterprise/reference/config-options/). ## Quick-Start Mode For development, testing, and home use, you can start InfluxDB 3 Enterprise by running `influxdb3` without the `serve` subcommand or any configuration parameters. The system automatically generates required values: * **`node-id`**: `{hostname}-node` (fallback: `primary-node`) * **`cluster-id`**: `{hostname}-cluster` (fallback: `primary-cluster`) * **`object-store`**: `file` * **`data-dir`**: `~/.influxdb` The system displays warning messages showing the auto-generated identifiers: ``` Using auto-generated node id: mylaptop-node. For production deployments, explicitly set --node-id Using auto-generated cluster id: mylaptop-cluster. For production deployments, explicitly set --cluster-id ``` ### Quick-start examples ```bash # Zero-config startup influxdb3 # Override specific defaults influxdb3 --object-store memory # Use environment variables to override defaults INFLUXDB3_NODE_ID=my-node influxdb3 ``` > [!Important] > #### Production deployments > > Quick-start mode is designed for development and testing environments. > For production deployments, use explicit configuration with the `serve` subcommand > and specify all required parameters as shown in the [Examples](#examples) below. **Configuration precedence**: CLI flags \> environment variables \> auto-generated defaults For more information about quick-start mode, see [Get started](/influxdb3/enterprise/get-started/setup/#quick-start-mode-development). ## Examples * [Run the InfluxDB 3 server](#run-the-influxdb-3-server) * [Run the InfluxDB 3 server with extra verbose logging](#run-the-influxdb-3-server-with-extra-verbose-logging) * [Run InfluxDB 3 with debug logging using LOG\_FILTER](#run-influxdb-3-with-debug-logging-using-log_filter) In the examples below, replace the following: * `my-host-01`: a unique string that identifies your InfluxDB 3 Enterprise server. * `my-cluster-01`: a unique string that identifies your InfluxDB 3 Enterprise cluster. The value you use must be different from `--node-id` values in the cluster. ### Run the InfluxDB 3 server ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 ``` ### Run a server in specific modes ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 \ --mode ingest,query ``` ### Run a server specifically for the Processing Engine ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 \ --plugin-dir ~/.influxdb3/plugins \ --mode process ``` For the complete setup, see [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). ### Run a server specifically for compacting data ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 \ --mode compact ``` ### Run the InfluxDB 3 server with extra verbose logging ```bash influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 --verbose ``` ### Run InfluxDB 3 with debug logging using LOG\_FILTER ```bash LOG_FILTER=debug influxdb3 serve \ --object-store file \ --data-dir ~/.influxdb3 \ --node-id my-host-01 \ --cluster-id my-cluster-01 ``` ## Troubleshooting ### Common Issues * **Error: “cluster-id cannot match any node-id in the cluster”** Ensure your `--cluster-id` value is different from all `--node-id` values in your cluster. * **Error: “Failed to connect to object store”** Verify your `--object-store` setting and ensure all required parameters for that storage type are provided. * **Permission errors when using S3, Google Cloud, or Azure storage** Check that your authentication credentials are correct and have sufficient permissions. #### Related * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) | Option | | Description | | --- | --- | --- | | Option | | Description | | | --admin-token-recovery-http-bind | See configuration options | | | --admin-token-recovery-tcp-listener-file-path | See configuration options | | | --admin-token-file | See configuration options | | | --aws-access-key-id | See configuration options | | | --aws-allow-http | See configuration options | | | --aws-credentials-file | See configuration options | | | --aws-default-region | See configuration options | | | --aws-endpoint | See configuration options | | | --aws-secret-access-key | See configuration options | | | --aws-session-token | See configuration options | | | --aws-skip-signature | See configuration options | | | --azure-allow-http | See configuration options | | | --azure-endpoint | See configuration options | | | --azure-storage-access-key | See configuration options | | | --azure-storage-account | See configuration options | | | --bucket | See configuration options | | | --catalog-sync-interval | See configuration options | | | --cluster-id | See configuration options | | | --compaction-check-interval | See configuration options | | | --compaction-cleanup-wait | See configuration options | | | --compaction-gen2-duration | See configuration options | | | --compaction-max-num-files-per-plan | See configuration options | | | --compaction-multipliers | See configuration options | | | --compaction-row-limit | See configuration options | | | --compactor-dispatch-ack-timeout | See configuration options | | | --compactor-dispatch-target | See configuration options | | | --compactor-sweep-audit-retention | See configuration options | | | --compactor-sweep-auto-resume | See configuration options | | | --compactor-sweep-grace | See configuration options | | | --compactor-sweep-interval | See configuration options | | | --compactor-sweep-mode | See configuration options | | | --conn-info | See configuration options | | | --data-dir | See configuration options | | | --datafusion-config | See configuration options | | | --datafusion-max-parquet-fanout | See configuration options | | | --datafusion-num-threads | See configuration options | | | --datafusion-runtime-disable-lifo-slot | Development-only Tokio runtime configuration | | | --datafusion-runtime-event-interval | Development-only Tokio runtime configuration | | | --datafusion-runtime-global-queue-interval | Development-only Tokio runtime configuration | | | --datafusion-runtime-max-blocking-threads | Development-only Tokio runtime configuration | | | --datafusion-runtime-max-io-events-per-tick | Development-only Tokio runtime configuration | | | --datafusion-runtime-thread-keep-alive | Development-only Tokio runtime configuration | | | --datafusion-runtime-thread-priority | Development-only Tokio runtime configuration | | | --datafusion-runtime-type | Development-only Tokio runtime configuration | | | --datafusion-use-cached-parquet-loader | See configuration options | | | --delete-grace-period | See configuration options | | | --disable-authz | See configuration options | | | --disable-file-cache | See configuration options | | | --distinct-cache-eviction-interval | See configuration options | | | --distinct-value-cache-disable-from-history | See configuration options | | | --exec-mem-pool-size | See configuration options | | | --force-snapshot-mem-size | See configuration options | | | --gen1-duration | See configuration options | | | --gen1-lookback-duration | See configuration options | | | --google-service-account | See configuration options | | | --hard-delete-default-duration | See configuration options | | -h | --help | Print help information | | | --help-all | Print detailed help information | | | --http-bind | See configuration options | | | --import-attempt-server-side-copy | See configuration options | | | --jwt-default-ttl-seconds | See configuration options | | | --jwt-issuer | See configuration options | | | --jwt-key-id | See configuration options | | | --jwt-private-key | See configuration options | | | --last-cache-eviction-interval | See configuration options | | | --last-value-cache-disable-from-history | See configuration options | | | --license-email | See configuration options | | | --license-file | See configuration options | | | --log-destination | See configuration options | | | --log-filter | See configuration options | | | --log-format | See configuration options | | | --max-concurrent-queries | See configuration options | | | --max-http-request-size | See configuration options | | | --mode | See configuration options | | | --node-id | See configuration options | | | --node-id-from-env | See configuration options | | | --num-cores | See configuration options | | | --num-datafusion-threads | See configuration options | | | --num-database-limit | See configuration options | | | --num-table-limit | See configuration options | | | --num-total-columns-per-table-limit | See configuration options | | | --oauth-audience | See configuration options | | | --oauth-client-id | See configuration options | | | --oauth-issuer | See configuration options | | | --oauth-scopes | See configuration options | | | --object-store | See configuration options | | | --object-store-connection-limit | See configuration options | | | --object-store-http2-max-frame-size | See configuration options | | | --object-store-http2-only | See configuration options | | | --object-store-max-retries | See configuration options | | | --object-store-retry-timeout | See configuration options | | | --package-manager | See configuration options | | | --parquet-mem-cache-prune-interval | See configuration options | | | --parquet-mem-cache-prune-percentage | See configuration options | | | --file-cache-recency | See configuration options | | | --file-cache-size | See configuration options | | | --permission-tokens-file | See configuration options | | | --plugin-dir | See configuration options | | | --plugin-dir-only | See configuration options | | | --plugin-repo | See configuration options | | | --preemptive-cache-age | See configuration options | | | --processing-engine-restart-state-snapshot-interval | See configuration options | | | --query-file-limit | See configuration options | | | --query-log-max-entries | See configuration options | | | --rbac-authoring-disabled | See configuration options | | | --replication-interval | See configuration options | | | --restrict-plugin-triggers-to | See configuration options | | | --retention-check-interval | See configuration options | | | --snapshotted-wal-files-to-keep | See configuration options | | | --table-index-cache-concurrency-limit | See configuration options | | | --table-index-cache-max-entries | See configuration options | | | --tcp-listener-file-path | See configuration options | | | --disable-telemetry-upload | See configuration options | | | --telemetry-endpoint | See configuration options | | | --tls-cert | See configuration options | | | --tls-key | See configuration options | | | --tls-minimum-version | See configuration options | | | --traces-exporter | See configuration options | | | --traces-exporter-jaeger-agent-host | See configuration options | | | --traces-exporter-jaeger-agent-port | See configuration options | | | --traces-exporter-jaeger-service-name | See configuration options | | | --traces-exporter-jaeger-trace-context-header-name | See configuration options | | | --traces-jaeger-debug-name | See configuration options | | | --traces-jaeger-max-msgs-per-second | See configuration options | | | --traces-jaeger-tags | See configuration options | | | --trigger-retry-max-attempts | See configuration options | | | --trigger-work-silence-timeout | See configuration options | | | --upgrade-pacha-tree | Migrate existing Parquet data to the upgraded storage engine (the default for new clusters). Replaces the deprecated --use-pacha-tree . See configuration options . | | | --user-auth-type | See configuration options | | | --virtual-env-location | See configuration options | | | --wait-for-running-ingester | See configuration options | | | --wal-flush-interval | See configuration options | | | --wal-max-buffered-writes | See configuration options | | | --wal-replay-concurrency-limit | See configuration options | | | --wal-replay-fail-on-error | See configuration options | | | --wal-files-per-snapshot | See configuration options | | | --webui-public-uri | See configuration options | | | --webui-session-secret | See configuration options | | | --without-auth | See configuration options | | | --without-user-auth | See configuration options | | | --write-timestamp-max-future | See configuration options | | | --write-timestamp-max-past | See configuration options | --- # influxdb3 show Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/ The `influxdb3 show` command lists resources in your InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 show ``` ## Subcommands | Subcommand | Description | |--------------------------------------------------------------------------|----------------------------------------------| |[databases](/influxdb3/enterprise/reference/cli/influxdb3/show/databases/)| List database | | [plugins](/influxdb3/enterprise/reference/cli/influxdb3/show/plugins/) | List loaded plugins | | [system](/influxdb3/enterprise/reference/cli/influxdb3/show/system/) | Display system table data | | [tokens](/influxdb3/enterprise/reference/cli/influxdb3/show/tokens/) | List authentication tokens | | help |Print command help or the help of a subcommand| Additional InfluxDB 3 Enterprise subcommands: | Subcommand | Description | |----------------------------------------------------------------------|---------------------------| |[license](/influxdb3/enterprise/reference/cli/influxdb3/show/license/)|Display license information| | [nodes](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/) | Display node information | | query\_groups | List distributed query groups (not yet operational) | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | databases | List database | | plugins | List loaded plugins | | system | Display system table data | | tokens | List authentication tokens | | help | Print command help or the help of a subcommand | | Subcommand | Description | | --- | --- | | Subcommand | Description | | license | Display license information | | nodes | Display node information | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 show databases Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/databases/ The `influxdb3 show databases` command lists databases in your InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 show databases [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--token` | *(Required)* Authentication token | | |`--show-deleted` | Include databases marked as deleted in the output | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [List all databases](#list-all-databases) * [List all databases, including deleted databases](#list-all-databases-including-deleted-databases) * [List databases in JSON-formatted output](#list-databases-in-json-formatted-output) ### List all databases ```bash influxdb3 show databases ``` ### List all databases, including deleted databases ```bash influxdb3 show databases --show-deleted ``` ### List databases in JSON-formatted output ```bash influxdb3 show databases --format json ``` ### List databases in Parquet format output [Parquet](https://parquet.apache.org/) is a binary format. Use the `--output` option to specify the file where you want to save the Parquet data. ``` influxdb3 show databases --format parquet \ --output /Users/me/databases.parquet ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --token | ( Required ) Authentication token | | | --show-deleted | Include databases marked as deleted in the output | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 show license Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/license/ The `influxdb3 show license` command displays license information for your InfluxDB 3 Enterprise instance. ## Usage ```bash influxdb3 show license [OPTIONS] ``` ## Options |Option| | Description | |------|----------------|-----------------------------------------------------------------------------------------| | `-H` | `--host` |Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`)| | | `--cluster-id` | *(Required)* Cluster identifier | | | `--node-id` | *(Required)* Node identifier | | |`--object-store`| *(Required)* Object store type (file, memory, s3, gcs, azure) | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | > [!Note] > **CLI help documentation bug in v3.2.0** > > The `influxdb3 show license --help` output in v3.2.0 does not display the required `--object-store`, `--cluster-id`, and `--node-id` options and related object store configuration options. > This command requires object store configuration and cluster/node identification to function properly. ### Additional object store options Depending on the `--object-store` type specified, additional configuration options may be required: * **S3**: AWS credentials and bucket configuration * **GCS**: Google Cloud credentials and bucket configuration * **Azure**: Azure credentials and container configuration * **File**: Local file system path configuration ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |----------------------|---------| | `INFLUXDB3_HOST_URL` |`--host` | |`INFLUXDB3_AUTH_TOKEN`|`--token`| ## Examples ### Display license information with file object store ```bash influxdb3 show license \ --cluster-id CLUSTER_ID \ --node-id NODE_ID \ --object-store file \ --token AUTH_TOKEN ``` In the example above, replace the following: * `AUTH_TOKEN`: Authentication token * `CLUSTER_ID`: Your cluster identifier * `NODE_ID`: Your node identifier The command displays information about your Enterprise license, including license type, expiration date, and usage limits. | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --cluster-id | ( Required ) Cluster identifier | | | --node-id | ( Required ) Node identifier | | | --object-store | ( Required ) Object store type (file, memory, s3, gcs, azure) | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | --- # influxdb3 show nodes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/ The `influxdb3 show nodes` command displays information about nodes in your InfluxDB 3 Enterprise cluster, including their state, mode, and resource usage. ## Usage ```bash # Syntax influxdb3 show nodes [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--output` | Path where to save output when using the `parquet` format | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Output The command displays the following information for each node: * **node\_id**: The unique identifier for the node * **node\_catalog\_id**: The internal catalog identifier * **instance\_id**: The unique instance identifier * **mode**: The node’s operating modes (ingest, query, process, compact) * **core\_count**: Number of CPU cores allocated to the node * **state**: Current node state (`running` or `stopped`) * **updated\_at**: Timestamp of the last update * **cli\_params**: Command-line parameters used to start the node ## Examples * [List all nodes in pretty format](#list-all-nodes-in-pretty-format) * [List nodes in JSON format](#list-nodes-in-json-format) * [Export nodes data to Parquet format](#export-nodes-data-to-parquet-format) * [List nodes on a remote server](#list-nodes-on-a-remote-server) ### List all nodes in pretty format ```bash influxdb3 show nodes ``` ``` +----------+-----------------+--------------------------------------+--------+------------+---------+---------------------------+ | node_id | node_catalog_id | instance_id | mode | core_count | state | updated_at | +----------+-----------------+--------------------------------------+--------+------------+---------+---------------------------+ | node-1 | 0 | e38944e4-1204-4bb4-92f3-71138894d674 | ingest | 1 | running | 2025-09-04T10:15:57.126 | | node-2 | 1 | f5418c97-eb6d-47b5-8176-efc0ad7b882e | ingest | 1 | stopped | 2025-09-04T10:16:57.503 | +----------+-----------------+--------------------------------------+--------+------------+---------+---------------------------+ ``` ### List nodes in JSON format ```bash influxdb3 show nodes --format json ``` The output is similar to the following: ```json [ { "node_id": "node-1", "node_catalog_id": 0, "instance_id": "e38944e4-1204-4bb4-92f3-71138894d674", "mode": [ "ingest" ], "core_count": 1, "state": "running", "updated_at": "2025-09-04T10:15:57.126", "cli_params": "{\"http-bind\":\"127.0.0.1:8181\",\"node-id\":\"node-1\",\"data-dir\":\"/path/to/data\",\"object-store\":\"file\",\"mode\":\"ingest\"}" }, { "node_id": "node-2", "node_catalog_id": 1, "instance_id": "f5418c97-eb6d-47b5-8176-efc0ad7b882e", "mode": [ "ingest" ], "core_count": 1, "state": "stopped", "updated_at": "2025-09-04T10:16:57.503", "cli_params": "{\"http-bind\":\"127.0.0.1:8182\",\"node-id\":\"node-2\",\"data-dir\":\"/path/to/data\",\"object-store\":\"file\",\"mode\":\"ingest\"}" } ] ``` ### Export nodes data to Parquet format [Parquet](https://parquet.apache.org/) is a binary format. Use the `--output` option to specify the file where you want to save the Parquet data. ```bash influxdb3 show nodes \ --format parquet \ --output nodes-data.parquet ``` ### List nodes on a remote server ```bash influxdb3 show nodes \ --host INFLUXDB_HOST \ --token AUTH_TOKEN ``` Replace the following: * `AUTH_TOKEN`: Authentication token with sufficient privileges * `INFLUXDB_HOST`: Host URL of the running InfluxDB 3 Enterprise server #### Related * [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) * [influxdb3 stop node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/) * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --output | Path where to save output when using the parquet format | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 show plugins Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/plugins/ The `influxdb3 show plugins` command lists loaded Processing Engine plugins in your InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 show plugins [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--token` | *(Required)* Authentication token | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--output` | Path where to save output when using the `parquet` format | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Output The command returns information about loaded plugin files: * **plugin\_name**: Name of a trigger using this plugin * **file\_name**: Plugin filename * **file\_path**: Full server path to the plugin file * **size\_bytes**: File size in bytes * **last\_modified**: Last modification timestamp (milliseconds since epoch) > [!Note] > This command queries the `system.plugin_files` table in the `_internal` database. > For more advanced queries and filtering, see [Query system data](/influxdb3/enterprise/admin/query-system-data/). ## Examples * [List all plugins](#list-all-plugins) * [List plugins in different output formats](#list-plugins-in-different-output-formats) * [Output plugins to a Parquet file](#output-plugins-to-a-parquet-file) ### List all plugins ```bash influxdb3 show plugins ``` ### List plugins in different output formats You can specify the output format using the `--format` option: ```bash # JSON format influxdb3 show plugins --format json # JSON Lines format influxdb3 show plugins --format jsonl # CSV format influxdb3 show plugins --format csv ``` ### Output plugins to a Parquet file [Parquet](https://parquet.apache.org/) is a binary format. Use the `--output` option to specify the file where you want to save the Parquet data. ```bash influxdb3 show plugins \ --format parquet \ --output /Users/me/plugins.parquet ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --token | ( Required ) Authentication token | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --output | Path where to save output when using the parquet format | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 show retention Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/retention/ The `influxdb3 show retention` command displays effective retention periods for tables in your InfluxDB 3 Enterprise server. ## Usage ```bash # Syntax influxdb3 show retention [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--token` | *(Required)* Authentication token | | | `--database` | Filter retention information by database name | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Show retention for all tables](#show-retention-for-all-tables) * [Show retention for a specific database](#show-retention-for-a-specific-database) * [Show retention in JSON format](#show-retention-in-json-format) * [Export retention data to Parquet format](#export-retention-data-to-parquet-format) In the examples below, replace `AUTH_TOKEN` with your authentication token. ### Show retention for all tables ```bash influxdb3 show retention \ --host http://localhost:8181 \ --token AUTH_TOKEN ``` ### Show retention for a specific database ```bash influxdb3 show retention \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --database mydb ``` ### Show retention in JSON format ```bash influxdb3 show retention \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --format json ``` ### Export retention data to Parquet format [Parquet](https://parquet.apache.org/) is a binary format. When using the `parquet` format, data is written to standard output by default. Use output redirection or the `--output` option to save the data to a file. ```bash influxdb3 show retention \ --host http://localhost:8181 \ --token AUTH_TOKEN \ --format parquet \ --output retention-data.parquet ``` ## Output The command displays the following information for each table: * **Database**: The database name * **Table**: The table name * **Retention**: The *effective* retention period in human-readable format (for example, “7d” for 7 days, “24h” for 24 hours, “infinite” for no retention) * **Source**: Where the retention is defined (`table`, `database`, or `infinite`) ### Example output ``` Database | Table | Retention | Source ---------|-------------|-----------|---------- mydb | cpu | 7d | database mydb | mem | 24h | table mydb | disk | infinite | infinite ``` Tables with table-level retention policies override the database-level retention. Tables without explicit retention policies inherit the database retention or have infinite retention if none is set. | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --token | ( Required ) Authentication token | | | --database | Filter retention information by database name | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 show system Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/system/ The `influxdb3 show system` command displays data from InfluxDB 3 Enterprise system tables. ## Usage ``` influxdb3 show system [OPTIONS] --database ``` ##### Aliases `system`, `s` ## Subcommands | Subcommand | Description | |-----------------------------------------------------------------------------------|----------------------------------------------| | [summary](/influxdb3/enterprise/reference/cli/influxdb3/show/system/summary) | Summarize system table data | | [table](/influxdb3/enterprise/reference/cli/influxdb3/show/system/table/) |Retrieve entries from a specific system table | |[table-list](/influxdb3/enterprise/reference/cli/influxdb3/show/system/table-list/)| List available system tables | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-----------------------------------------------------------------------------------------| | `-H` | `--host` |Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`)| | `-d` |`--database`| *(Required)* Name of the database to operate on | | | `--token` | Authentication token | | `-h` | `--help` | Print help information | | |`--help-all`| Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`|`--database`| | `INFLUXDB3_AUTH_TOKEN` | `--token` | | Subcommand | Description | | --- | --- | | Subcommand | Description | | summary | Summarize system table data | | table | Retrieve entries from a specific system table | | table-list | List available system tables | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | Authentication token | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | --- # influxdb3 show system summary Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/system/summary/ The `influxdb3 show system summary` command returns a summary of various types of system table data. ## Usage ```bash influxdb3 show system --database summary [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-l` | `--limit` | Maximum number of entries from each table to display (default is `10`, `0` indicates no limit) | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ## Examples * [Summarize system table data](#summarize-system-table-data) * [Summarize system table data in JSON-formatted output](#summarize-system-table-data-in-json-formatted-output) In the examples below, replace`DATABASE_NAME` with the name of the database to operate on. ### Summarize system table data ```bash influxdb3 show system --database DATABASE_NAME summary ``` ### Summarize system table data in JSON-formatted output ```bash influxdb3 show system --database DATABASE_NAME summary --format json ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -l | --limit | Maximum number of entries from each table to display (default is 10 , 0 indicates no limit) | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 show system table-list Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/system/table-list/ The `influxdb3 show system table-list` command lists available system tables. ## Usage ```bash influxdb3 show system --database table-list [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ## Examples * [List system tables](#list-system-tables) * [List system tables in JSON-formatted output](#list-system-tables-in-json-formatted-output) In the examples below, replace`DATABASE_NAME` with the name of the database to operate on. ### List system tables ```bash influxdb3 show system --database DATABASE_NAME summary ``` ### List system tables in JSON-formatted output ```bash influxdb3 show system --database DATABASE_NAME summary --format json ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 show system table Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/system/table/ The `influxdb3 show system table` command queries data from a system table. ## Usage ```bash influxdb3 show system --database table [OPTIONS] ``` ## Arguments * **SYSTEM\_TABLE**: the system table to query ## Options |Option| | Description | |------|-----------------|--------------------------------------------------------------------------------------------------------------| | `-l` | `--limit` | Maximum number of tables entries to display (default is `10`, `0` indicates no limit) | | `-o` | `--order-by` | Order by the specified columns | | `-s` | `--select` | Select specific columns from the system table | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification. (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ## Examples * [Query a system table](#query-a-system-table) * [Query specific columns from a system table](#query-specific-columns-from-a-system-table) * [Query a system table and order by a specific column](#query-a-system-table-and-order-by-a-specific-column) * [Query a system table and return JSON-formatted output](#query-a-system-table-and-return-json-formatted-output) In the examples below, replace`DATABASE_NAME` with the name of the database to operate on. ### Query a system table ```bash # Query the parquet_files system table influxdb3 show system --database DATABASE_NAME table parquet_files ``` ### Query specific columns from a system table ```bash # Select specific columns from the parquet_files system table influxdb3 show system \ --database DATABASE_NAME \ table \ --select table_name,size_bytes,row_count \ parquet_files ``` ### Query a system table and order by a specific column ```bash influxdb3 show system \ --database DATABASE_NAME \ table \ --order-by size_bytes,row_count \ parquet_files ``` ### Query a system table and return JSON-formatted output ```bash influxdb3 show system \ --database DATABASE_NAME \ table \ --format json \ parquet_files ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -l | --limit | Maximum number of tables entries to display (default is 10 , 0 indicates no limit) | | -o | --order-by | Order by the specified columns | | -s | --select | Select specific columns from the system table | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification. ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 show tokens Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/show/tokens/ The `influxdb3 show tokens` command lists authentication tokens in your InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 show tokens [OPTIONS] ``` ## Options |Option| | Description | |------|-----------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--token` | *(Required)* Authentication token | | | `--format` | Output format (`pretty` *(default)*, `json`, `jsonl`, `csv`, or `parquet`) | | | `--output` | Path where to save output when using the `parquet` format | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`|Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [List all tokens](#list-all-tokens) * [List tokens in different output formats](#list-tokens-in-different-output-formats) * [Output tokens to a Parquet file](#output-tokens-to-a-parquet-file) ### List all tokens ```bash influxdb3 show tokens ``` ### List tokens in different output formats You can specify the output format using the `--format` option: ```bash # JSON format influxdb3 show tokens --format json # JSON Lines format influxdb3 show tokens --format jsonl # CSV format influxdb3 show tokens --format csv ``` ### Output tokens to a Parquet file [Parquet](https://parquet.apache.org/) is a binary format. Use the `--output` option to specify the file where you want to save the Parquet data. ```bash influxdb3 show tokens \ --format parquet \ --output /Users/me/tokens.parquet ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --token | ( Required ) Authentication token | | | --format | Output format ( pretty (default) , json , jsonl , csv , or parquet ) | | | --output | Path where to save output when using the parquet format | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 stop Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/stop/ The `influxdb3 stop` command gracefully stops nodes in your InfluxDB 3 Enterprise cluster. ## Usage ```bash influxdb3 stop ``` ## Subcommands | Subcommand | Description | |----------------------------------------------------------------|----------------------------------------------| |[node](/influxdb3/enterprise/reference/cli/influxdb3/stop/node/)| Gracefully stop a node | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | node | Gracefully stop a node | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 stop node Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/stop/node/ The `influxdb3 stop node` command requests a graceful, two-phase stop of a node in your InfluxDB 3 Enterprise cluster. When you run this command against a **live** node: 1. The node is marked as `stopping` in the catalog. 2. The node completes its stop cascade, then confirms the stop. The cascade drains the node’s [WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail)—the writes buffered in the write-ahead log (WAL) since the last snapshot. 3. The node reads as `stopped` in the catalog, and its licensed cores are freed for other nodes. By default, the command waits until the node reaches the `stopped` state (up to [`--timeout`](#options), default `5m`) and exits with a non-zero status if the node does not reach `stopped` in time. Use `--no-wait` to return as soon as the stop request is accepted. > [!Important] > #### Run this command against the live node—do not kill it first > > `stop node` is how a node drains its[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail). > If you kill the process first (for example, with `kill -9` or by > force-stopping a container) and run `stop node` afterward, the dead process > cannot drain anything: recently acknowledged writes that were not yet > flushed or snapshotted remain stranded in the WAL, and[removing the node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)can then permanently delete them. > If a node has already stopped ungracefully, follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) instead. > [!Note] > Sending the server process a bare `SIGTERM` (without calling `stop node`) > does not force a WAL snapshot on the upgraded storage engine. > Always use `stop node` to stop a cluster node gracefully. ## Usage ```bash influxdb3 stop node [OPTIONS] --node-id ``` ## Options |Option| | Description | |------|-----------------|------------------------------------------------------------------------------------------------------------------| | | `--node-id` | *(Required)* The node ID to stop | | | `--no-confirm` | Skip the confirmation prompt (`--force` is a deprecated alias) | | | `--timeout` |How long to wait for the node to reach `stopped` before giving up—for example: `30s`, `5m`, `1h` (default is `5m`)| | | `--no-wait` | Return as soon as the stop request is accepted instead of waiting for `stopped` | | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify`| Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Behavior The node lifecycle is two-phase: `stop node` marks the node `stopping`, but the node only reads as `stopped` after its own stop cascade completes and it confirms the stop. That cascade is what drains the node’s[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail). How the node drains it depends on the storage engine: on the Parquet engine, the node persists the buffered writes to Parquet files; on the upgraded storage engine, it captures them in a WAL snapshot. * By default, the command polls the node state (from `system.nodes`) until it reaches `stopped`, up to `--timeout` (default `5m`). * If the node does not reach `stopped` within the timeout, the command prints the last observed state and exits with a non-zero status. The node may be wedged in `stopping`—for example, if the process died mid-cascade. To recover, follow[Recover a crashed node](/influxdb3/enterprise/admin/recover-node/). * With `--no-wait`, the command returns after the stop request is accepted. Verify that the node reaches `stopped` before removing it. * Other nodes in the cluster see the state change after their catalog sync interval (default 1 second, set with `--catalog-sync-interval`). * The command requires authentication if the server has auth enabled. Running `stop node` against a node whose process is **already dead** cannot drain anything: the catalog still moves the node toward `stopped`, but the[WAL tail](/influxdb3/enterprise/reference/internals/durability/#wal-tail)remains at risk until the node is restarted. See [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/). ## Examples * [Stop a node gracefully](#stop-a-node-gracefully) * [Stop a node without waiting](#stop-a-node-without-waiting) * [Stop a node without confirmation](#stop-a-node-without-confirmation) * [Stop a node on a remote server](#stop-a-node-on-a-remote-server) In the examples below, replace the following: * `NODE_ID`: The node identifier of the node to stop * `AUTH_TOKEN`: Authentication token with sufficient privileges * `INFLUXDB_HOST`: Host URL of the running InfluxDB 3 Enterprise server ### Stop a node gracefully Run `stop node` against the live node. The command prompts for confirmation, requests the stop, and waits for the node to finish its final flush and confirm the stop: ```bash influxdb3 stop node --node-id NODE_ID ``` After the node reads as `stopped`, you can[remove it from the cluster](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/)if you intend to permanently remove it. ### Stop a node without waiting Use `--no-wait` to return as soon as the stop request is accepted. Verify that the node reaches `stopped` (for example, with[`influxdb3 show nodes`](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/)) before removing it: ```bash influxdb3 stop node --node-id NODE_ID --no-wait ``` ### Stop a node without confirmation ```bash influxdb3 stop node --node-id NODE_ID --no-confirm ``` ### Stop a node on a remote server ```bash influxdb3 stop node \ --host INFLUXDB_HOST \ --node-id NODE_ID \ --token AUTH_TOKEN ``` ## Verify node status Verify the node state using the[`influxdb3 show nodes`](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/) command: ```bash influxdb3 show nodes ``` A gracefully stopped node appears with `state: "stopped"` in the output. #### Related * [Manage the node lifecycle](/influxdb3/enterprise/admin/node-lifecycle/) * [influxdb3 remove node](/influxdb3/enterprise/reference/cli/influxdb3/remove/node/) * [Recover a crashed node](/influxdb3/enterprise/admin/recover-node/) * [influxdb3 show nodes](/influxdb3/enterprise/reference/cli/influxdb3/show/nodes/) * [influxdb3 serve](/influxdb3/enterprise/reference/cli/influxdb3/serve/) | Option | | Description | | --- | --- | --- | | Option | | Description | | | --node-id | ( Required ) The node ID to stop | | | --no-confirm | Skip the confirmation prompt ( --force is a deprecated alias) | | | --timeout | How long to wait for the node to reach stopped before giving up—for example: 30s , 5m , 1h (default is 5m ) | | | --no-wait | Return as soon as the stop request is accepted instead of waiting for stopped | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 test Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/test/ The `influxdb3 test` command tests InfluxDB 3 resources, such as plugins. ## Usage ```bash influxdb3 test ``` ## Subcommands | Subcommand | Description | |-----------------------------------------------------------------------------|----------------------------------------------| |[wal\_plugin](/influxdb3/enterprise/reference/cli/influxdb3/test/wal_plugin/)| Test a write-ahead log (WAL) plugin | | help |Print command help or the help of a subcommand| ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | wal_plugin | Test a write-ahead log (WAL) plugin | | help | Print command help or the help of a subcommand | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 test schedule_plugin Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/test/schedule_plugin/ The `influxdb3 test schedule_plugin` command tests a schedule plugin. Use this command to verify plugin behavior without creating a trigger. ## Usage ```bash influxdb3 test schedule_plugin [OPTIONS] --database ``` ## Arguments * **FILENAME**: Path to the plugin file. Use the absolute path or the path relative to the current working directory, such as `/.py`. ## Options |Option| Flag | Description | |------|-------------------|----------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | URL of the running InfluxDB 3 Enterprise server
(default: `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database you want to test the plugin against | | | `--token` | *(Required)* Authentication token | | |`--input-arguments`|Comma-separated list of `key=value` pairs to pass as plugin input arguments (for example, `key1=val1,key2=val2`)| | | `--schedule` | Cron schedule to simulate when testing the plugin
(default: `* * * * *`) | | | `--cache-name` | Optional cache name to associate with the test | | | `--tls-ca` | Path to a custom TLS certificate authority for self-signed certs | | | `--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Show basic help information | | | `--help-all` | Show all available help options | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable |Corresponding Option| |-------------------------|--------------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | | `INFLUXDB3_TLS_CA` | `--tls-ca` | |`INFLUXDB3_TLS_NO_VERIFY`| `--tls-no-verify` | ## Examples In the examples below, replace the following: * `DATABASE_NAME`: Your target database * `AUTH_TOKEN`: Your authentication token * `PLUGIN_DIR`: the path to the plugin directory you provided when starting the server * `FILENAME`: Plugin file name ### Test a schedule plugin ```bash influxdb3 test schedule_plugin \ --database DATABASE_NAME \ --token AUTH_TOKEN \ PLUGIN_DIR/FILENAME.py ``` ### Test with input arguments and a custom cron schedule You can pass input arguments to your plugin as key-value pairs and specify a custom cron schedule (using Quartz cron syntax with six fields): ```bash influxdb3 test schedule_plugin \ --host http://localhost:8182 \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --input-arguments threshold=10,unit=seconds \ --schedule "0 0 * * * ?" \ PLUGIN_DIR/FILENAME.py ``` * Pass plugin parameters using `--input-arguments` as comma-separated key=value pairs. * Use `--schedule` to set the plugin’s execution time with a Quartz cron expression. For example, “0 0 \* \* \* ?” runs the plugin at the start of every hour. | Option | Flag | Description | | --- | --- | --- | | Option | Flag | Description | | -H | --host | URL of the running InfluxDB 3 Enterprise server (default: http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database you want to test the plugin against | | | --token | ( Required ) Authentication token | | | --input-arguments | Comma-separated list of key=value pairs to pass as plugin input arguments (for example, key1=val1,key2=val2 ) | | | --schedule | Cron schedule to simulate when testing the plugin (default: * * * * * ) | | | --cache-name | Optional cache name to associate with the test | | | --tls-ca | Path to a custom TLS certificate authority for self-signed certs | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Show basic help information | | | --help-all | Show all available help options | | Environment Variable | Corresponding Option | | --- | --- | | Environment Variable | Corresponding Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_CA | --tls-ca | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 test wal_plugin Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/test/wal_plugin/ The `influxdb3 test wal_plugin` command tests a write-ahead log (WAL) plugin. ## Usage ```bash influxdb3 test wal_plugin [OPTIONS] --database ``` ## Arguments * **PLUGIN\_NAME**: The name of the plugin file on the server–for example:`/.py` ## Options |Option| | Description | |------|-------------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | | `--lp` | Line protocol to use as input | | | `--file` | Line protocol file to use as input | | |`--input-arguments`| Comma-separated list of `key=value` pairs to pass as plugin input arguments | | | `--cache-name` | Name of the cache to use in the test | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | `--tls-no-verify` |Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | | `INFLUXDB3_TLS_CA` | `--tls-ca` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Test a WAL plugin](#test-a-wal-plugin) * [Test a WAL plugin with a line protocol string](#test-a-wal-plugin-with-a-line-protocol-string) * [Test a WAL plugin with a file containing line protocol](#test-a-wal-plugin-with-a-file-containing-line-protocol) * [Test a WAL plugin using input arguments](#test-a-wal-plugin-using-input-arguments) In the examples below, replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `PLUGIN_DIR`: Plugin directory name * `PLUGIN_NAME`: Plugin file name ### Test a WAL plugin ```bash influxdb3 test wal_plugin \ --database DATABASE_NAME \ --token AUTH_TOKEN \ PLUGIN_DIR/PLUGIN_NAME.py ``` ### Test a WAL plugin with a line protocol string ```bash influxdb3 test wal_plugin \ --lp 'home,room=Kitchen temp=21.0,hum=35.9,co=0i' \ --database DATABASE_NAME \ --token AUTH_TOKEN \ PLUGIN_DIR/PLUGIN_NAME.py ``` ### Test a WAL plugin with a file containing line protocol ```bash influxdb3 test wal_plugin \ --file PLUGIN_DIR/PLUGIN_NAME_test/input-file.lp` --database DATABASE_NAME \ --token AUTH_TOKEN \ PLUGIN_DIR/PLUGIN_NAME.py ``` ### Test a WAL plugin using input arguments ```bash influxdb3 test wal_plugin \ --input-arguments arg1=foo,arg2=baz \ --database DATABASE_NAME \ --token AUTH_TOKEN \ PLUGIN_DIR/PLUGIN_NAME.py ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | | --lp | Line protocol to use as input | | | --file | Line protocol file to use as input | | | --input-arguments | Comma-separated list of key=value pairs to pass as plugin input arguments | | | --cache-name | Name of the cache to use in the test | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_CA | --tls-ca | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 update Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/update/ The `influxdb3 update` command updates resources in your InfluxDB 3 Enterprise instance. ## Usage ```bash influxdb3 update ``` ## Subcommands | Subcommand | Description | |--------------------------------------------------------------------------|-----------------| |[database](/influxdb3/enterprise/reference/cli/influxdb3/update/database/)|Update a database| | [table](/influxdb3/enterprise/reference/cli/influxdb3/update/table/) | Update a table | | [trigger](/influxdb3/enterprise/reference/cli/influxdb3/update/trigger/) |Update a trigger | | query\_group | Update a distributed query group (not yet operational) | | help | Print command help or the help of a subcommand | ## Options |Option| | Description | |------|------------|-------------------------------| | `-h` | `--help` | Print help information | | |`--help-all`|Print detailed help information| | Subcommand | Description | | --- | --- | | Subcommand | Description | | database | Update a database | | table | Update a table | | trigger | Update a trigger | | Option | | Description | | --- | --- | --- | | Option | | Description | | -h | --help | Print help information | | | --help-all | Print detailed help information | --- # influxdb3 update database Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/update/database/ The `influxdb3 update database` command updates an existing database in your InfluxDB 3 Enterprise instance. Use this command to update a database’s retention period. ## Usage ```bash influxdb3 update database [OPTIONS] --database ``` ## Arguments * **`DATABASE_NAME`**: (Required) The name of the database to update. You can also set the database name using the `INFLUXDB3_DATABASE_NAME` environment variable. ## Options |Option| | Description | |------|--------------------|---------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | The name of the database to update | | | `--token` | Authentication token | | `-r` |`--retention-period`|The retention period as a [duration](/influxdb3/enterprise/reference/glossary/#duration) value (for example: `30d`, `24h`) or `none` to clear| | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | `--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables instead of providing CLI options directly: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | | `INFLUXDB3_TLS_CA` | `--tls-ca` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples The following examples show how to update a database. In your commands replace the following: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token ### Update a database retention period Updates a database retention period to 30 days. ```bash influxdb3 update database --retention-period 30d --database DATABASE_NAME ``` ### Clear a database retention period Removes the retention period from a database by setting it to `none`. ```bash influxdb3 update database --retention-period none --database DATABASE_NAME ``` ### Update a database with authentication Updates a database using an authentication token. ```bash influxdb3 update database --token AUTH_TOKEN --retention-period 7d --database DATABASE_NAME ``` #### Related * [Update database API](/influxdb3/enterprise/api/database/#operation/update_database) * [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | The name of the database to update | | | --token | Authentication token | | -r | --retention-period | The retention period as a duration value (for example: 30d , 24h ) or none to clear | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_CA | --tls-ca | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 update table Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/update/table/ The `influxdb3 update table` command updates an existing table in a database in your InfluxDB 3 Enterprise instance. Use this command to add tag and field columns to a table or update a table’s retention period . ## Usage ```bash influxdb3 update table [OPTIONS] --database ``` ## Arguments * **`TABLE_NAME`**: (Required) The name of the table to update ## Options |Option| | Description | |------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | The name of the database containing the table | | | `--token` | Authentication token | | `-r` |`--retention-period`| The retention period as a [duration](/influxdb3/enterprise/reference/glossary/#duration) value (for example: `30d`, `24h`) or `none` to clear | | | `--tags` | Comma-separated list of tag columns to add to the table. Takes one or more values, so place the table name before it or follow it with another option | | `-f` | `--fields` |Comma-separated list of field columns and their [data types](/influxdb3/enterprise/reference/cli/influxdb3/create/table/#field-data-types) to add to the table| | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | `--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | > [!Important] > #### Add columns and update retention in separate commands > > `--tags` and `--fields` add columns to the table.`--retention-period` updates the table’s retention period. > These are separate operations against separate API endpoints, so > InfluxDB 3 Enterprise rejects a command that combines `--retention-period`with `--tags` or `--fields`. > Run them as separate commands. ### Option environment variables You can use the following environment variables instead of providing CLI options directly: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | | `INFLUXDB3_TLS_CA` | `--tls-ca` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples The following examples show how to update a table. In your commands replace the following: * `DATABASE_NAME`: Database name * `TABLE_NAME`: Table name * `AUTH_TOKEN`: Authentication token ### Add tag columns to a table Adds two tag columns to a table. This works whether the database uses implicit or explicit[schema mode](/influxdb3/enterprise/reference/cli/influxdb3/create/database/#schema-mode). ```bash influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME \ --tags rack,zone ``` ### Add field columns to a table Adds two field columns to a table. ```bash influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --fields temp:float64,active:bool \ TABLE_NAME ``` ### Add tag and field columns in one command ```bash influxdb3 update table \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --tags rack \ --fields temp:float64 \ TABLE_NAME ``` > [!Note] > Adding a column that already exists with the same data type is a no-op. > Adding a column that already exists with a different data type returns an error. ### Update a table retention period Updates a table retention period to 30 days. ```bash influxdb3 update table --database DATABASE_NAME --token AUTH_TOKEN --retention-period 30d TABLE_NAME ``` ### Clear a table retention period Removes the retention period from a table by setting it to `none`. ```bash influxdb3 update table --database DATABASE_NAME --retention-period none TABLE_NAME ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | The name of the database containing the table | | | --token | Authentication token | | -r | --retention-period | The retention period as a duration value (for example: 30d , 24h ) or none to clear | | | --tags | Comma-separated list of tag columns to add to the table. Takes one or more values, so place the table name before it or follow it with another option | | -f | --fields | Comma-separated list of field columns and their data types to add to the table | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_CA | --tls-ca | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 update trigger Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/update/trigger/ The `influxdb3 update trigger` command updates an existing trigger in your InfluxDB 3 Enterprise instance. Use this command to update trigger plugin code, configuration, or behavior without recreating the trigger. This preserves trigger history and configuration while allowing you to iterate on plugin development. > [!Important] > `influxdb3 update trigger` does not work with triggers that reference plugins using the `gh:` prefix. > To fetch updated code for a `gh:` trigger, [disable and re-enable the trigger](#update-a-gh-trigger). ## Usage ```bash influxdb3 update trigger [OPTIONS] \ --database \ --trigger-name ``` ## Arguments * **`DATABASE_NAME`**: (Required) The name of the database containing the trigger. * **`TRIGGER_NAME`**: (Required) The name of the trigger to update. ## Options |Option| | Description | |------|---------------------|-------------------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database containing the trigger | | | `--trigger-name` | *(Required)* Name of the trigger to update | | `-p` | `--path` |Local path to a plugin file or directory (single `.py` file or directory containing `__init__.py` for multifile plugins).| | |`--trigger-arguments`|Additional arguments for the trigger, in the format `key=value`, separated by commas (for example, `arg1=val1,arg2=val2`)| | | `--disabled` | Set the trigger state to disabled | | | `--enabled` | Set the trigger state to enabled | | | `--error-behavior` | Error handling behavior: `log`, `retry`, or `disable` | | | `--token` | Authentication token | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | `--tls-no-verify` | Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates) | | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables instead of providing CLI options directly: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | | `INFLUXDB3_TLS_CA` | `--tls-ca` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples The following examples show how to update triggers in different scenarios. * [Update trigger plugin code](#update-trigger-plugin-code) * [Upload and update with a local plugin](#upload-and-update-with-a-local-plugin) * [Update a gh: trigger](#update-a-gh-trigger) * [Update trigger arguments](#update-trigger-arguments) * [Enable or disable a trigger](#enable-or-disable-a-trigger) * [Update error handling behavior](#update-error-handling-behavior) Replace the following placeholders with your values: * `DATABASE_NAME`: Database name * `AUTH_TOKEN`: Authentication token * `TRIGGER_NAME`: Name of the trigger to update ### Update trigger plugin code Update a trigger with modified plugin code from a local file. The command transfers the file to the server and requires an admin token. ```bash influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --path "my_plugin.py" \ --token AUTH_TOKEN ``` ### Upload and update with a local plugin Upload new plugin code from your local machine and update the trigger in a single operation. Requires admin token. ```bash # Upload single-file plugin influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --path "/local/path/to/updated_plugin.py" \ --token AUTH_TOKEN # Upload multifile plugin directory influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --path "/local/path/to/plugin_directory" \ --token AUTH_TOKEN ``` The command transfers local files to the server’s plugin directory. ### Update a gh: trigger For a trigger that references a plugin with the `gh:` prefix, publish your changes at the configured plugin source, then disable and re-enable the trigger to fetch the updated code. Use the dedicated `disable trigger` and `enable trigger` commands. ```bash influxdb3 disable trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TRIGGER_NAME influxdb3 enable trigger \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TRIGGER_NAME ``` ### Update trigger arguments Modify the arguments passed to a trigger’s plugin code. ```bash influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --trigger-arguments threshold=100,window=5m \ --token AUTH_TOKEN ``` ### Enable or disable a trigger Change the trigger’s enabled state without modifying other configuration. ```bash # Disable a trigger influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --disabled \ --token AUTH_TOKEN # Enable a trigger influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --enabled \ --token AUTH_TOKEN ``` ### Update error handling behavior Change how the trigger responds to errors. ```bash # Log errors without retrying influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --error-behavior log \ --token AUTH_TOKEN # Retry on errors influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --error-behavior retry \ --token AUTH_TOKEN # Disable trigger on error influxdb3 update trigger \ --database DATABASE_NAME \ --trigger-name TRIGGER_NAME \ --error-behavior disable \ --token AUTH_TOKEN ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database containing the trigger | | | --trigger-name | ( Required ) Name of the trigger to update | | -p | --path | Local path to a plugin file or directory (single .py file or directory containing __init__.py for multifile plugins). | | | --trigger-arguments | Additional arguments for the trigger, in the format key=value , separated by commas (for example, arg1=val1,arg2=val2 ) | | | --disabled | Set the trigger state to disabled | | | --enabled | Set the trigger state to enabled | | | --error-behavior | Error handling behavior: log , retry , or disable | | | --token | Authentication token | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_CA | --tls-ca | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # influxdb3 write Source: https://docs.influxdata.com/influxdb3/enterprise/reference/cli/influxdb3/write/ The `influxdb3 write` command writes data to your InfluxDB 3 Enterprise server. ## Usage ```bash influxdb3 write [OPTIONS] --database [LINE_PROTOCOL]... ``` ##### Aliases `write`, `w` ## Arguments * **LINE\_PROTOCOL**: The line protocol to write to InfluxDB 3 Enterprise. Provide the line protocol in one of the following ways: * a string * a path to a file that contains the line protocol using the `--file` option * from stdin ## Options |Option| | Description | |------|------------------|-------------------------------------------------------------------------------------------------------------| | `-H` | `--host` | Host URL of the running InfluxDB 3 Enterprise server (default is `http://127.0.0.1:8181`) | | `-d` | `--database` | *(Required)* Name of the database to operate on | | | `--token` | *(Required)* Authentication token | | `-f` | `--file` | A file that contains line protocol to write | | |`--accept-partial`| Accept partial writes | | | `--no-sync` | Do not wait for WAL sync before acknowledging the write request | | | `--precision` | Precision of data timestamps (`ns`, `us`, `ms`, or `s`) | | | `--tls-ca` | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | |`--tls-no-verify` |Disable TLS certificate verification (**Not recommended in production**, useful for self-signed certificates)| | `-h` | `--help` | Print help information | | | `--help-all` | Print detailed help information | ### Option environment variables You can use the following environment variables to set command options: | Environment Variable | Option | |-------------------------|-----------------| | `INFLUXDB3_HOST_URL` | `--host` | |`INFLUXDB3_DATABASE_NAME`| `--database` | | `INFLUXDB3_AUTH_TOKEN` | `--token` | |`INFLUXDB3_TLS_NO_VERIFY`|`--tls-no-verify`| ## Examples * [Write line protocol to your InfluxDB 3 server](#write-line-protocol-to-your-influxdb-3-server) * [Write line protocol and accept partial writes](#write-line-protocol-and-accept-partial-writes) * [Write line protocol with specific timestamp precision](#write-line-protocol-with-specific-timestamp-precision) * [Write line protocol and immediately return a response](#write-line-protocol-and-immediately-return-a-response) In the examples below, replace the following: * `DATABASE_NAME`: the name of the database to query * `AUTH_TOKEN`: Authentication token ### Write line protocol to your InfluxDB 3 server [string](#string)[file](#file)[stdin](#stdin) ```bash influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000' ``` ```bash influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./data.lp ``` ```bash cat ./data.lp | influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN ``` ### Write line protocol and accept partial writes [string](#string-2)[file](#file-2)[stdin](#stdin-2) ```bash influxdb3 write \ --accept-partial \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000' ``` ```bash influxdb3 write \ --accept-partial \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./data.lp ``` ```bash cat ./data.lp | influxdb3 write \ --accept-partial \ --database DATABASE_NAME \ --token AUTH_TOKEN ``` ### Write line protocol with specific timestamp precision By default, in CLI and HTTP API write requests, InfluxDB 3 Enterprise uses the timestamp magnitude to auto-detect the precision. To avoid any ambiguity, specify the `--precision {ns|us|ms|s}` option: ```bash influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --precision s \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 ' ``` ### Write line protocol and immediately return a response By default, InfluxDB 3 Enterprise waits to respond to write requests until the written data is flushed from the Write-Ahead Log (WAL) to object storage (every 1s by default). Use the `--no-sync` option to immediately return a response without waiting for the WAL to flush. This improves perceived write response times, but may hide certain types of write errors–for example: malformed line protocol or type conflicts. > [!Tip] > Only use `--no-sync` when low write latency is more important than guaranteed data durability. > Avoid using this option for critical or irreplaceable data, as it increases the risk of silent data loss. ```bash influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --no-sync \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000' ``` | Option | | Description | | --- | --- | --- | | Option | | Description | | -H | --host | Host URL of the running InfluxDB 3 Enterprise server (default is http://127.0.0.1:8181 ) | | -d | --database | ( Required ) Name of the database to operate on | | | --token | ( Required ) Authentication token | | -f | --file | A file that contains line protocol to write | | | --accept-partial | Accept partial writes | | | --no-sync | Do not wait for WAL sync before acknowledging the write request | | | --precision | Precision of data timestamps ( ns , us , ms , or s ) | | | --tls-ca | Path to a custom TLS certificate authority (for testing or self-signed certificates) | | | --tls-no-verify | Disable TLS certificate verification ( Not recommended in production , useful for self-signed certificates) | | -h | --help | Print help information | | | --help-all | Print detailed help information | | Environment Variable | Option | | --- | --- | | Environment Variable | Option | | INFLUXDB3_HOST_URL | --host | | INFLUXDB3_DATABASE_NAME | --database | | INFLUXDB3_AUTH_TOKEN | --token | | INFLUXDB3_TLS_NO_VERIFY | --tls-no-verify | --- # InfluxDB API client libraries Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/ InfluxDB client libraries are language-specific packages that integrate with InfluxDB APIs. ### [Apache Arrow Flight RPC clients](/influxdb3/enterprise/reference/client-libraries/flight/) Flight clients are language-specific drivers that can interact with Flight servers using the Arrow in-memory format and the Flight RPC framework. View the list of available clients. ### [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) InfluxDB 3 client libraries use InfluxDB HTTP APIs to write data and use [Flight clients](/influxdb3/enterprise/reference/client-libraries/flight-sql) to execute SQL and InfluxQL queries. View the list of available client libraries. ### [InfluxDB v2 API client libraries](/influxdb3/enterprise/reference/client-libraries/v2/) InfluxDB v2 client libraries use InfluxDB `/api/v2` endpoints and work with [InfluxDB 2.x API compatibility endpoints](/influxdb/v1/api/buckets-v2-compatible/). View the list of available client libraries. ### [InfluxDB v1 client libraries](/influxdb3/enterprise/reference/client-libraries/v1/) InfluxDB v1 client libraries use the InfluxDB 1.7 API and should be fully compatible with InfluxDB 1.5+. View the list of available client libraries. [client libraries](/influxdb3/enterprise/tags/client-libraries/)[API](/influxdb3/enterprise/tags/api/)[developer tools](/influxdb3/enterprise/tags/developer-tools/) --- # Apache Arrow Flight RPC clients Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/flight/ Flight RPC and Flight SQL clients are language-specific drivers that interact with databases using the Arrow in-memory format and the Flight RPC protocol. Apache Arrow Flight RPC and Flight SQL protocols define APIs for servers and clients. > [!Note] > #### Use InfluxDB 3 client libraries > > We recommend using [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) for integrating InfluxDB 3 with your application code. > Client libraries wrap Apache Arrow Flight clients > and provide convenient methods for [writing](/influxdb3/enterprise/write-data/client-libraries/), [querying](/influxdb3/enterprise/query-data/execute-queries/client-libraries), and processing data stored in InfluxDB 3 Enterprise. **Flight RPC clients** can use SQL or InfluxQL to query data stored in an InfluxDB 3 Enterprise database. Using InfluxDB 3’s IOx-specific Flight RPC protocol, clients send a single `DoGet()` request to authenticate, query, and retrieve data. **Flight SQL clients** use the [Flight SQL protocol](https://arrow.apache.org/docs/format/FlightSql.html) for querying an SQL database server. They can use SQL to query data stored in an InfluxDB 3 Enterprise database, but they can’t use InfuxQL. > [!Important] > #### Flight SQL requires HTTP/2 > > Flight SQL uses gRPC, which requires **HTTP/2**. > If you connect to InfluxDB 3 Enterprise through a proxy (such as HAProxy, nginx, or a load balancer), > verify that your proxy is configured to support HTTP/2. > Without HTTP/2 support, Flight SQL connections will fail. Clients are maintained by Apache Arrow projects or third-parties. For specifics about a Flight client, see the client’s GitHub repository. ### [C# .NET Flight client](/influxdb3/enterprise/reference/client-libraries/flight/csharp-flight/) The C# .NET Flight client integrates with C# .NET scripts and applications to query data stored in InfluxDB. ### [Go Flight client](/influxdb3/enterprise/reference/client-libraries/flight/go-flight/) The Go Flight client integrates with Go scripts and applications to query data stored in InfluxDB. ### [Java Flight SQL package](/influxdb3/enterprise/reference/client-libraries/flight/java-flightsql/) The Java Flight SQL client integrates with Java applications to query and retrieve data from Flight database servers using RPC and SQL. ```java public class Query { public static void main(String[] args) { String query = "SELECT * FROM home"; Location location = Location.forGrpcTls(HOST, 443); CredentialCallOption auth = new CredentialCallOption(new BearerCredentialWriter(TOKEN)); BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); FlightClientMiddleware.Factory f = info -> new FlightClientMiddleware() { @Override public void onBeforeSendingHeaders(CallHeaders outgoingHeaders) { outgoingHeaders.insert(DATABASE_FIELD, DATABASE_NAME); } }; FlightClient client = FlightClient.builder(allocator, location) .intercept(f) .build(); FlightSqlClient sqlClient = new FlightSqlClient(client); FlightInfo flightInfo = sqlClient.execute(query, auth); } } ``` ### [Python Flight client](/influxdb3/enterprise/reference/client-libraries/flight/python-flight/) The Python Flight client integrates with Python scripts and applications to query data stored in InfluxDB. ``` from pyarrow.flight import FlightClient, Ticket, FlightCallOptions import json import pandas import tabulate # Downsampling query groups data into 2-hour bins sql=""" SELECT DATE_BIN(INTERVAL '2 hours', time, '1970-01-01T00:00:00Z') AS time, room, selector_max(temp, time)['value'] AS 'max temp', selector_min(temp, time)['value'] AS 'min temp', avg(temp) AS 'average temp' FROM home GROUP BY 1, room ORDER BY room, 1""" flight_ticket = Ticket(json.dumps({ "namespace_name": "DATABASE_NAME", "sql_query": sql, "query_type": "sql" })) token = (b"authorization", bytes(f"Bearer DATABASE_TOKEN".encode('utf-8'))) options = FlightCallOptions(headers=[token]) client = FlightClient(f"grpc+tls://localhost:8181:443") reader = client.do_get(flight_ticket, options) arrow_table = reader.read_all() ``` ### [Python Flight SQL DBAPI client](/influxdb3/enterprise/reference/client-libraries/flight/python-flightsql-dbapi/) The Python `flightsql-dbapi` library uses SQL and the Flight SQL protocol to query data stored in an InfluxDB Core database. [client libraries](/influxdb3/enterprise/tags/client-libraries/)[Flight RPC](/influxdb3/enterprise/tags/flight-rpc/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/) --- # C# .NET Flight client Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/flight/csharp-flight/ [Apache Arrow for C# .NET](https://github.com/apache/arrow/blob/main/csharp/README.md) integrates with C# .NET scripts and applications to query data stored in InfluxDB. For more information, see the [C# client example on GitHub](https://github.com/apache/arrow/tree/main/csharp/examples/FlightClientExample). > [!Note] > #### Use InfluxDB 3 client libraries > > We recommend using the [`influxdb3-csharp` C# client library](/influxdb3/enterprise/reference/client-libraries/v3/csharp/) for integrating InfluxDB 3 with your C# application code. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) wrap Apache Arrow Flight clients > and provide convenient methods for [writing](/influxdb3/enterprise/write-data/api-client-libraries/), [querying](/influxdb3/enterprise/query-data/execute-queries/), and processing data stored in InfluxDB 3 Enterprise. > Client libraries can query using SQL or InfluxQL. [gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # Go Flight client Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/flight/go-flight/ [Apache Arrow for Go](https://pkg.go.dev/github.com/apache/arrow/go/v12) integrates with Go scripts and applications to query data stored in InfluxDB. > [!Note] > #### Use InfluxDB 3 client libraries > > We recommend using the [`influxdb3-go` Go client library](/influxdb3/enterprise/reference/client-libraries/v3/go/) for integrating InfluxDB 3 with your Go application code. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) wrap Apache Arrow Flight clients > and provide convenient methods for [writing](/influxdb3/enterprise/write-data/api-client-libraries/), [querying](/influxdb3/enterprise/query-data/execute-queries/), and processing data stored in InfluxDB 3 Enterprise. > Client libraries can query using SQL or InfluxQL. ## Flight SQL client ### Example query using Flight SQL The following example shows how to use the Arrow Flight SQL client for Go to query an InfluxDB 3 Enterprise database: 1. In your editor, open a new file named `query.go` and enter the following sample code: ``` package main import ( "context" "crypto/x509" "encoding/json" "fmt" "os" "github.com/apache/arrow/go/v14/arrow/flight/flightsql" "google.golang.org/grpc" "google.golang.org/grpc/credentials" "google.golang.org/grpc/metadata" ) func dbQuery(ctx context.Context) error { url := "localhost:8181:443" // INFLUX_TOKEN is an environment variable you created for your database READ token token := os.Getenv("INFLUX_TOKEN") database := "get-started" // Create a gRPC transport pool, err := x509.SystemCertPool() if err != nil { return fmt.Errorf("x509: %s", err) } transport := grpc.WithTransportCredentials(credentials.NewClientTLSFromCert(pool, "")) opts := []grpc.DialOption{ transport, } // Create query client client, err := flightsql.NewClient(url, nil, nil, opts...) if err != nil { return fmt.Errorf("flightsql: %s", err) } ctx = metadata.AppendToOutgoingContext(ctx, "authorization", "Bearer "+token) ctx = metadata.AppendToOutgoingContext(ctx, "database", database) // Execute query query := `SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z'` info, err := client.Execute(ctx, query) if err != nil { return fmt.Errorf("flightsql flight info: %s", err) } reader, err := client.DoGet(ctx, info.Endpoint[0].Ticket) if err != nil { return fmt.Errorf("flightsql do get: %s", err) } // Print results as JSON for reader.Next() { record := reader.Record() b, err := json.MarshalIndent(record, "", " ") if err != nil { return err } fmt.Println("RECORD BATCH") fmt.Println(string(b)) if err := reader.Err(); err != nil { return fmt.Errorf("flightsql reader: %s", err) } } return nil } func main() { if err := dbQuery(context.Background()); err != nil { fmt.Fprintf(os.Stderr, "error: %v\n", err) os.Exit(1) } } ``` The sample does the following: 1. Imports the following packages: * `context` * `crypto/x509` * `encoding/json` * `fmt` * `os` * `github.com/apache/arrow/go/v14/arrow/flight/flightsql` * `google.golang.org/grpc` * `google.golang.org/grpc/credentials` * `google.golang.org/grpc/metadata` 2. Creates a `dbQuery` function that does the following: 1. Defines variables for InfluxDB credentials. * **`url`**: InfluxDB 3 Enterprise cluster hostname and port (`:443`) *(no protocol)* * **`database`**: the name of the InfluxDB 3 Enterprise database to query * **`token`**: a [database token](/influxdb3/enterprise/admin/tokens/database/) with read permission on the specified database.*For security reasons, we recommend setting this as an environment variable rather than including the raw token string.* 2. Defines an `opts` options list that includes a gRPC transport for communicating with InfluxDB over the *gRPC+TLS* protocol. 3. Calls the `flightsql.NewClient()` method with `url` and `opts` to create a new Flight SQL client. 4. Appends the following InfluxDB credentials as key-value pairs to the outgoing context: * **`authorization`**: Bearer \ * **`database`**: Database name 5. Defines the SQL query to execute. 6. Calls the `client.execute()` method to send the query request. 7. Calls the `client.doGet()` method with the *ticket* from the query response to retrieve result data from the endpoint. 8. Creates a reader to read the Arrow table returned by the endpoint and print the results as JSON. 3. Creates a `main` module function that executes the `dbQuery` function. 2. Enter the following commands to install all the necessary packages and run the program to query InfluxDB 3 Enterprise: ``` go get ./... go run ./query.go ``` [](#view-program-output) View program output ```text RECORD BATCH [ { "co": 0, "hum": 35.9, "room": "Kitchen", "temp": 21, "time": "2022-01-01 08:00:00" }, { "co": 0, "hum": 36.2, "room": "Kitchen", "temp": 23, "time": "2022-01-01 09:00:00" }, { "co": 0, "hum": 36.1, "room": "Kitchen", "temp": 22.7, "time": "2022-01-01 10:00:00" }, { "co": 0, "hum": 36, "room": "Kitchen", "temp": 22.4, "time": "2022-01-01 11:00:00" }, { "co": 0, "hum": 36, "room": "Kitchen", "temp": 22.5, "time": "2022-01-01 12:00:00" }, ... ] ``` For more information, see the [Go Arrow Flight Client documentation](https://pkg.go.dev/github.com/apache/arrow/go/v14/arrow/flight#Client). #### Related * [Go client library for InfluxDB 3](/influxdb3/enterprise/reference/client-libraries/v3/go/) [Flight client](/influxdb3/enterprise/tags/flight-client/)[Go](/influxdb3/enterprise/tags/go/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # Java Flight SQL package Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/flight/java-flightsql/ [Apache Arrow Flight SQL for Java](https://arrow.apache.org/docs/java/reference/org/apache/arrow/flight/sql/package-summary.html) integrates with Java applications to query and retrieve data from Flight database servers using RPC and SQL. > [!Note] > #### Use InfluxDB 3 client libraries > > We recommend using the [`influxdb3-java` Java client library](/influxdb3/enterprise/reference/client-libraries/v3/java/) for integrating InfluxDB 3 with your Java application code. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) wrap Apache Arrow Flight clients > and provide convenient methods for [writing](/influxdb3/enterprise/write-data/api-client-libraries/), [querying](/influxdb3/enterprise/query-data/execute-queries/), and processing data stored in InfluxDB 3 Enterprise. > Client libraries can query using SQL or InfluxQL. * [Get started using the Java Flight SQL client to query InfluxDB](#get-started-using-the-java-flight-sql-client-to-query-influxdb) * [Set up InfluxDB](#set-up-influxdb) * [Install prerequisites](#install-prerequisites) * [Create the FlightQuery class](#create-the-flightquery-class) * [Create a query client](#create-a-query-client) * [Execute a query](#execute-a-query) * [Retrieve and process Arrow data](#retrieve-and-process-arrow-data) * [Run the application](#run-the-application) * [Troubleshoot Arrow Flight requests](#troubleshoot-arrow-flight-requests) ## Get started using the Java Flight SQL client to query InfluxDB Write a Java class for a Flight SQL client that connects to InfluxDB 3 Enterprise, executes an SQL query, and retrieves data stored in an InfluxDB 3 Enterprise database. The example uses the [Apache Arrow Java implementation (`org.apache.arrow`)](https://arrow.apache.org/docs/java/index.html) for interacting with Flight database servers like InfluxDB 3. * **`org.apache.arrow`**: Provides classes and methods for integrating Java applications with Apache Arrow data and protocols. * **`org.apache.arrow.flight.sql`**: Provides classes and methods for interacting with Flight database servers using Arrow Flight RPC and Flight SQL. 1. [Set up InfluxDB](#set-up-influxdb) 2. [Install prerequisites](#install-prerequisites) 3. [Create the FlightQuery class](#create-the-flightquery-class) 4. [Create a query client](#create-a-query-client) 5. [Execute a query](#execute-a-query) 6. [Retrieve and process Arrow data](#retrieve-and-process-arrow-data) To clone or download the example application that you can run with Docker, see the [InfluxCommunity/ArrowFlightClient\_Query\_Examples repository](https://github.com/InfluxCommunity/ArrowFlightClient_Query_Examples) on GitHub. ### Set up InfluxDB To configure the application for querying InfluxDB 3 Enterprise, you’ll need the following InfluxDB resources: * InfluxDB 3 Enterprise **database** * InfluxDB 3 Enterprise **database token** with *read* permission to the database If you don’t already have a database token and a database, see how to [set up InfluxDB](/influxdb3/enterprise/get-started/setup/). If you don’t already have data to query, see how to[write data](/influxdb3/enterprise/get-started/write/) to a database. ### Install prerequisites The following uses Docker and Maven to build and run the Java application and avoid platform-specific dependency problems. The example `Dockerfile` installs compatible versions of Maven and Java JDK in the Docker container, and then runs the Maven commands to download dependencies and compile the application. Follow the instructions to download and install Docker for your system: * **macOS**: [Install Docker for macOS](https://docs.docker.com/desktop/install/mac-install/) * **Linux**: [Install Docker for Linux](https://docs.docker.com/desktop/install/linux-install/) [](#view-the-dockerfile) View the Dockerfile ```dockerfile # Use the official Maven image as the base image FROM maven:3.8.3-openjdk-11 AS build # Set the working directory WORKDIR /app # Copy the pom.xml file into the container COPY pom.xml . # Download and cache dependencies RUN mvn dependency:go-offline # Copy the rest of the source code into the container COPY src/ ./src/ # Compile the source code and copy dependencies RUN mvn compile dependency:copy-dependencies # Use the official OpenJDK image as the runtime base image FROM openjdk:11-jre-slim # Set the working directory WORKDIR /app # Copy the compiled classes and dependencies from the build stage COPY --from=build /app/target/classes ./classes COPY --from=build /app/target/dependency ./dependency # Set ARGs for --build-arg options passed in the build command ARG DATABASE_FIELD ARG DATABASE_NAME ARG HOST ARG TOKEN # Set run-time ENVs from ARGs ENV DATABASE_FIELD=${DATABASE_FIELD} ENV DATABASE_NAME=${DATABASE_NAME} ENV HOST=${HOST} ENV TOKEN=${TOKEN} # Set the entrypoint to run your Java application ENTRYPOINT ["java", "-cp", "classes:dependency/*", "com.influxdb.examples.FlightExamples"] ``` [](#view-the-maven-pomxml) View the Maven pom.xml ```xml 4.0.0 com.influxdb examples 1.0-SNAPSHOT org.apache.maven.plugins maven-jar-plugin 3.2.0 true lib/ com.influxdb.examples.FlightExamples org.apache.maven.plugins maven-help-plugin 3.2.0 org.apache.maven.plugins maven-shade-plugin 3.4.1 shade true false *:* META-INF/*.SF META-INF/*.DSA META-INF/*.RSA 1.8 1.8 org.apache.arrow flight-sql 11.0.0 io.netty netty-all 4.1.74.Final org.slf4j slf4j-simple 1.7.30 ``` ### Create the FlightQuery class [](#view-flightqueryjava) View FlightQuery.java ``` package com.influxdb.examples; import org.apache.arrow.flight.auth2.BearerCredentialWriter; import org.apache.arrow.flight.CallHeaders; import org.apache.arrow.flight.CallStatus; import org.apache.arrow.flight.grpc.CredentialCallOption; import org.apache.arrow.flight.Location; import org.apache.arrow.flight.FlightClient; import org.apache.arrow.flight.FlightClientMiddleware; import org.apache.arrow.flight.FlightInfo; import org.apache.arrow.flight.FlightStream; import org.apache.arrow.flight.sql.FlightSqlClient; import org.apache.arrow.flight.Ticket; import org.apache.arrow.memory.BufferAllocator; import org.apache.arrow.memory.RootAllocator; import org.apache.arrow.vector.VectorSchemaRoot; public class FlightQuery { /* Get server credentials from environment variables */ public static final String DATABASE_NAME = System.getenv("DATABASE_NAME"); public static final String HOST = System.getenv("HOST"); public static final String TOKEN = System.getenv("TOKEN"); public static void main() { System.out.println("Query InfluxDB with the Java Flight SQL Client"); // Create an interceptor that injects header metadata (database name) in every request. FlightClientMiddleware.Factory f = info -> new FlightClientMiddleware() { @Override public void onBeforeSendingHeaders(CallHeaders outgoingHeaders) { outgoingHeaders.insert("database", DATABASE_NAME); } @Override public void onHeadersReceived(CallHeaders incomingHeaders) { } @Override public void onCallCompleted(CallStatus status) { } }; // Create a gRPC+TLS channel URI with HOST and port 443. Location location = Location.forGrpcTls(HOST, 443); // Set the allowed memory. BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE); // Create a client with the allocator and gRPC channel. FlightClient client = FlightClient.builder(allocator, location) .intercept(f) .build(); System.out.println("client" + client); FlightSqlClient sqlClient = new FlightSqlClient(client); System.out.println("sqlClient: " + sqlClient); // Define the SQL query to execute. String query = "SELECT * FROM home"; /* Construct a bearer credential using TOKEN. Construct a credentials option using the bearer credential. */ CredentialCallOption auth = new CredentialCallOption(new BearerCredentialWriter(TOKEN)); /* Execute the query. If successful, execute returns a FlightInfo object that contains metadata and an endpoints list. Each endpoint contains the following: - A list of addresses where you can retrieve the data. - A `ticket` value that identifies the data to retrieve. */ FlightInfo flightInfo = sqlClient.execute(query, auth); // Extract the Flight ticket from the response. Ticket ticket = flightInfo.getEndpoints().get(0).getTicket(); // Pass the ticket to request the Arrow stream data from the endpoint. final FlightStream stream = sqlClient.getStream(ticket, auth); // Process all the Arrow stream data. while (stream.next()) { try { // Get the current vector data from the stream. final VectorSchemaRoot root = stream.getRoot(); System.out.println(root.contentToTSVString()); } catch (Exception e) { // Handle exceptions. System.out.println("Error executing FlightSqlClient: " + e.getMessage()); } } try { // Close the stream and release resources. stream.close(); } catch (Exception e) { // Handle exceptions. System.out.println("Error closing stream: " + e.getMessage()); } try { // Close the client sqlClient.close(); } catch (Exception e) { // Handle exceptions. System.out.println("Error closing client: " + e.getMessage()); } } } ``` 1. In your `/src/main/java` directory, create the `com/influxdb/examples` subdirectories for the `com.influxdb.examples` package. 2. In the `examples` directory from the preceding step, create the `FlightQuery.java` class file. You should have the following directory structure: ``` PROJECT_ROOT └──src └──main └──java └──com └──influxdb └──examples └──FlightQuery.java ``` 3. In `FlightQuery.java`: 1. Add the package name: ``` package com.influxdb.examples; ``` 2. Add `import` statements for the following packages. You’ll use classes and methods from these packages in the remaining steps: * `org.apache.arrow.flight.auth2.BearerCredentialWriter` * `org.apache.arrow.flight.CallHeaders` * `org.apache.arrow.flight.CallStatus` * `org.apache.arrow.flight.grpc.CredentialCallOption` * `org.apache.arrow.flight.Location` * `org.apache.arrow.flight.FlightClient` * `org.apache.arrow.flight.FlightClientMiddleware` * `org.apache.arrow.flight.FlightInfo` * `org.apache.arrow.flight.FlightStream` * `org.apache.arrow.flight.sql.FlightSqlClient` * `org.apache.arrow.flight.Ticket` * `org.apache.arrow.memory.BufferAllocator` * `org.apache.arrow.memory.RootAllocator` * `org.apache.arrow.vector.VectorSchemaRoot` 3. Create a `FlightQuery` class. 4. In the `FlightQuery` class: 1. Define constants for server credentials. * `DATABASE_NAME` * `HOST` * `TOKEN` *The example `Dockerfile` defines environment variables for these credentials.* 2. Create a `main()` method. ### Create a query client In the `FlightQuery.main()` method, do the following to create an SQL client that can connect to `HOST` and `DATABASE_NAME`: 1. Construct a *gRPC+TLS* channel URI with `HOST` and port `443` for communicating with a [gRPC server over TLS](https://grpc.io/docs/guides/auth/#with-server-authentication-ssltls-4). 2. Instantiate `FlightClientMiddleware` and define an event callback that inserts the following Flight request metadata header property: ``` { "database": "DATABASE_NAME" } ``` 3. Instantiate a `BufferAllocator` that sets the memory allowed for the client. 4. Create a `FlightClient` with the allocator and gRPC channel. 5. Instantiate a `FlightSqlClient` that wraps the `FlightClient` instance. ### Execute a query In the `FlightQuery.main` method: 1. Instantiate a `CredentialCallOption` with `TOKEN` as a *bearer* credential. The result is a credential object that you’ll pass in each request to the server. 2. Define a string that contains the SQL query to execute–for example: ``` String query = "SELECT * FROM home"; ``` 3. Call the `FlightSqlClient.execute` method with the SQL query and the `CredentialCallOption`. 4. If successful, the `FlightSqlClient.execute` method responds with a `FlightInfo` object that contains metadata and an `endpoints: [...]` list. Each endpoint contains the following: * A list of addresses where you can retrieve the data. * A `ticket` value that identifies the data to retrieve. 5. Extract the ticket from the response. ### Retrieve and process Arrow data In the `FlightQuery.main()` method, do the following to retrieve the data stream described in the `FlightInfo` response: 1. Call the `FlightSqlClient.getStream` method with the *ticket* and the `CredentialCallOption` to fetch the [Arrow stream](https://arrow.apache.org/docs/format/CStreamInterface.html). 2. Call the `FlightStream.getRoot` method to get the current vector data from the stream. 3. Process the data and handle exceptions. The example converts the vector data into tab-separated values and prints the result to `System.out`. For more examples using Java to work with Arrow data, see the [Apache Arrow Java Cookbook](https://arrow.apache.org/cookbook/java/). 4. Finally, close the stream and client. ### Run the application Follow these steps to build and run the application using Docker: 1. Copy the `Dockerfile` and `pom.xml` to your project root directory. 2. Open a terminal in your project root directory. 3. In your terminal, run the `docker build` command and pass `--build-arg` flags for the server credentials: * **`DATABASE_NAME`**: your [InfluxDB 3 Enterprise database](/influxdb3/enterprise/admin/databases/) * **`HOST`**: your InfluxDB 3 Enterprise hostname (URL without the “https://”) * **`TOKEN`**: your [InfluxDB 3 Enterprise database token](/influxdb3/enterprise/get-started/setup/) with *read* permission to the database ``` docker build \ --build-arg DATABASE_NAME=INFLUX_DATABASE \ --build-arg HOST=localhost:8181\ --build-arg TOKEN=INFLUX_TOKEN \ -t javaflight . ``` The command builds a Docker image named `javaflight`. 4. To run the application in a new Docker container, enter the following command: ``` docker run javaflight ``` The output is the query data in TSV-format. ## Troubleshoot Arrow Flight requests For the list of Arrow Flight error response codes, see the [Arrow Flight RPC documentation](https://arrow.apache.org/docs/format/Flight.html#error-handling). #### Related * [Java client library for InfluxDB 3](/influxdb3/enterprise/reference/client-libraries/v3/java/) [Flight client](/influxdb3/enterprise/tags/flight-client/)[Java](/influxdb3/enterprise/tags/java/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/) --- # Python Flight client Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/flight/python-flight/ [Apache Arrow Python bindings](https://arrow.apache.org/docs/python/index.html) integrate with Python scripts and applications to query data stored in InfluxDB. > [!Note] > #### Use InfluxDB 3 client libraries > > We recommend using the [`influxdb3-python` Python client library](/influxdb3/enterprise/reference/client-libraries/v3/python/) for integrating InfluxDB 3 with your Python application code. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) wrap Apache Arrow Flight clients > and provide convenient methods for [writing](/influxdb3/enterprise/write-data/api-client-libraries/), [querying](/influxdb3/enterprise/query-data/execute-queries/), and processing data stored in InfluxDB 3 Enterprise. > Client libraries can query using SQL or InfluxQL. The following examples show how to use the `pyarrow.flight` and `pandas` Python modules to query and format data stored in an InfluxDB 3 Enterprise database: [SQL](#sql)[InfluxQL](#influxql) ``` # Using pyarrow>=12.0.0 FlightClient from pyarrow.flight import FlightClient, Ticket, FlightCallOptions import json import pandas import tabulate # Downsampling query groups data into 2-hour bins sql=""" SELECT DATE_BIN(INTERVAL '2 hours', time) AS time, room, selector_max(temp, time)['value'] AS 'max temp', selector_min(temp, time)['value'] AS 'min temp', avg(temp) AS 'average temp' FROM home GROUP BY 1, room ORDER BY room, 1""" flight_ticket = Ticket(json.dumps({ "namespace_name": "DATABASE_NAME", "sql_query": sql, "query_type": "sql" })) token = (b"authorization", bytes(f"Bearer DATABASE_TOKEN".encode('utf-8'))) options = FlightCallOptions(headers=[token]) client = FlightClient(f"grpc+tls://localhost:8181:443") reader = client.do_get(flight_ticket, options) arrow_table = reader.read_all() # Use pyarrow and pandas to view and analyze data data_frame = arrow_table.to_pandas() print(data_frame.to_markdown()) ``` ``` # Using pyarrow>=12.0.0 FlightClient from pyarrow.flight import FlightClient, Ticket, FlightCallOptions import json import pandas import tabulate # Downsampling query groups data into 2-hour bins influxql=""" SELECT FIRST(temp) FROM home WHERE room = 'kitchen' AND time >= now() - 100d AND time <= now() - 10d GROUP BY time(2h)""" flight_ticket = Ticket(json.dumps({ "namespace_name": "DATABASE_NAME", "sql_query": influxql, "query_type": "influxql" })) token = (b"authorization", bytes(f"Bearer DATABASE_TOKEN".encode('utf-8'))) options = FlightCallOptions(headers=[token]) client = FlightClient(f"grpc+tls://localhost:8181:443") reader = client.do_get(flight_ticket, options) arrow_table = reader.read_all() # Use pyarrow and pandas to view and analyze data data_frame = arrow_table.to_pandas() print(data_frame.to_markdown()) ``` Replace the following: * `DATABASE_NAME`: your InfluxDB 3 Enterprise database * `DATABASE_TOKEN`: a [database token](/influxdb3/enterprise/admin/tokens/database/)with sufficient permissions to the specified database [Flight client](/influxdb3/enterprise/tags/flight-client/)[Python](/influxdb3/enterprise/tags/python/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # Python Flight SQL DBAPI client Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/flight/python-flightsql-dbapi/ The [Python `flightsql-dbapi` Flight SQL DBAPI library](https://github.com/influxdata/flightsql-dbapi) integrates with Python applications using SQL to query data stored in an InfluxDB 3 Enterprise database. The `flightsql-dbapi` library uses the [Flight SQL protocol](https://arrow.apache.org/docs/format/FlightSql.html) to query and retrieve data. > [!Note] > #### Use InfluxDB 3 client libraries > > We recommend using the [`influxdb3-python` Python client library](/influxdb3/enterprise/reference/client-libraries/v3/python/) for integrating InfluxDB 3 with your Python application code. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) wrap Apache Arrow Flight clients > and provide convenient methods for [writing](/influxdb3/enterprise/write-data/api-client-libraries/), [querying](/influxdb3/enterprise/query-data/execute-queries/), and processing data stored in InfluxDB 3 Enterprise. > Client libraries can query using SQL or InfluxQL. ## Installation The [`flightsql-dbapi`](https://github.com/influxdata/flightsql-dbapi) Flight SQL library for Python provides a[DB API 2](https://peps.python.org/pep-0249/) interface and[SQLAlchemy](https://www.sqlalchemy.org/) dialect for[Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html). Installing `flightsql-dbapi` also installs the [`pyarrow`](https://arrow.apache.org/docs/python/index.html) library that you’ll use for working with Arrow data. In your terminal, use `pip` to install `flightsql-dbapi`: ```sh pip install flightsql-dbapi ``` ## Importing the module The `flightsql-dbapi` package provides the `flightsql` module. From the module, import the `FlightSQLClient` class method: ```py from flightsql import FlightSQLClient ``` * `flightsql.FlightSQLClient` class: an interface for [initializing a client](#initialize-a-client) and interacting with a Flight SQL server. ## API reference * [Class FlightSQLClient](#class-flightsqlclient) * [Syntax](#syntax) * [Initialize a client](#initialize-a-client) * [Instance methods](#instance-methods) * [FlightSQLClient.execute](#flightsqlclientexecute) * [Syntax](#execute-query-syntax) * [Example](#execute-query-example) * [FlightSQLClient.do\_get](#flightsqlclientdo_get) * [Syntax](#retrieve-data-syntax) * [Example](#retrieve-data-example) ## Class FlightSQLClient Provides an interface for [initializing a client](#initialize-a-client) and interacting with a Flight SQL server. ### Syntax ```py __init__(self, host=None, token=None, metadata=None, features=None) ``` Initializes and returns a `FlightSQLClient` instance for interacting with the server. ## Initialize a client The following example shows how to use Python with `flightsql-dbapi`and the *DB API 2* interface to instantiate a Flight SQL client configured for an InfluxDB database. ```py from flightsql import FlightSQLClient # Instantiate a FlightSQLClient configured for a database client = FlightSQLClient(host='localhost:8181', token='DATABASE_TOKEN', metadata={'database': 'DATABASE_NAME'}, features={'metadata-reflection': 'true'}) ``` Replace the following: * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with read permissions on the databases you want to query * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) ### Instance methods ### FlightSQLClient.execute Sends a Flight SQL RPC request to execute the specified SQL Query. #### Syntax ```py execute(query: str, call_options: Optional[FlightSQLCallOptions] = None) ``` #### Example ```py # Execute the query info = client.execute("SELECT * FROM home") ``` The response contains a `flight.FlightInfo` object that contains metadata and an `endpoints: [...]` list. Each endpoint contains the following: * A list of addresses where you can retrieve query result data. * A `ticket` value that identifies the data to [retrieve](#retrieve-data-example). ### FlightSQLClient.do\_get Passes a Flight ticket (obtained from a `FlightSQLClient.execute` response) and retrieves Arrow data identified by the ticket. Returns a `pyarrow.flight.FlightStreamReader` for streaming the data. #### Syntax ``` do_get(ticket, call_options: Optional[FlightSQLCallOptions] = None) ``` #### Example The following sample shows how to use Python with `flightsql-dbapi` and `pyarrow` to query InfluxDB and retrieve data. ```py from flightsql import FlightSQLClient # Instantiate a FlightSQLClient configured for a database client = FlightSQLClient(host='localhost:8181', token='DATABASE_TOKEN', metadata={'database': 'DATABASE_NAME'}, features={'metadata-reflection': 'true'}) # Execute the query to retrieve FlightInfo info = client.execute("SELECT * FROM home") # Extract the token for retrieving data ticket = info.endpoints[0].ticket # Use the ticket to request the Arrow data stream. # Return a FlightStreamReader for streaming the results. reader = client.do_get(ticket) # Read all data to a pyarrow.Table table = reader.read_all() print(table) ``` `do_get(ticket)` returns a [`pyarrow.flight.FlightStreamReader`](https://arrow.apache.org/docs/python/generated/pyarrow.flight.FlightStreamReader.html) for streaming Arrow [record batches](https://arrow.apache.org/docs/python/data.html#record-batches). To read data from the stream, call one of the following `FlightStreamReader` methods: * `read_all()`: Read all record batches as a [`pyarrow.Table`](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). * `read_chunk()`: Read the next RecordBatch and metadata. * `read_pandas()`: Read all record batches and convert them to a [`pandas.DataFrame`](https://pandas.pydata.org/docs/reference/frame.html). [Flight client](/influxdb3/enterprise/tags/flight-client/)[Python](/influxdb3/enterprise/tags/python/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/) --- # InfluxDB v1 client libraries Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v1/ InfluxDB client libraries are language-specific tools that integrate with InfluxDB APIs. Client libraries for InfluxDB v1 work with the InfluxDB 1.7 API and should be fully compatible with InfluxDB 1.5+. > [!Note] > Upgrade to InfluxDB 3 to use new client libraries compatible with InfluxDB write APIs, SQL, and InfluxQL. For more information, see [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/). Functionality varies among client libraries. For specifics about a client library, see the library’s GitHub repository. ## C++ * [influxdb-cxx](https://github.com/offa/influxdb-cxx) ## Elixir * [Instream (instream)](https://github.com/mneudert/instream) ## Erlang * [Erlang InfluxDB UDP Writer](https://github.com/palkan/influx_udp) * [InfluxDB line encoder](https://github.com/Pouriya-Jahanbakhsh/influxdb_encoderl) ## Go * [InfluxDB Client](https://github.com/influxdata/influxdb1-client) ## Haskell * [influxdb-haskell](https://github.com/maoe/influxdb-haskell) ## Java * [influxdb-java](https://github.com/influxdb/influxdb-java) * [Alpakka InfluxDB](https://doc.akka.io/docs/alpakka/current/influxdb.html) * [JFlux](https://github.com/nickRm/jflux) ## JavaScript/Node.js * [node-influx](https://github.com/node-influx/node-influx) ## Lisp * [CL-INFLUXDB](https://github.com/mmaul/cl-influxdb) ## MATLAB * [influxdb-matlab](https://github.com/EnricSala/influxdb-matlab) ## .Net * [InfluxDB.Client.Net](https://github.com/AdysTech/InfluxDB.Client.Net) * Supports .Net and .Net Core * [InfluxData.Net](https://github.com/pootzko/InfluxData.Net) * [InfluxDB Client for .NET](https://github.com/MikaelGRA/InfluxDB.Client) * [InfluxClient](https://github.com/danesparza/InfluxClient) ## Perl * [AnyEvent::InfluxDB](https://github.com/ajgb/anyevent-influxdb) * [InfluxDB-LineProtocol](http://search.cpan.org/~domm/InfluxDB-LineProtocol/) * [InfluxDB::HTTP](https://github.com/raphaelthomas/InfluxDB-HTTP) ## PHP * [influxdb-php](https://github.com/influxdb/influxdb-php) * [InfluxDB PHP SDK (influxdb-php-sdk)](https://github.com/corley/influxdb-php-sdk) ## Python * [InfluxDB-Python (influxdb-python)](https://github.com/influxdb/influxdb-python) R ---------- * [influxdbr](https://cran.r-project.org/web/packages/influxdbr/) ## Ruby * [influxdb-ruby](https://github.com/influxdb/influxdb-ruby) * [Influxer (influxer)](https://github.com/palkan/influxer) ## Rust * [Flux (flux)](https://crates.io/crates/flux) * [Influent (influent)](https://crates.io/crates/influent) ## Scala * [scala-influxdb-client](https://github.com/paulgoldbaum/scala-influxdb-client) * [chronicler](https://github.com/fsanaulla/chronicler) * [Alpakka InfluxDB](https://doc.akka.io/docs/alpakka/current/influxdb.html) ## Sensu * [sensu-influxdb-extension](https://github.com/jhrv/sensu-influxdb-extension) ## SNMP agent * [SnmpCollector (snmpcollector)](https://github.com/toni-moreno/snmpcollector) * A full featured Generic SNMP data collector with Web Administration Interface for InfluxDB. [client libraries](/influxdb3/enterprise/tags/client-libraries/)[API](/influxdb3/enterprise/tags/api/)[developer tools](/influxdb3/enterprise/tags/developer-tools/) --- # InfluxDB v2 API client libraries Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 > client libraries and the `influx` CLI, *can’t query* data stored in > InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/)are available that integrate with your code to write and query data stored > in InfluxDB 3 Enterprise. > > InfluxDB 3 supports many different tools for [**writing**](/influxdb3/enterprise/write-data/)and [**querying**](/influxdb3/enterprise/query-data/) data.[**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use)to interact with InfluxDB 3 Enterprise. ## Client libraries for InfluxDB 2.x and 1.8+ InfluxDB client libraries are language-specific tools that integrate with InfluxDB APIs. InfluxDB v2 client libraries use InfluxDB `/api/v2` endpoints and work with [InfluxDB 2.x API compatibility endpoints](/influxdb/v1/api/buckets-v2-compatible/). Functionality varies among client libraries. InfluxDB client libraries are maintained by the InfluxDB community. For specifics about a client library, see the library’s GitHub repository. * [Arduino](https://github.com/tobiasschuerg/InfluxDB-Client-for-Arduino) *– contributed by [tobiasschuerg](https://github.com/tobiasschuerg)* * [C#](https://github.com/influxdata/influxdb-client-csharp) * [Dart](https://github.com/influxdata/influxdb-client-dart) * [Go](/influxdb3/enterprise/reference/client-libraries/v2/go/) * [Java](https://github.com/influxdata/influxdb-client-java) * [JavaScript](/influxdb3/enterprise/reference/client-libraries/v2/javascript/) * [Kotlin](https://github.com/influxdata/influxdb-client-java/tree/master/client-kotlin) * [PHP](https://github.com/influxdata/influxdb-client-php) * [Python](/influxdb3/enterprise/reference/client-libraries/v2/python/) * [R](https://github.com/influxdata/influxdb-client-r) * [Ruby](https://github.com/influxdata/influxdb-client-ruby) * [Scala](https://github.com/influxdata/influxdb-client-java/tree/master/client-scala) * [Swift](https://github.com/influxdata/influxdb-client-swift) [client libraries](/influxdb3/enterprise/tags/client-libraries/)[API](/influxdb3/enterprise/tags/api/)[developer tools](/influxdb3/enterprise/tags/developer-tools/) --- # Arduino client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/arduino/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Arduino is an open source hardware and software platform used for building electronics projects. The documentation for this client library is available on GitHub. [Arduino InfluxDB client](https://github.com/tobiasschuerg/InfluxDB-Client-for-Arduino) --- # C# client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/csharp/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. C# is a general-purpose object-oriented programming language. The documentation for this client library is available on GitHub. [C# InfluxDB client](https://github.com/influxdata/influxdb-client-csharp) --- # Dart client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/dart/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Dart is a programming language created for quick application development for both web and mobile apps. The documentation for this client library is available on GitHub. [Dart InfluxDB client](https://github.com/influxdata/influxdb-client-dart) --- # InfluxDB v2 Go client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/go/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. > > Use the [InfluxDB Go client library](https://github.com/influxdata/influxdb-client-go) to write data to an InfluxDB 3 Enterprise database. This guide presumes some familiarity with Go and InfluxDB. If just getting started, see [Get started with InfluxDB](/influxdb3/enterprise/get-started/). ## Before you begin 1. [Install Go 1.13 or later](https://golang.org/doc/install). 2. Add the client package your to your project dependencies. ``` # Add InfluxDB Go client package to your project go.mod go get github.com/influxdata/influxdb-client-go/v2 ``` 3. Ensure that InfluxDB is running and you can connect to it. For information about what URL to use to connect to your InfluxDB 3 Enterprise cluster, contact your InfluxData account representative. ## Boilerplate for the InfluxDB Go Client Library Use the Go library to write and query data from InfluxDB. 1. In your Go program, import the necessary packages and specify the entry point of your executable program. ``` package main import ( "context" "fmt" "time" "github.com/influxdata/influxdb-client-go/v2" ) ``` 2. Define variables for your InfluxDB [database](/influxdb3/enterprise/admin/databases/) (bucket), organization (required, but ignored), and [database token](/influxdb3/enterprise/admin/tokens/#database-tokens). ``` bucket := "DATABASE_NAME" org := "ignored" token := "DATABASE_TOKEN" // Store the URL of your InfluxDB instance url := "https://localhost:8181" ``` 3. Create the the InfluxDB Go client and pass in the `url` and `token` parameters. ``` client := influxdb2.NewClient(url, token) ``` 4. Create a **write client** with the `WriteAPIBlocking` method and pass in the `org` and `bucket` parameters. ``` writeAPI := client.WriteAPIBlocking(org, bucket) ``` ## Write data to InfluxDB with Go Use the Go library to write data to InfluxDB. 1. Create a [point](/influxdb3/enterprise/reference/glossary/#point) and write it to InfluxDB using the `WritePoint` method of the API writer struct. 2. Close the client to flush all pending writes and finish. ``` p := influxdb2.NewPoint("stat", map[string]string{"unit": "temperature"}, map[string]interface{}{"avg": 24.5, "max": 45}, time.Now()) writeAPI.WritePoint(context.Background(), p) client.Close() ``` ### Complete example write script ```go package main import ( "context" "fmt" "time" "github.com/influxdata/influxdb-client-go/v2" ) func main() { bucket := "DATABASE_NAME" org := "ignored" token := "DATABASE_TOKEN" // Store the URL of your InfluxDB instance url := "https://localhost:8181" // Create new client with default option for server url authenticate by token client := influxdb2.NewClient(url, token) // User blocking write client for writes to desired bucket writeAPI := client.WriteAPIBlocking(org, bucket) // Create point using full params constructor p := influxdb2.NewPoint("stat", map[string]string{"unit": "temperature"}, map[string]interface{}{"avg": 24.5, "max": 45}, time.Now()) // Write point immediately writeAPI.WritePoint(context.Background(), p) // Ensures background processes finishes client.Close() } ``` [client libraries](/influxdb3/enterprise/tags/client-libraries/)[Go](/influxdb3/enterprise/tags/go/) --- # Java client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/java/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Java is one of the oldest and most popular class-based, object-oriented programming languages. The documentation for this client library is available on GitHub. [Java InfluxDB client](https://github.com/influxdata/influxdb-client-java) --- # JavaScript client library for the InfluxDB v2 API Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/javascript/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. The [InfluxDB v2 JavaScript client library](https://github.com/influxdata/influxdb-client-js)for Node.js and browsers integrates with the InfluxDB v2 API to write data to an InfluxDB 3 Enterprise cluster. ### [JavaScript for browsers](/influxdb3/enterprise/reference/client-libraries/v2/javascript/browser/) Use the InfluxDB v2 JavaScript client library in browsers and front-end clients to write data to an InfluxDB Core database. ### [Node.js](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/) The [InfluxDB v2 JavaScript client library](https://github.com/influxdata/influxdb-client-js) for Node.js integrates with the InfluxDB v2 API to write data to an InfluxDB Core database. [client libraries](/influxdb3/enterprise/tags/client-libraries/)[JavaScript](/influxdb3/enterprise/tags/javascript/)[NodeJS](/influxdb3/enterprise/tags/nodejs/) --- # InfluxDB v2 JavaScript client library for web browsers Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/javascript/browser/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Use the [InfluxDB v2 JavaScript client library](https://github.com/influxdata/influxdb-client-js) in browsers and front-end clients to write data to an InfluxDB 3 Enterprise database. This library supports both front-end and server-side environments and provides the following distributions: * ECMAScript modules (ESM) and CommonJS modules (CJS) * Bundled ESM * Bundled UMD This guide presumes some familiarity with JavaScript, browser environments, and InfluxDB. If you’re just getting started with InfluxDB, see [Get started with InfluxDB](/influxdb/v2/get-started/). > [!Warning] > ### Tokens in production applications > > The examples below configure the authentication token in source code for demonstration purposes only. > To protect your data, take the following steps: 1. Avoid sending tokens to public clients such as web browsers and mobile apps. Regard any application secret sent to client devices as public and not confidential. 2. Use short-lived, [**read-only tokens**](/influxdb/v2/reference/cli/influx/auth/create/#create-a-read-only-authentication-token) whenever possible to prevent unauthorized writes and deletes. * [Before you begin](#before-you-begin) * [Use with module bundlers](#use-with-module-bundlers) * [Use bundled distributions with browsers and module loaders](#use-bundled-distributions-with-browsers-and-module-loaders) * [Get started with the example app](#get-started-with-the-example-app) ## Before you begin 1. Install [Node.js](https://nodejs.org/en/download/package-manager/) to serve your front-end app. 2. Ensure that InfluxDB is running and you can connect to it. For information about what URL to use to connect to your InfluxDB 3 Enterprise cluster, contact your InfluxData account representative. ## Use with module bundlers If you use a module bundler like Webpack or Parcel, install `@influxdata/influxdb-client-browser`. ## Use bundled distributions with browsers and module loaders 1. Configure InfluxDB properties for your script. ``` ``` 2. Import modules from the latest client library browser distribution.`@influxdata/influxdb-client-browser` exports bundled ESM and UMD syntaxes. [ESM](#esm)[UMD](#umd) ``` ``` ``` ``` After you’ve imported the client library, you’re ready to [get started writing data with the example app](#get-started-with-the-example-app). ## Get started with the example app The client library includes an example browser app that writes to your InfluxDB instance. 1. Clone the [influxdb-client-js](https://github.com/influxdata/influxdb-client-js) repository. 2. Navigate to the `examples` directory: ``` cd examples ``` 3. Update `./env_browser.js` with your InfluxDB 3 Enterprise cluster URL, your database name as `bucket`, an arbitrary string as `org`, and your database token. 4. Run the following command to start the application at [http://localhost:3001/examples/index.html]() ``` npm run browser ``` `index.html` loads the `env_browser.js` configuration, the client library ESM modules, and the application in your browser. For more examples, see how to [write data using the JavaScript client library for Node.js](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/write/). #### Related * [Write data with the InfluxDB v2 JavaScript client library](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/write/) [client libraries](/influxdb3/enterprise/tags/client-libraries/)[JavaScript](/influxdb3/enterprise/tags/javascript/) --- # Node.js JavaScript client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. The [InfluxDB v2 JavaScript client library](https://github.com/influxdata/influxdb-client-js)integrates with the InfluxDB v2 API to write data from Node.js and browser applications to an InfluxDB 3 Enterprise database. ## Use the client library in a Node.js application ### [Install the InfluxDB v2 JavaScript client library](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/install/) Install the Node.js JavaScript client library to write data to an InfluxDB Core database. ### [Write data](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/write/) The InfluxDB v2 JavaScript client library integrates with Node.js applications to write data to the InfluxDB v2 API. [client libraries](/influxdb3/enterprise/tags/client-libraries/)[JavaScript](/influxdb3/enterprise/tags/javascript/)[NodeJS](/influxdb3/enterprise/tags/nodejs/) --- # Install the InfluxDB v2 JavaScript client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/install/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. ## Install Node.js 1. Install [Node.js](https://nodejs.org/en/download/package-manager/). 2. Ensure that InfluxDB is running and you can connect to it. For information about what URL to use to connect to your InfluxDB 3 Enterprise cluster, contact your InfluxData account representative. 3. In your terminal, create a directory for your Node.js project and change to it. ``` mkdir influx-node-app && cd influx-node-app ``` 4. Enter the following command to generate an npm package for your project. * `npm`: the package manager included with Node.js * `-y`: uses defaults for the package and bypasses prompts ``` npm init -y ``` ## Install TypeScript Many of the client library examples use [TypeScript](https://www.typescriptlang.org/). Follow these steps to initialize the TypeScript project: 1. Install TypeScript and type definitions for Node.js. ``` npm i -g typescript && npm i --save-dev @types/node ``` 2. Enter the following command to create a TypeScript configuration (`tsconfig.json`) with default values: ``` tsc --init ``` 3. Run the TypeScript compiler. To recompile your code automatically as you make changes, pass the `--watch, -w` flag to the compiler. ``` tsc --watch ``` ## Install dependencies Use the `@influxdata/influxdb-client` JavaScript client library to write data in InfluxDB 3 Enterprise. Open a new terminal window and install the `@influxdata/influxdb-client` package for querying and writing data: ```sh npm i --save @influxdata/influxdb-client ``` The `@influxdata/influxdb-client-apis` client library package won’t work with InfluxDB 3 Enterprise. It only works with InfluxDB v2 management APIs. ## Configure credentials The client examples include an [`env`](https://github.com/influxdata/influxdb-client-js/blob/master/examples/env.js) module for accessing your InfluxDB properties from environment variables or from `env.js`. The examples use these properties to interact with the InfluxDB API. Set environment variables or update `env.js` with your InfluxDB [database](/influxdb3/enterprise/admin/databases/), organization (required, but ignored), [database token](/influxdb3/enterprise/admin/tokens/#database-tokens), and cluster URL. ```sh export INFLUX_URL=https://localhost:8181 export INFLUX_TOKEN=DATABASE_TOKEN export INFLUX_ORG=ORG_ID export INFLUX_DATABASE=DATABASE_NAME ``` Replace the following: * **`DATABASE_TOKEN`**: InfluxDB database token * **`ORG_ID`**: An arbitrary string (InfluxDB ignores this credential, but the client library requires it) * **`DATABASE_NAME`**: InfluxDB database name ## Next steps Once you’ve installed the client library and configured credentials, you’re ready to [write data](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/write/) to InfluxDB. [Write data](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/write/) [client libraries](/influxdb3/enterprise/tags/client-libraries/)[JavaScript](/influxdb3/enterprise/tags/javascript/) --- # Write data with the InfluxDB v2 JavaScript client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/write/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Use the [InfluxDB v2 JavaScript client library](https://github.com/influxdata/influxdb-client-js) to write data from a Node.js environment to InfluxDB. The JavaScript client library includes the following convenient features for writing data to InfluxDB: * Apply default tags to data points. * Buffer points into batches to optimize data transfer. * Automatically retry requests on failure. * Set an optional HTTP proxy address for your network. ### Before you begin * [Install the client library and other dependencies](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/install/). ### Write data with the client library 1. Instantiate a client by calling the `new InfluxDB()` constructor with your InfluxDB URL and database token (environment variables you already set in the [Install section](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/install/)). ``` import {InfluxDB, Point} from '@influxdata/influxdb-client' const influxDB = new InfluxDB({url: process.env.INFLUX_URL, token: process.env.INFLUX_TOKEN}) ``` 2. Use the `getWriteApi()` method of the client to create a **write client**. Provide your InfluxDB organization ID and database name. ``` const writeApi = influxDB.getWriteApi(process.env.INFLUX_ORG, process.env.INFLUX_DATABASE) ``` 3. To apply one or more [tags](/influxdb3/enterprise/reference/glossary/#tag) to all points, use the `useDefaultTags()` method. Provide tags as an object of key/value pairs. ``` writeApi.useDefaultTags({region: 'west'}) ``` 4. Use the `Point()` constructor to create a [point](/influxdb3/enterprise/reference/glossary/#point). 1. Call the constructor and provide a [measurement](/influxdb3/enterprise/reference/glossary/#measurement). 2. To add one or more tags, chain the `tag()` method to the constructor. Provide a `name` and `value`. 3. To add a field of type `float`, chain the `floatField()` method to the constructor. Provide a `name` and `value`. ``` const point1 = new Point('temperature') .tag('sensor_id', 'TLM010') .floatField('value', 24.0) ``` 5. Use the `writePoint()` method to write the point to your InfluxDB bucket. Finally, use the `close()` method to flush all pending writes. The example logs the new data point followed by “WRITE FINISHED” to stdout. ``` writeApi.writePoint(point1) writeApi.close().then(() => { console.log('WRITE FINISHED') }) ``` ### Complete example ```js 'use strict' /** @module write * Writes a data point to InfluxDB using the JavaScript client library with Node.js. **/ import {InfluxDB, Point} from '@influxdata/influxdb-client' /** * Instantiate the InfluxDB client * with a configuration object. **/ const influxDB = new InfluxDB({url: process.env.INFLUX_URL, token: process.env.INFLUX_TOKEN}) /** * Create a write client from the getWriteApi method. * Provide your org and database. **/ const writeApi = influxDB.getWriteApi(process.env.INFLUX_ORG, process.env.INFLUX_DATABASE) /** * Apply default tags to all points. **/ writeApi.useDefaultTags({region: 'west'}) /** * Create a point and write it to the buffer. **/ const point1 = new Point('temperature') .tag('sensor_id', 'TLM01') .floatField('value', 24.0) console.log(` ${point1}`) writeApi.writePoint(point1) /** * Flush pending writes and close writeApi. **/ writeApi.close().then(() => { console.log('WRITE FINISHED') }) ``` In your terminal with [environment variables or `env.js` set](/influxdb3/enterprise/reference/client-libraries/v2/javascript/nodejs/install/#configure-credentials), run the following command to execute the JavaScript file: ```sh node write.js ``` ### Response codes *For information about **InfluxDB API response codes**, see[InfluxDB API Write documentation](/influxdb3/enterprise/api/write-data/#operation/PostWrite).* #### Related * [Troubleshoot issues writing data](/influxdb3/enterprise/write-data/troubleshoot/) [client libraries](/influxdb3/enterprise/tags/client-libraries/)[JavaScript](/influxdb3/enterprise/tags/javascript/) --- # Kotlin client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/kotlin/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Kotlin is an open source programming language that runs on the Java Virtual Machine (JVM). The documentation for this client library is available on GitHub. [Kotlin InfluxDB client](https://github.com/influxdata/influxdb-client-java/tree/master/client-kotlin) --- # PHP client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/php/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. PHP is a popular general-purpose scripting language primarily used for web development. The documentation for this client library is available on GitHub. [PHP InfluxDB client](https://github.com/influxdata/influxdb-client-php) --- # Python client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/python/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. > > Use the [InfluxDB Python client library](https://github.com/influxdata/influxdb-client-python) to integrate InfluxDB into Python scripts and applications. This guide presumes some familiarity with Python and InfluxDB. If just getting started, see [Get started with InfluxDB](/influxdb3/enterprise/get-started/). ## Before you begin You’ll need the following prerequisites: 1. Install the InfluxDB Python library: ``` pip install influxdb-client ``` 2. InfluxDB Cloud Dedicated cluster URL using the HTTPS protocol–for example: ``` https://localhost:8181 ``` 3. Name of the [database](/influxdb3/enterprise/admin/databases/) to write to. 4. InfluxDB [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with permission to write to the database.*For security reasons, we recommend setting an environment variable to store your token and avoid exposing the raw token value in your script.* ## Write data to InfluxDB with Python Follow the steps to write [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) data to an InfluxDB Cloud Dedicated database. 1. In your editor, create a file for your Python program–for example: `write.py`. 2. In the file, import the InfluxDB client library. ``` import influxdb_client from influxdb_client.client.write_api import SYNCHRONOUS import os ``` 3. Define variables for your [database name](/influxdb3/enterprise/admin/databases/), organization (required, but ignored), and [database token](/influxdb3/enterprise/admin/tokens/#database-tokens). ``` database = "DATABASE_NAME" org = "ignored" # INFLUX_TOKEN is an environment variable you created for your database WRITE token token = os.getenv('INFLUX_TOKEN') url="https://localhost:8181" ``` 4. To instantiate the client, call the `influxdb_client.InfluxDBClient()` method with the following keyword arguments: `url`, `org`, and `token`. ``` client = influxdb_client.InfluxDBClient( url=url, token=token, org=org ) ``` The `InfluxDBClient` object has a `write_api` method used for configuration. 5. Instantiate a **write client** by calling the `client.write_api()` method with write configuration options. ``` write_api = client.write_api(write_options=SYNCHRONOUS) ``` 6. Create a [point](/influxdb3/enterprise/reference/glossary/#point) object and write it to InfluxDB using the `write` method of the API writer object. The write method requires three parameters: `bucket`, `org`, and `record`. ``` p = influxdb_client.Point("my_measurement").tag("location", "Prague").field("temperature", 25.3) write_api.write(bucket=database, org=org, record=p) ``` ### Complete example write script ```python import influxdb_client from influxdb_client.client.write_api import SYNCHRONOUS import os database = "DATABASE_NAME" org = "ignored" # INFLUX_TOKEN is an environment variable you created for your database WRITE token token = os.getenv('INFLUX_TOKEN') url="https://localhost:8181" client = influxdb_client.InfluxDBClient( url=url, token=token, org=org ) # Write script write_api = client.write_api(write_options=SYNCHRONOUS) p = influxdb_client.Point("my_measurement").tag("location", "Prague").field("temperature", 25.3) write_api.write(bucket=database, org=org, record=p) ``` ## Query data from InfluxDB with Python The InfluxDB v2 Python client can’t query InfluxDB Cloud Dedicated. To query your dedicated instance, use a Python [Flight SQL client with gRPC](/influxdb3/enterprise/reference/client-libraries/flight-sql/). [client libraries](/influxdb3/enterprise/tags/client-libraries/)[python](/influxdb3/enterprise/tags/python/) --- # R package client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/r/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. R is a programming language and software environment for statistical analysis, reporting, and graphical representation primarily used in data science. The documentation for this client library is available on GitHub. [R InfluxDB client](https://github.com/influxdata/influxdb-client-r) --- # Ruby client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/ruby/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Ruby is a highly flexible, open-source, object-oriented programming language. The documentation for this client library is available on GitHub. [Ruby InfluxDB client](https://github.com/influxdata/influxdb-client-ruby) --- # Scala client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/scala/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Scala is a general-purpose programming language that supports both object-oriented and functional programming. The documentation for this client library is available on GitHub. [Scala InfluxDB client](https://github.com/influxdata/influxdb-client-java/tree/master/client-scala) --- # Swift client library Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v2/swift/ > [!Warning] > #### Use InfluxDB 3 clients to query > > InfluxDB 3 supports [compatibility endpoints for *writing data*](/influxdb3/enterprise/write-data/compatibility-apis/) using InfluxDB v2 and v1 tools. However, the `/api/v2/query` API endpoint and associated tooling, such as InfluxDB v2 client libraries and the `influx` CLI, *can’t query* data stored in InfluxDB 3 Enterprise. > > [InfluxDB 3 client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) are available that integrate with your code to write and query data stored in InfluxDB 3 Enterprise. > > [**Compare tools you can use**](/influxdb3/enterprise/get-started/#tools-to-use) to interact with InfluxDB 3 Enterprise. Swift is a programming language created by Apple for building applications across multiple Apple platforms. The documentation for this client library is available on GitHub. [Swift InfluxDB client](https://github.com/influxdata/influxdb-client-swift) --- # InfluxDB 3 API client libraries Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/ ## Client libraries for InfluxDB 3 InfluxDB 3 client libraries are language-specific packages that work with and integrate with your application to write to and query data in InfluxDB 3 Enterprise. InfluxData and the user community maintain client libraries for developers who want to take advantage of: * Idioms for InfluxDB requests, responses, and errors. * Common patterns in a familiar programming language. * Faster development and less boilerplate code. InfluxDB client libraries provide configurable batch writing of data to InfluxDB HTTP APIs. They can be used to construct line protocol data and transform data from other formats to line protocol. InfluxDB 3 client libraries can query InfluxDB 3 using the Flight protocol to execute SQL and InfluxQL queries, request database information, and retrieve data stored in InfluxDB 3 Enterprise. Additional features may vary among client libraries. For specifics about a client library, see the library’s GitHub repository. InfluxDB 3 client libraries are part of the [Influx Community](https://github.com/InfluxCommunity). ### [C# .NET](https://github.com/InfluxCommunity/influxdb3-csharp) The InfluxDB 3 `influxdb3-csharp` C# .NET client library integrates with C# .NET scripts and applications to write and query data stored in an InfluxDB Core database. ### [Go](/influxdb3/enterprise/reference/client-libraries/v3/go/) The InfluxDB 3 `influxdb3-go` Go client library integrates with Go scripts and applications to write and query data stored in an InfluxDB 3 Enterprise database. ### [Java](/influxdb3/enterprise/reference/client-libraries/v3/java/) The InfluxDB 3 `influxdb3-java` Java client library integrates with application code to write and query data stored in an InfluxDB Core database. ### [JavaScript](https://github.com/InfluxCommunity/influxdb3-js) The InfluxDB 3 `influxdb3-js` JavaScript client library integrates with JavaScript scripts and applications to write and query data stored in an InfluxDB Core database. ### [Python](/influxdb3/enterprise/reference/client-libraries/v3/python/) The InfluxDB 3 `influxdb3-python` Python client library integrates with Python scripts and applications to write and query data stored in an InfluxDB Core database. ``` # Example: Write and query data # Write sensor data in batches from a CSV file to a database client.write_file(file='./data/home-sensor-data.csv', timestamp_column='time', tag_columns=["room"]) # Execute a query and retrieve data from the last 90 days table = client.query( '''SELECT * FROM home WHERE time >= now() - INTERVAL '90 days' ORDER BY time''') # This script assumes the client object is correctly configured # with your database name, token, and host URL. # After the script runs, the table variable contains the data # formatted as a PyArrow table. ``` ### [Rust](https://github.com/InfluxCommunity/influxdb3-rust) The InfluxDB 3 `influxdb3-rust` Rust client library integrates with Rust applications to write and query data stored in an InfluxDB 3 Enterprise database. [client libraries](/influxdb3/enterprise/tags/client-libraries/)[API](/influxdb3/enterprise/tags/api/)[developer tools](/influxdb3/enterprise/tags/developer-tools/) --- # C# .NET client library for InfluxDB 3 Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/csharp/ The InfluxDB 3 [`influxdb3-csharp` C# .NET client library](https://github.com/InfluxCommunity/influxdb3-csharp) integrates with C# .NET scripts and applications to write and query data stored in an InfluxDB 3 Enterprise database. **Latest version:**[v1.10.0](/influxdb3/enterprise/reference/client-libraries/v3/csharp/release-notes/)(2026-08-27) The documentation for this client library is available on GitHub. [InfluxDB 3 C# .NET client library](https://github.com/InfluxCommunity/influxdb3-csharp) [gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # influxdb3-csharp release notes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/csharp/release-notes/ ## v1.10.0 ### Features 1. [#308](https://github.com/InfluxCommunity/influxdb3-csharp/pull/308): Add `writeOptions` argument to write async methods. The passed `WriteOptions` argument will replace `WriteOptions` defined in `ClientConfig`, except for `Precision`, when the `writeOptions.Precision` argument is null. * **Breaking change**: Note this changes the `IInfluxDBClient` interface in that write methods now have an additional optional argument `writeOptions`. The CLR signature for this part of the library is now changed. ## v1.9.0 ### Features 1. [#269](https://github.com/InfluxCommunity/influxdb3-csharp/pull/269): Add partial writes support and default writes to the V2 API endpoint. * `NoSync` requires `UseV2Api=false` and the V3 API endpoint. * `AcceptPartial` applies only when writes are sent to the V3 API endpoint and is ignored when using the V2 API endpoint. See [Partial writes](https://docs.influxdata.com/influxdb3/core/write-data/http-api/v3-write-lp/#partial-writes) for more. ## v1.8.0 ### Features 1. [#227](https://github.com/InfluxCommunity/influxdb3-csharp/pull/227): Add example for write with interceptors. ## v1.7.0 ### Features 1. [#220](https://github.com/InfluxCommunity/influxdb3-csharp/pull/220): Add InfluxDB 3 Core/Enterprise structured errors handling. * CI updates ## v1.6.0 ### Features 1. [#193](https://github.com/InfluxCommunity/influxdb3-csharp/pull/193): Add option to disable gRPC query response compression: * New query option (`QueryOptions.DisableGrpcCompression`) added: `true` value disables gRPC response compression. Default value: `false` (compression enabled). * Also configurable via connection string query parameter (`disableGrpcCompression`). * Also configurable via environment variable (`INFLUX_DISABLE_GRPC_COMPRESSION`). ## v1.5.0 ### Features 1. [#175](https://github.com/InfluxCommunity/influxdb3-csharp/pull/175): Add QueryTimeout and WriteTimeout to ClientConfig. 2. [#179](https://github.com/InfluxCommunity/influxdb3-csharp/pull/179): Allows create ClientConfig from ClientConfig(string connectionString) and ClientConfig(IDictionary env) * CI updates ## v1.4.0 ### Features 1. [#174](https://github.com/InfluxCommunity/influxdb3-csharp/pull/174): Support passing HttpClient to InfluxDBClient. * CI updates ## v1.3.0 ### Features 1. [#164](https://github.com/InfluxCommunity/influxdb3-csharp/pull/164): Add function to get InfluxDB version. 2. [#168](https://github.com/InfluxCommunity/influxdb3-csharp/pull/168): Run integration tests against a locally started InfluxDB 3 Core server. ## v1.2.0 ### Features 1. [#155](https://github.com/InfluxCommunity/influxdb3-csharp/pull/155): Allows setting grpc options. 2. [#157](https://github.com/InfluxCommunity/influxdb3-csharp/pull/157): Fix: always clone `DefaultOptions` to keep it immutable. 3. [#158](https://github.com/InfluxCommunity/influxdb3-csharp/pull/158): Support fast writes without waiting for WAL persistence: * New write option (`WriteOptions.NoSync`) added: `true` value means faster write but without the confirmation that the data was persisted. Default value: `false`. * **Supported by self-managed InfluxDB 3 Core and Enterprise servers only!** * Also configurable via connection string query parameter (`writeNoSync`). * Also configurable via environment variable (`INFLUX_WRITE_NO_SYNC`). * Long precision string values added from v3 HTTP API: `"nanosecond"`, `"microsecond"`, `"millisecond"`,`"second"` ( in addition to the existing `"ns"`, `"us"`, `"ms"`, `"s"`). ## v1.1.0 ### Features 1. [#153](https://github.com/InfluxCommunity/influxdb3-csharp/pull/153): Add custom SSL root certificate support. * New configuration items: * `SslRootsFilePath` * `DisableCertificateRevocationListCheck` * **Disclaimer:** Using custom SSL root certificate configurations is recommended for development and testing purposes only. For production deployments, ensure custom certificates are added to the operating system’s trusted certificate store. ## v1.0.0 ### Features 1. [#132](https://github.com/InfluxCommunity/influxdb3-csharp/pull/132): Respect iox::column\_type::field metadata when mapping query results into values. * iox::column\_type::field::integer: =\> Long * iox::column\_type::field::uinteger: =\> Long * iox::column\_type::field::float: =\> Double * iox::column\_type::field::string: =\> String * iox::column\_type::field::boolean: =\> Boolean ## v0.8.0 ### Features 1.[#118](https://github.com/InfluxCommunity/influxdb3-csharp/pull/118): Simplify getting response headers and status code from `InfluxDBApiException`. Includes example runnable through `Examples/General`. ## v0.7.0 ### Migration Notice * `InfluxDBClient` constructor with connection options has new option `authScheme` with `null` default value: ``` - public InfluxDBClient(string host, string token, string? organization = null, string? database = null); + public InfluxDBClient(string host, string token, string? organization = null, string? database = null, string? authScheme = null) ``` This new option is used for Edge (OSS) authentication. ### Features 1. [#101](https://github.com/InfluxCommunity/influxdb3-csharp/pull/101): Add standard `user-agent` header to all calls. 2. [#111](https://github.com/InfluxCommunity/influxdb3-csharp/pull/111): Add InfluxDB Edge (OSS) authentication support. ### Bug Fixes 1. [#110](https://github.com/InfluxCommunity/influxdb3-csharp/pull/110): InfluxDB Edge (OSS) error handling. ## v0.6.0 ### Features 1. [#90](https://github.com/InfluxCommunity/influxdb3-csharp/pull/90): Custom `HTTP/gRPC` headers can be specified globally by config or per request ## v0.5.0 ### Features 1. [#71](https://github.com/InfluxCommunity/influxdb3-csharp/pull/71): Add support for named query parameters ### Others 1. [#80](https://github.com/InfluxCommunity/influxdb3-csharp/pull/80): Use net8.0 as a default target framework in Tests and Examples ## v0.4.0 ### Features 1. [#66](https://github.com/InfluxCommunity/influxdb3-csharp/pull/66): Default Tags for Writes ## v0.3.0 ### Features 1. [#36](https://github.com/InfluxCommunity/influxdb3-csharp/pull/46): Add client creation from connection string and environment variables. 2. [#52](https://github.com/InfluxCommunity/influxdb3-csharp/pull/52): Add structured query support ### Docs 1. [#52](https://github.com/InfluxCommunity/influxdb3-csharp/pull/52): Add downsampling example ## v0.2.0 ### Features 1. [#33](https://github.com/InfluxCommunity/influxdb3-csharp/pull/33): Add GZIP support 2. [#34](https://github.com/InfluxCommunity/influxdb3-csharp/pull/34): Add HTTP proxy and custom HTTP headers support ### Breaking Changes 1. [#35](https://github.com/InfluxCommunity/influxdb3-csharp/pull/35): Renamed config types and some options ## v0.1.0 * initial release of new client version --- # Go client library for InfluxDB 3 Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/go/ The InfluxDB 3 [`influxdb3-go` Go client library](https://github.com/InfluxCommunity/influxdb3-go) integrates with Go scripts and applications to write and query data stored in an InfluxDB 3 Enterprise database. **Latest version:**[v2.17.0](/influxdb3/enterprise/reference/client-libraries/v3/go/release-notes/)(2026-07-01) ## Installation ```sh go get github.com/InfluxCommunity/influxdb3-go/v2 ``` ## Importing the package The `influxdb3-go` client library module provides the `influxdb3` package. Import the package: ```go import ( "github.com/InfluxCommunity/influxdb3-go/v2/influxdb3" ) ``` ## API reference ### Function `New` Create a client to interact with InfluxDB. #### Syntax ```go New(config ClientConfig) ``` Initializes and returns a `influxdb3.Client` instance with the following: * Configuration and functions for writing to the database. * A `*flight.Client` and functions for querying the database. #### Parameters * **`config`**: A `ClientConfig` struct with the following configuration properties: * **`Host`** (string): the InfluxDB 3 Enterprise server URL * **`Token`** (string): a database token string * **`Database`** (string): the database to use for writing and querying. * **`Organization`** (string): *Optional*. The organization name or ID. * **`HTTPClient`** (`*http.Client`): *Optional*. Specifies a custom HTTP client, TLS configuration, or timeout to use. * **`WriteOptions`** (`*WriteOptions`): *Optional*. Options passed to the write client for writing to the database. * **`Headers`** (`http.Header`): *Optional*. Headers to include in all requests. #### Examples ##### Create an InfluxDB client ```go package main import ( "github.com/InfluxCommunity/influxdb3-go/v2/influxdb3" ) func main() { client, err := influxdb3.New(influxdb3.ClientConfig{ Host: "https://localhost:8181", Token: "DATABASE_TOKEN", Database: "DATABASE_NAME", }) defer func(client *influxdb3.Client) { err := client.Close() if err != nil { panic(err) } }(client) if(err != nil) { panic(err) } } ``` Replace the following configuration values: * `DATABASE_NAME`: the name of the InfluxDB [database](/influxdb3/enterprise/admin/databases/) to query * `DATABASE_TOKEN`: an InfluxDB [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with *read* permission on the specified database ## Class influxdb3.Client ### Function `Client.Query()` Query data from InfluxDB 3 using SQL. #### Syntax ```go client.Query(ctx context.Context, query string) ``` Sends a Flight query request with SQL to InfluxDB. Returns the following: * A custom iterator (\*QueryIterator) for accessing query result data and metadata. * An error, if any. #### Parameters * **`ctx`** (`context.Context`): the context to use for the request * **`query`** (string): the SQL query to execute. #### Examples ##### Query using SQL ```go query := `SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z'` iterator, err := client.Query(context.Background(), query) ``` ### Function `Client.QueryWithOptions()` Query data from InfluxDB 3 with query options such as **query type** for querying with InfluxQL. #### Syntax ```go client.QueryWithOptions(ctx context.Context, options *QueryOptions, query string) ``` Sends a query request with the specified query options to InfluxDB. Returns the following: * A custom iterator (\*QueryIterator) for accessing query result data and metadata. * An error, if any. #### Parameters * **`ctx`** (`context.Context`): the context to use for the request * **`options`**: query options (query type, optional database) * **`query`** (string): the SQL or InfluxQL query to execute. #### Examples ##### Query using InfluxQL ```go query := `SELECT * FROM home WHERE time >= 1641024000s AND time <= 1641024000s + 8h` queryOptions := influxdb3.QueryOptions{QueryType: influxdb3.InfluxQL} iterator, err := client.QueryWithOptions(context.Background(), &queryOptions, query) ``` [Flight client](/influxdb3/enterprise/tags/flight-client/)[go](/influxdb3/enterprise/tags/go/)[InfluxQL](/influxdb3/enterprise/tags/influxql/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight](/influxdb3/enterprise/tags/flight/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # influxdb3-go release notes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/go/release-notes/ ## v2.17.0 * CI updates ### Dependencies 1. [#268](https://github.com/InfluxCommunity/influxdb3-go/pull/268): Upgrade Google gRPC to 1.82.0. ## v2.16.0 ### Dependencies 1. [#261](https://github.com/InfluxCommunity/influxdb3-go/pull/261): Upgrade dependencies. ## v2.15.0 ### Bug Fixes 1. [#258](https://github.com/InfluxCommunity/influxdb3-go/pull/258): Restores write behavior changed in 2.14 by defaulting writes to the V2 API endpoint. * All InfluxDB 3 products now work with default write settings. * Write option `NoSync` requires `UseV2Api=false` and the V3 API endpoint. `AcceptPartial` applies only when writes are sent to the V3 API endpoint and is ignored otherwise. ## v2.14.0 ### BREAKING CHANGES 1. [#245](https://github.com/InfluxCommunity/influxdb3-go/pull/245): Adds `UseV2Api` write option and fixes partial writes support introduced in [#234](https://github.com/InfluxCommunity/influxdb3-go/pull/234). Users of InfluxDB Clustered version must set this option to `true` for writing. ### Features 1. [#227](https://github.com/InfluxCommunity/influxdb3-go/pull/227): Support custom tag order via `WithTagOrder` write option. See [Sort tags by priority](https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/schema-design/#sort-tags-by-query-priority) for more. * Point and struct writes now use custom line protocol marshaling in this client; dependency on `github.com/influxdata/line-protocol/v2` will be dropped in a future release. * Direct use of `lineprotocol.Precision` is deprecated; prefer `influxdb3.Precision` constants. 2. [#234](https://github.com/InfluxCommunity/influxdb3-go/pull/234): Support partial writes via `AcceptPartial` write option. See [Partial writes](https://docs.influxdata.com/influxdb3/core/write-data/http-api/v3-write-lp/#partial-writes) for more. 3. [#238](https://github.com/InfluxCommunity/influxdb3-go/pull/238): Support `arrow.NULL` data type in query response iterator. ### Dependencies 1. [#249](https://github.com/InfluxCommunity/influxdb3-go/pull/249): Requires Go 1.25 or newer. ## v2.13.0 ### Features 1. [#223](https://github.com/InfluxCommunity/influxdb3-go/pull/223): Support passing middleware functions to the Flight client. ## v2.12.0 ### Features 1. [#209](https://github.com/InfluxCommunity/influxdb3-go/pull/209): Document the global nature of gRPC compression in README.md ### Bug Fixes 1. [#213](https://github.com/InfluxCommunity/influxdb3-go/pull/213): Fixed InfluxDB 3 Core/Enterprise write error deserialization. ### Dependencies 1. Minimum supported Go version is now 1.24 as required by `google.golang.org/grpc` v1.76.0 - see [grpc-go#8509](https://github.com/grpc/grpc-go/issues/8509) ## v2.11.0 * CI updates ### Features 1. [192](https://github.com/InfluxCommunity/influxdb3-go/pull/192): Add clearer timeout parameters. * The `ClientConfig` property `Timeout` is now deprecated. * In `ClientConfig` a new property `WriteTimeout` supersedes it. * In `ClientConfig` the new property `QueryTimeout` sets a default context deadline to be used with all queries. * New environment variables `INFLUXDB_WRITE_TIMEOUT` and `INFLUXDB_QUERY_TIMEOUT` are added. ### Bug Fixes 1. [#193](https://github.com/InfluxCommunity/influxdb3-go/pull/193): Use `influxdb3_test` package in the documentation examples to enforce public API usage. ## v2.10.0 ### Features 1. [#189](https://github.com/InfluxCommunity/influxdb3-go/pull/189): Add transparent gRPC compression support. * CI updates ## v2.9.0 ### Features 1. [#169](https://github.com/InfluxCommunity/influxdb3-go/pull/169): Support user-defined type converter function for writes points. 2. [#171](https://github.com/InfluxCommunity/influxdb3-go/pull/171): Add function to get InfluxDB version. 3. [#176](https://github.com/InfluxCommunity/influxdb3-go/pull/176): Add comment warning null when calling getMeasurement function. 4. [#174](https://github.com/InfluxCommunity/influxdb3-go/pull/174): Run integration tests against a locally started InfluxDB 3 Core server. ## v2.8.0 ### Features 1. [#159](https://github.com/InfluxCommunity/influxdb3-go/pull/159): Support fast writes without waiting for WAL persistence: * New write option (`WriteOptions.NoSync`) added: `true` value means faster write but without the confirmation that the data was persisted. Default value: `false`. * **Supported by self-managed InfluxDB 3 Core and Enterprise servers only!** * Also configurable via connection string query parameter (`writeNoSync`). * Also configurable via environment variable (`INFLUX_WRITE_NO_SYNC`). * Long precision string values added from v3 HTTP API: `"nanosecond"`, `"microsecond"`, `"millisecond"`, `"second"` ( in addition to the existing `"ns"`, `"us"`, `"ms"`, `"s"`). ### Bug Fixes 1. [#164](https://github.com/InfluxCommunity/influxdb3-go/pull/164): Updates * Go version to 1.23.9 * `golang.org/x/net` to v0.38.0 2. [#166](https://github.com/InfluxCommunity/influxdb3-go/pull/166): Upgrades dependency `apache/arrow/go` to `apache/arrow-go` v18. ## v2.7.0 ### Bug Fixes 1. [#158](https://github.com/InfluxCommunity/influxdb3-go/pull/158): Refactor Batcher and LPBatcher: * Fields and methods using `capacity` renamed to `initialCapacity`. * Log messages when buffer data is not being emitted are simplified. * `SetCapacity` methods on both structures are now deprecated. * `WithCapacity` and `WithBufferCapacity` options are now deprecated. ## v2.6.0 ### Features 1. [#155](https://github.com/InfluxCommunity/influxdb3-go/pull/155): Batcher warnings when buffer data is not being automatically emitted changed to level `Debug` in slog. ## v2.5.0 ### Features 1. [#146](https://github.com/InfluxCommunity/influxdb3-go/pull/146): Add error field to QueryIterator to hold first possible error encountered when retrieving records from the flight Reader. 2. [#147](https://github.com/InfluxCommunity/influxdb3-go/pull/147): Ability to pass `grpc.CallOption` functions to the underlying flight Client. 3. [#149](https://github.com/InfluxCommunity/influxdb3-go/pull/149): Add default configuration for the built-in HTTP client and expose new configurable parameters: * `Timeout` - The overall time limit for requests made by the Client. A negative value means no timeout. Default value: 10 seconds. * `IdleConnectionTimeout` - Maximum time an idle connection will remain idle before closing itself. A negative value means no timeout. Default value: 90 seconds. * `MaxIdleConnections` - Maximum number of idle connections. It sets both `transport.MaxIdleConn` and`transport.MaxIdleConnsPerHost` to the same value. A negative value means no limit. Default value: 100. 4. [#154](https://github.com/InfluxCommunity/influxdb3-go/pull/154): Export functions `NewQueryIterator` and`NewPointValueIterator` to simplify testing. ## v2.4.0 ### Features 1. [#141](https://github.com/InfluxCommunity/influxdb3-go/pull/141): Add proxy and custom SSL root certificate support. ## v2.3.0 ### Features 1. [#131](https://github.com/InfluxCommunity/influxdb3-go/pull/131): Add new PointValueIterator based on google guidelines [Guidelines](https://github.com/googleapis/google-cloud-go/wiki/Iterator-Guidelines) ## v2.2.0 ### Bug fixes 1. [#134](https://github.com/InfluxCommunity/influxdb3-go/pull/134): Reduce minimal Go version to 1.22, remove unnecessary toolchain constraints. ## v2.1.0 ### Bug fixes 1. [#127](https://github.com/InfluxCommunity/influxdb3-go/pull/127): LPBatcher now returns first line of the internal buffer when the line length exceeds the batch size. ## v2.0.0 ### Breaking Changes :warning: **This is a breaking change release.** > > > Previously, the Query API did not respect the metadata type for columns returned from InfluxDB v3. This release fixes this issue. As a result, the type of some columns may differ from previous versions. For example, the timestamp column will now be `time.Time` instead of `arrow.Timestamp`. > > Update steps: 1. Update library: `go get github.com/InfluxCommunity/influxdb3-go/v2/influxdb3` 2. Update import path in Go files to `github.com/InfluxCommunity/influxdb3-go/v2/influxdb3`. ### Features 1. [#114](https://github.com/InfluxCommunity/influxdb3-go/pull/114): Query API respects metadata types for columns returned from InfluxDB v3. Tags are mapped as a “string”, timestamp as “time.Time”, and fields as their respective types: * iox::column\_type::field::integer: =\> int64 * iox::column\_type::field::uinteger: =\> uint64 * iox::column\_type::field::float: =\> float64 * iox::column\_type::field::string: =\> string * iox::column\_type::field::boolean: =\> bool ## v1.0.0 :warning: **The v1.0.0 release had a malformed module path regarding the [Go Module Requirements](https://go.dev/ref/mod#major-version-suffixes). For a Go Module project, you need to use version 2 of the client.** ### Breaking Changes :warning: **This is a breaking change release.** > > > Previously, the Query API did not respect the metadata type for columns returned from InfluxDB v3. This release fixes this issue. As a result, the type of some columns may differ from previous versions. For example, the timestamp column will now be `time.Time` instead of `arrow.Timestamp`. > > ### Features 1. [#114](https://github.com/InfluxCommunity/influxdb3-go/pull/114): Query API respects metadata types for columns returned from InfluxDB v3. Tags are mapped as a “string”, timestamp as “time.Time”, and fields as their respective types: * iox::column\_type::field::integer: =\> int64 * iox::column\_type::field::uinteger: =\> uint64 * iox::column\_type::field::float: =\> float64 * iox::column\_type::field::string: =\> string * iox::column\_type::field::boolean: =\> bool ## v0.14.0 ### Features 1. [#112](https://github.com/InfluxCommunity/influxdb3-go/pull/112): Adds `LPBatcher` for lineprotocol batching following the model of the Point `Batcher`. ### Bug Fixes 1. [#113](https://github.com/InfluxCommunity/influxdb3-go/pull/113): Honor struct tags on WriteData, avoid panic for unexported fields ## v0.13.0 ### Features 1. [#108](https://github.com/InfluxCommunity/influxdb3-go/pull/108): Allow Request.GetBody to be set when writing gzipped data to make calls more resilient. 2. [#111](https://github.com/InfluxCommunity/influxdb3-go/pull/111): Support tabs in tag values. ## v0.12.0 ### Features 1. [#107](https://github.com/InfluxCommunity/influxdb3-go/pull/107): Add `Batcher` to simplify the process of writing data in batches. ## v0.11.0 ### Bug Fixes 1. [#105](https://github.com/InfluxCommunity/influxdb3-go/pull/105): Support newlines in tag values. 2. [#106](https://github.com/InfluxCommunity/influxdb3-go/pull/106): Close `resp.Body` after HTTP error response is encountered. ## v0.10.0 ### Features 1. [#100](https://github.com/InfluxCommunity/influxdb3-go/pull/100): Expose HTTP Response headers in `ServerError` ### Bug Fixes 1. [#94](https://github.com/InfluxCommunity/influxdb3-go/pull/94): Resource leak from unclosed `Response` 2. [#97](https://github.com/InfluxCommunity/influxdb3-go/pull/97): Style and performance improvements discovered by `golangci-lint` 3. [#98](https://github.com/InfluxCommunity/influxdb3-go/pull/98): Cloud Dedicated database creation ignores the name given by an argument * CI updates ## v0.9.0 ### Features 1. [#87](https://github.com/InfluxCommunity/influxdb3-go/pull/87): Add Cloud Dedicated database creation support 2. [#91](https://github.com/InfluxCommunity/influxdb3-go/pull/91): Add Edge (OSS) authentication support. ### Bug Fixes 1. [#89](https://github.com/InfluxCommunity/influxdb3-go/pull/89): InfluxDB Edge (OSS) error handling ## v0.8.0 ### Features 1. [#85](https://github.com/InfluxCommunity/influxdb3-go/pull/85): Add standard `user-agent` header to gRPC requests. 2. [#86](https://github.com/InfluxCommunity/influxdb3-go/pull/86): Add Serverless bucket creation support ## v0.7.0 ### Features 1. [#74](https://github.com/InfluxCommunity/influxdb3-go/pull/74): Use `log/slog` to print debug information instead of `fmt.Printf` 2. [#76](https://github.com/InfluxCommunity/influxdb3-go/pull/76): Add custom headers support for queries (gRPC requests) ### Bug Fixes 1. [#71](https://github.com/InfluxCommunity/influxdb3-go/pull/71): Rename `FlightSQL` constant to `SQL` ### Others 1. [#68](https://github.com/InfluxCommunity/influxdb3-go/pull/68): Upgrade Go version to 1.22. ## v0.6.0 ### Features 1. [#56](https://github.com/InfluxCommunity/influxdb3-go/pull/56): Add support for named query parameters ### Bug Fixes 1. [#59](https://github.com/InfluxCommunity/influxdb3-go/pull/59): Export Default Tags from package ## v0.5.0 ### Features 1. [#50](https://github.com/InfluxCommunity/influxdb3-go/pull/50): Default Tags for Writes ## v0.4.0 ### Features 1. [#45](https://github.com/InfluxCommunity/influxdb3-go/pull/45): Add structured query support ### Docs 1. [#45](https://github.com/InfluxCommunity/influxdb3-go/pull/45): Add downsampling example ## v0.3.0 ### Features 1. [#36](https://github.com/InfluxCommunity/influxdb3-go/pull/36): Add client creation from connection string and environment variables. ### Bug Fixes 1. [#37](https://github.com/InfluxCommunity/influxdb3-go/pull/37): `runtime error` for iterating Arrow Record without rows ## v0.2.0 ### Features 1. [#30](https://github.com/InfluxCommunity/influxdb3-go/pull/30): Add custom HTTP headers support ### Breaking Changes 1. [#31](https://github.com/InfluxCommunity/influxdb3-go/pull/31): Changed package to `influxdb3`. Renamed config types and some options. ## v0.1.0 * initial release of new client version * write using v2 api * query using SQL * query using influxQL --- # Java client library for InfluxDB 3 Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/java/ The InfluxDB 3 [`influxdb3-java` Java client library](https://github.com/InfluxCommunity/influxdb3-java) integrates with Java application code to write and query data stored in InfluxDB 3 Enterprise. **Latest version:**[v1.11.0](/influxdb3/enterprise/reference/client-libraries/v3/java/release-notes/)(2026-08-27) InfluxDB client libraries provide configurable batch writing of data to InfluxDB 3 Enterprise. Use client libraries to construct line protocol data, transform data from other formats to line protocol, and batch write line protocol data to InfluxDB HTTP APIs. InfluxDB 3 client libraries can query InfluxDB 3 Enterprise using SQL or InfluxQL. The `influxdb3-java` Java client library wraps the Apache Arrow `org.apache.arrow.flight.FlightClient`in a convenient InfluxDB 3 interface for executing SQL and InfluxQL queries, requesting server metadata, and retrieving data from InfluxDB 3 Enterprise using the Flight protocol with gRPC. * [Installation](#installation) * [Using Maven](#using-maven) * [Using Gradle](#using-gradle) * [Importing the client](#importing-the-client) * [API reference](#api-reference) * [Classes](#classes) * [InfluxDBClient interface](#influxdbclient-interface) * [Initialize with credential parameters](#initialize-with-credential-parameters) * [InfluxDBClient instance methods](#influxdbclient-instance-methods) * [InfluxDBClient.writePoint](#influxdbclientwritepoint) * [InfluxDBClient.query](#influxdbclientquery) #### Example: write and query data The following example shows how to use `influxdb3-java` to write and query data stored in InfluxDB 3 Enterprise. ``` package com.influxdata.demo; import com.influxdb.v3.client.InfluxDBClient; import com.influxdb.v3.client.Point; import com.influxdb.v3.client.query.QueryOptions; import com.influxdb.v3.client.query.QueryType; import java.time.Instant; import java.util.stream.Stream; public class HelloInfluxDB { private static final String HOST_URL = "https://localhost:8181"; // your cluster URL private static final String DATABASE = "DATABASE_NAME"; // your InfluxDB database name private static final char[] TOKEN = System.getenv("DATABASE_TOKEN"); // a local environment variable that stores your database token // Create a client instance that writes and queries data in your database. public static void main(String[] args) { // Instantiate the client with your InfluxDB credentials try (InfluxDBClient client = InfluxDBClient.getInstance(HOST_URL, TOKEN, DATABASE)) { writeData(client); queryData(client); } catch (Exception e) { System.err.println("An error occurred while connecting to InfluxDB!"); e.printStackTrace(); } } // Use the Point class to construct time series data. private static void writeData(InfluxDBClient client) { Point point = Point.measurement("temperature") .setTag("location", "London") .setField("value", 30.01) .setTimestamp(Instant.now().minusSeconds(10)); try { client.writePoint(point); System.out.println("Data is written to the database."); } catch (Exception e) { System.err.println("Failed to write data to the database."); e.printStackTrace(); } } // Use SQL to query the most recent 10 measurements private static void queryData(InfluxDBClient client) { System.out.printf("--------------------------------------------------------%n"); System.out.printf("| %-8s | %-8s | %-30s |%n", "location", "value", "time"); System.out.printf("--------------------------------------------------------%n"); String sql = "select time,location,value from temperature order by time desc limit 10"; try (Stream stream = client.query(sql)) { stream.forEach(row -> System.out.printf("| %-8s | %-8s | %-30s |%n", row[1], row[2], row[0])); } catch (Exception e) { System.err.println("Failed to query data from the database."); e.printStackTrace(); } } } ``` Source: [suyashcjoshi/SimpleJavaInfluxDB](https://github.com/suyashcjoshi/SimpleJavaInfluxDB/) on GitHub Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise[database](/influxdb3/enterprise/admin/databases/) to read and write data to * `DATABASE_TOKEN`: a local environment variable that stores your[token](/influxdb3/enterprise/admin/tokens/database/)–the token must have read and write permissions on the specified database. ### Run the example to write and query data 1. Build an executable JAR for the project–for example, using Maven: ``` mvn package ``` 2. In your terminal, run the `java` command to write and query data in your database: ``` java \ --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED \ -jar target/PROJECT_NAME.jar ``` Include the following in your command: * [`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`](https://arrow.apache.org/docs/java/install.html#id3): with Java version 9 or later and Apache Arrow version 16 or later, exposes JDK internals for Arrow. For more options, see the [Apache Arrow Java install documentation](https://arrow.apache.org/docs/java/install.html). * `-jar target/PROJECT_NAME.jar`: your `.jar` file to run. The output is the newly written data from your InfluxDB 3 Enterprise database. ## Installation Include `com.influxdb.influxdb3-java` in your project dependencies. [Maven pom.xml](#maven-pomxml)[Gradle dependency script](#gradle-dependency-script) ```xml com.influxdb influxdb3-java RELEASE ``` ```groovy dependencies { implementation group: 'com.influxdb', name: 'influxdb3-java', version: 'latest.release' } ``` ## Importing the client The `influxdb3-java` client library package provides`com.influxdb.v3.client` classes for constructing, writing, and querying data stored in InfluxDB 3 Enterprise. ## API reference * [Interface InfluxDBClient](#interface-influxdbclient) * [Initialize with credential parameters](#initialize-with-credential-parameters) * [InfluxDBClient instance methods](#influxdbclient-instance-methods) * [InfluxDBClient.writePoint](#influxdbclientwritepoint) * [InfluxDBClient.query](#influxdbclientquery) ## InfluxDBClient interface `InfluxDBClient` provides an interface for interacting with InfluxDB APIs for writing and querying data. The `InfluxDBClient.getInstance` constructor initializes and returns a client instance with the following: * A *write client* configured for writing to the database. * An Arrow *Flight client* configured for querying the database. To initialize a client, call `getInstance` and pass your credentials as one of the following types: * [parameters](#initialize-with-credential-parameters) * a [`ClientConfig`](https://github.com/InfluxCommunity/influxdb3-java/blob/main/src/main/java/com/influxdb/v3/client/config/ClientConfig.java) * a [database connection string](#initialize-using-a-database-connection-string) ### Initialize with credential parameters ```java static InfluxDBClient getInstance(@Nonnull final String host, @Nullable final char[] token, @Nullable final String database) ``` * `host` (string): The host URL of the InfluxDB instance. * `database` (string): The [database](/influxdb3/enterprise/admin/databases/) to use for writing and querying. * `token` (char array): A [database token](/influxdb3/enterprise/admin/tokens/database/) with read/write permissions. #### Example: initialize with credential parameters ```java package com.influxdata.demo; import com.influxdb.v3.client.InfluxDBClient; import com.influxdb.v3.client.Point; import com.influxdb.v3.client.query.QueryOptions; import com.influxdb.v3.client.query.QueryType; import java.time.Instant; import java.util.stream.Stream; public class HelloInfluxDB { private static final String HOST_URL = "https://localhost:8181"; private static final String DATABASE = "DATABASE_NAME"; private static final char[] TOKEN = System.getenv("DATABASE_TOKEN"); // Create a client instance, and then write and query data in InfluxDB. public static void main(String[] args) { try (InfluxDBClient client = InfluxDBClient.getInstance(HOST_URL, DATABASE_TOKEN, DATABASE)) { writeData(client); queryData(client); } catch (Exception e) { System.err.println("An error occurred while connecting to InfluxDB!"); e.printStackTrace(); } } } ``` Replace the following: * `DATABASE_NAME`: your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: a[database token](/influxdb3/enterprise/admin/tokens/database/) that has the necessary permissions on the specified database. #### Default tags To include default [tags](/influxdb3/enterprise/reference/glossary/#tag) in all written data, pass a `Map` of tag keys and values. ```java InfluxDBClient getInstance(@Nonnull final String host, @Nullable final char[] token, @Nullable final String database, @Nullable Map defaultTags) ``` ### Initialize using a database connection string ```java "https://localhost:8181" + "?token=DATABASE_TOKEN&database=DATABASE_NAME" ``` Replace the following: * `DATABASE_NAME`: your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: a[database token](/influxdb3/enterprise/admin/tokens/database/) that has the necessary permissions on the specified database. ### InfluxDBClient instance methods #### InfluxDBClient.writePoint To write points as line protocol to a database: 1. [Initialize the `client`](#initialize-with-credential-parameters)–your token must have write permission on the specified database. 2. Use the `com.influxdb.v3.client.Point` class to create time series data. 3. Call the `client.writePoint()` method to write points as line protocol in your database. ``` // Use the Point class to construct time series data. // Call client.writePoint to write the point in your database. private static void writeData(InfluxDBClient client) { Point point = Point.measurement("temperature") .setTag("location", "London") .setField("value", 30.01) .setTimestamp(Instant.now().minusSeconds(10)); try { client.writePoint(point); System.out.println("Data written to the database."); } catch (Exception e) { System.err.println("Failed to write data to the database."); e.printStackTrace(); } } ``` #### InfluxDBClient.query To query data and process the results: 1. [Initialize the `client`](#initialize-with-credential-parameters)–the token must have read permission on the database you want to query. 2. Call `client.query()` and provide your SQL query as a string. 3. Use the result stream’s built-in iterator to process row data. ``` // Query the latest 10 measurements using SQL private static void queryData(InfluxDBClient client) { System.out.printf("--------------------------------------------------------%n"); System.out.printf("| %-8s | %-8s | %-30s |%n", "location", "value", "time"); System.out.printf("--------------------------------------------------------%n"); String sql = "select time,location,value from temperature order by time desc limit 10"; try (Stream stream = client.query(sql)) { stream.forEach(row -> System.out.printf("| %-8s | %-8s | %-30s |%n", row[1], row[2], row[0])); } catch (Exception e) { System.err.println("Failed to query data from the database."); e.printStackTrace(); } } ``` [View the InfluxDB 3 Java client library](https://github.com/InfluxCommunity/influxdb3-java/) [Flight client](/influxdb3/enterprise/tags/flight-client/)[Java](/influxdb3/enterprise/tags/java/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # influxdb3-java release notes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/java/release-notes/ ## v1.11.0 ### Bug Fixes 1. [#425](https://github.com/InfluxCommunity/influxdb3-java/pull/425): * Only throws `InfluxDBPartialWriteException` when: * Error response status code is `400`. * Error response format `{"error":"...","data":[{"error_message":"...","line_number":2,"original_line": "..."}]}` is returned with `data` must be an array. * `accept_partial` is set to `true`. * Write endpoint must be `api/v3/write_lp`. ### Breaking Changes 1. [#407](https://github.com/InfluxCommunity/influxdb3-java/pull/407): Upgrade minimum JDK requirement to version 17. ## v1.10.0 ### Features 1. [#388](https://github.com/InfluxCommunity/influxdb3-java/pull/388): Adds partial writes support and defaults writes to the V2 API endpoint. * `noSync` requires `useV2Api=false` and the V3 API endpoint. * `acceptPartial` applies only when writes are sent to the V3 API endpoint and is ignored when using the V2 API endpoint. * See [Partial writes](https://docs.influxdata.com/influxdb3/core/write-data/http-api/v3-write-lp/#partial-writes) for more. ### Bug Fixes 1. [#384](https://github.com/InfluxCommunity/influxdb3-java/pull/384): Always set `precision` to `nanosecond` when writing Points. ## v1.9.0 ### Features 1. [#360](https://github.com/InfluxCommunity/influxdb3-java/pull/360): Support passing interceptors to the Flight client. 2. [#363](https://github.com/InfluxCommunity/influxdb3-java/pull/363): Support custom tag order via `tagOrder` write option. See [Sort tags by priority](https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/schema-design/#sort-tags-by-query-priority) for more. ## v1.8.0 ### Features 1. [#352](https://github.com/InfluxCommunity/influxdb3-java/pull/352): Upgrade Arrow Flight client dependencies. ### Bug Fixes 1. [#351](https://github.com/InfluxCommunity/influxdb3-java/pull/351): Enterprise/Core structured errors handling. * CI updates ## v1.7.0 ### Bug Fixes 1. [#317](https://github.com/InfluxCommunity/influxdb3-java/pull/317): Fix Arrow memory leak when stream close fails due to thread interrupts. 2. [#318](https://github.com/InfluxCommunity/influxdb3-java/pull/318): Explicit releasing of the VectorSchemaRoot. ## v1.6.0 ### Features 1. [#306](https://github.com/InfluxCommunity/influxdb3-java/pull/306): Improve closing of Arrow `FlightStream`. ### Bug Fixes 1. [#310](https://github.com/InfluxCommunity/influxdb3-java/pull/310): Ensure `QueryOptions` objects are left unchanged within the `queryData` implementation. ## v1.5.0 ### Features 1. [#289](https://github.com/InfluxCommunity/influxdb3-java/pull/289) Add the possibility to disable gRPC compression via the `disableGRPCCompression` parameter in the `ClientConfig`. * CI updates ## v1.4.0 ### Features 1. [#265](https://github.com/InfluxCommunity/influxdb3-java/pull/265) Add more precise timeout properties to `ClientConfig`. 1. Current property `timeout` is deprecated, as it applies only to the Write API and can be confusing to some users. 2. Two new properties are added, along with getters and similar setters in the `ClientConfig.Builder`. 1. `writeTimeout` - a `java.time.Duration` that applies only to the Write API. 2. `queryTimeout` - a `java.time.Duration` used to calculate deadlines when using the Query API. 3. These properties can also be defined when creating a client using environment variables. Respectively: 1. `INFLUX_WRITE_TIMEOUT` - a positive integer. The time unit is in seconds. 2. `INFLUX_QUERY_TIMEOUT` - a positive integer. The time unit is in seconds. 4. These properties can also be defined when creating a client using system properties. Respectively: 1. `influx.writeTimeout` - a positive integer. The time unit is in seconds. 2. `influx.queryTimeout` - a positive integer. The time unit is in seconds. * CI updates ## v1.3.0 ### Features 1. [#250](https://github.com/InfluxCommunity/influxdb3-java/pull/250) Upgrade Netty version to 4.2.3.Final. 2. [#251](https://github.com/InfluxCommunity/influxdb3-java/pull/251) Add comment warning null when calling getMeasurement function. 3. [#252](https://github.com/InfluxCommunity/influxdb3-java/pull/252) Run integration tests against a locally started InfluxDB 3 Core server. ### Documentation 1. [#253](https://github.com/InfluxCommunity/influxdb3-java/pull/253) New Durable example showing client reuse for better resource management. ## v1.2.0 ### Features 1. [#209](https://github.com/InfluxCommunity/influxdb3-java/pull/209) Add query function returning row as map. 2. [#238](https://github.com/InfluxCommunity/influxdb3-java/pull/238): Support fast writes without waiting for WAL persistence: * New write option (`WriteOptions.noSync`) added: `true` value means faster write but without the confirmation that the data was persisted. Default value: `false`. * **Supported by self-managed InfluxDB 3 Core and Enterprise servers only!** * Also configurable via connection string query parameter (`writeNoSync`). * Also configurable via environment variable (`INFLUX_WRITE_NO_SYNC`). * Long precision string values added from v3 HTTP API: `"nanosecond"`, `"microsecond"`, `"millisecond"`,`"second"` ( in addition to the existing `"ns"`, `"us"`, `"ms"`, `"s"`). 3. [#241](https://github.com/InfluxCommunity/influxdb3-java/pull/241): Some default options will be used from a getter. 4. [#243](https://github.com/InfluxCommunity/influxdb3-java/pull/243): Add function to get InfluxDB version. ### Bug Fixes 1. [\#239](https://github.com/InfluxCommunity/influxdb3-java/pull/239): Use write options from `ClientConfig` in`InfluxDBClientImpl` write methods: ``` public void writeRecord(@Nullable final String record); public void writeRecords(@Nonnull final List records); public void writePoint(@Nullable final Point point); public void writePoints(@Nonnull final List points); ``` ## v1.1.0 ### Features 1. [#229](https://github.com/InfluxCommunity/influxdb3-java/pull/229): Support proxy and custom ssl root certificates 2. [#232](https://github.com/InfluxCommunity/influxdb3-java/pull/232): Allow set rpc max message size through maxInboundMessageSize in ClientConfig 3. [#233](https://github.com/InfluxCommunity/influxdb3-java/pull/233): More detailed documentation about timestamp handling for query and write functions 4. [#236](https://github.com/InfluxCommunity/influxdb3-java/pull/236): Supports Java 21. ## v1.0.0 ### Features 1. [#200](https://github.com/InfluxCommunity/influxdb3-java/pull/200): Respect iox::column\_type::field metadata when mapping query results into values. * iox::column\_type::field::integer: =\> Long * iox::column\_type::field::uinteger: =\> Long * iox::column\_type::field::float: =\> Double * iox::column\_type::field::string: =\> String * iox::column\_type::field::boolean: =\> Boolean ### Dependencies 1. [#202](https://github.com/InfluxCommunity/influxdb3-java/pull/202): Migrate from `flight-grpc` to `flight-core` package. ## v0.9.0 ### Features 1. [#158](https://github.com/InfluxCommunity/influxdb3-java/pull/158): Add InfluxDB Edge (OSS) authentication support. 2. [#163](https://github.com/InfluxCommunity/influxdb3-java/pull/163): Introduces `InfluxDBApiHttpException` to facilitate write retries and error recovery. ### Bug Fixes 1. [#148](https://github.com/InfluxCommunity/influxdb3-java/pull/148): InfluxDB Edge (OSS) error handling 2. [#153](https://github.com/InfluxCommunity/influxdb3-java/pull/153): Parsing timestamp columns ## v0.8.0 ### Features 1. [#144](https://github.com/InfluxCommunity/influxdb3-java/pull/133): user-agent header is updated for both REST and gRPC calls. ## v0.7.0 ### Features 1. [\#107](https://github.com/InfluxCommunity/influxdb3-java/pull/107): Custom headers are also supported for the query (gRPC request) ``` ClientConfig config = new ClientConfig.Builder() .host("https://us-east-1-1.aws.cloud2.influxdata.com") .token("my-token".toCharArray()) .database("my-database") .headers(Map.of("X-Tracing-Id", "123")) .build(); try (InfluxDBClient client = InfluxDBClient.getInstance(config)) { // // your code here // } catch (Exception e) { throw new RuntimeException(e); } ``` 2. [\#108](https://github.com/InfluxCommunity/influxdb3-java/pull/108): Custom headers can be specified per request (query/write): ``` ClientConfig config = new ClientConfig.Builder() .host("https://us-east-1-1.aws.cloud2.influxdata.com") .token("my-token".toCharArray()) .database("my-database") .build(); try (InfluxDBClient client = InfluxDBClient.getInstance(config)) { // // Write with custom headers // WriteOptions writeOptions = new WriteOptions( Map.of("X-Tracing-Id", "852") ); client.writeRecord("mem,tag=one value=1.0", writeOptions); // // Query with custom headers // QueryOptions queryOptions = new QueryOptions( Map.of("X-Tracing-Id", "852") ); Stream rows = client.query("select * from cpu", queryOptions); } catch (Exception e) { throw new RuntimeException(e); } ``` ## v0.6.0 ### Features 1. [#94](https://github.com/InfluxCommunity/influxdb3-java/pull/94): Add support for named query parameters ## v0.5.1 Resync artifacts with Maven Central. ## v0.5.0 ### Features 1. [#78](https://github.com/InfluxCommunity/influxdb3-java/pull/78): Default Tags can be used when writing points. ### Bug Fixes 1. [#77](https://github.com/InfluxCommunity/influxdb3-java/pull/77): Serialize InfluxDB response to `PointValues` ## v0.4.0 ### Features 1. [#41](https://github.com/InfluxCommunity/influxdb3-java/pull/41): Add structured query support ## v0.3.1 ### Bug Fixes 1. [#55](https://github.com/InfluxCommunity/influxdb3-java/pull/55): Iteration over more Arrow streams ## v0.3.0 ### Features 1. [#40](https://github.com/InfluxCommunity/influxdb3-java/pull/40): Add client creation from connection string, environment variables or system properties. ## v0.2.0 ### Features 1. [#27](https://github.com/InfluxCommunity/influxdb3-java/pull/27): Add GZIP support 2. [#30](https://github.com/InfluxCommunity/influxdb3-java/pull/30): Add HTTP proxy and custom headers support ### Breaking Changes 1. [#31](https://github.com/InfluxCommunity/influxdb3-java/pull/31): Renamed config types and some options ## v0.1.0 * initial release of new client version --- # JavaScript client library for InfluxDB 3 Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/javascript/ The InfluxDB 3 [`influxdb3-js` JavaScript client library](https://github.com/InfluxCommunity/influxdb3-js) integrates with JavaScript scripts and applications to write and query data stored in an InfluxDB 3 Enterprise database. **Latest version:**[v2.4.0](/influxdb3/enterprise/reference/client-libraries/v3/javascript/release-notes/)(2026-08-27) The documentation for this client library is available on GitHub. [InfluxDB 3 JavaScript client library](https://github.com/InfluxCommunity/influxdb3-js) [Flight client](/influxdb3/enterprise/tags/flight-client/)[JavaScript](/influxdb3/enterprise/tags/javascript/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # influxdb3-js release notes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/javascript/release-notes/ ## v2.4.0 > [!Important] > #### Breaking Change: Node v22 required for v2.4.0 > > With `v2.4.0`, Node v20 support is dropped. Node v22 or newer is now required ([#881](https://github.com/InfluxCommunity/influxdb3-js/pull/881)). ### Bugfix 1. [#864](https://github.com/InfluxCommunity/influxdb3-js/pull/864): Add support for connecting to InfluxDB servers using IPv6 addresses. * IPv6 addresses in server URLs must be enclosed in square brackets, for example, http://[2001:db8::1]:8086. * IPv6 zone identifiers are not currently supported. ## v2.3.0 ### Features 1. [#790](https://github.com/InfluxCommunity/influxdb3-js/pull/790): When setting Point timestamp to `new Date()`, the timestamp sent to the servers will be converted to the precision set in `WriteOptions`. 2. [#796](https://github.com/InfluxCommunity/influxdb3-js/pull/796): Adds partial write support and API endpoint selection for writes. * Writes use the V2 API endpoint by default. * `noSync` requires the V3 API endpoint; `acceptPartial` applies only to the V3 API endpoint and is ignored when using the V2 API endpoint. * See [Partial writes](https://docs.influxdata.com/influxdb3/core/write-data/http-api/v3-write-lp/#partial-writes) for more. ### Build 1. [#775](https://github.com/InfluxCommunity/influxdb3-js/pull/775): Upgrade typescript to 6.0.3. ## v2.2.0 ### Features 1. [#707](https://github.com/InfluxCommunity/influxdb3-js/pull/707): Support passing interceptor functions to the Flight client. 2. [#711](https://github.com/InfluxCommunity/influxdb3-js/pull/711): Support custom tag order via `tagOrder` write option. See [Sort tags by priority](https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/schema-design/#sort-tags-by-query-priority) for more. ## v2.1.0 ### Bugfix 1. [#688](https://github.com/InfluxCommunity/influxdb3-js/pull/688): InfluxDB 3 Core/Enterprise write errors details handling. ## v2.0.0 > > > :warning: **Breaking Changes**: Node v18 support is dropped because it is end of life. > > ### Features 1. [#622](https://github.com/InfluxCommunity/influxdb3-js/pull/622): * Deprecated ConnectionOptions.timeout. * Added ConnectionOptions.queryTimeout and ConnectionOptions.writeTimeout. * Added QueryOptions.timeout and WriteOptions.timeout. * Users can pass timeout directly to the query and write functions. * CI updates ### Docs 1. [#645](https://github.com/InfluxCommunity/influxdb3-js/pull/645): Document gRPC compression. ## v1.4.0 * CI updates ## v1.3.0 ### Features 1. [#585](https://github.com/InfluxCommunity/influxdb3-js/pull/585): Add function to get InfluxDB version. 2. [#588](https://github.com/InfluxCommunity/influxdb3-js/pull/588): Support grpc options available via [@grpc/grpc-js](https://github.com/grpc/grpc-node/blob/master/packages/grpc-js/README.md#supported-channel-options). * New client option (`grpcOptions`) added. Accepts key-value pairs available via @grpc/grpc-js. * New queryOptions option (`grpcOptions`) added. Accepts key-value pairs available via @grpc/grpc-js. * Configurable also with the environment variable (`INFLUXDB_GRPC_OPTIONS`) which takes a comma separated list of key-value pairs. * See new example `clientWithGrpcOptions.ts`. * Not supported in browser API. 3. [#591](https://github.com/InfluxCommunity/influxdb3-js/pull/591): Add comment warning null when calling getMeasurement function. 4. [#592](https://github.com/InfluxCommunity/influxdb3-js/pull/592): Run integration tests against a locally started InfluxDB 3 Core server. ## v1.2.0 ### Features 1. [#574](https://github.com/InfluxCommunity/influxdb3-js/pull/574): Support fast writes without waiting for WAL persistence: * New write option (`WriteOptions.noSync`) added: `true` value means faster write but without the confirmation that the data was persisted. Default value: `false`. * **Supported by self-managed InfluxDB 3 Core and Enterprise servers only!** * Also configurable via connection string query parameter (`writeNoSync`). * Also configurable via environment variable (`INFLUX_WRITE_NO_SYNC`). * Long precision string values added from v3 HTTP API: `"nanosecond"`, `"microsecond"`, `"millisecond"`,`"second"` (in addition to the existing `"ns"`, `"us"`, `"ms"`, `"s"`). ### Bugfix 1. [#570](https://github.com/InfluxCommunity/influxdb3-js/pull/570): Fixes the bug that makes query results duplicate rows [#553](https://github.com/InfluxCommunity/influxdb3-js/issues/553). 2. [#575](https://github.com/InfluxCommunity/influxdb3-js/pull/575): Upgrades build ecmascript to `es2023` ## v1.1.0 ### Features 1. [#545](https://github.com/InfluxCommunity/influxdb3-js/pull/545): Sets the correct versions for the client-browser package. ## v1.0.0 ### Features 1. [#491](https://github.com/InfluxCommunity/influxdb3-js/pull/491): Respect iox::column\_type::field metadata when mapping query results into values. * iox::column\_type::field::integer: =\> number * iox::column\_type::field::uinteger: =\> number * iox::column\_type::field::float: =\> number * iox::column\_type::field::string: =\> string * iox::column\_type::field::boolean: =\> boolean 2. [499](https://github.com/InfluxCommunity/influxdb3-js/pull/499): Migrate to new doc library ## v0.12.0 ### Bugfix 1. [437](https://github.com/InfluxCommunity/influxdb3-js/pull/437): Simplify iterating over Arrow’s batches in `QueryAPI` ## v0.11.0 ### Features 1. [410](https://github.com/InfluxCommunity/influxdb3-js/pull/410): Accepts HTTP responses with 2xx status codes as a success for writes. ## v0.10.0 ### Features 1. [369](https://github.com/InfluxCommunity/influxdb3-js/pull/369): Propagates headers from HTTP response to HttpError when an error is returned from the server. 2. [377](https://github.com/InfluxCommunity/influxdb3-js/pull/377): Add InfluxDB Edge (OSS) authentication support. ### Bugfix 1. [376](https://github.com/InfluxCommunity/influxdb3-js/pull/376): Handle InfluxDB Edge (OSS) errors better. ## v0.9.0 ### Features 1. [319](https://github.com/InfluxCommunity/influxdb3-js/pull/319): Adds standard `user-agent` header to calls. ## v0.8.0 ### Breaking Changes 1. [293](https://github.com/InfluxCommunity/influxdb3-js/pull/293): The Query API now uses a `QueryOptions` structure in `client.query()` methods. The `queryType` and `queryParams` values are now wrapped inside of it. QueryOptions also support adding custom headers. Query parameters are changed from type `Map` to type `Record`. ### Features 1. [293](https://github.com/InfluxCommunity/influxdb3-js/pull/293): `QueryOptions` also support adding custom headers. ## v0.7.0 ### Features 1. [#256](https://github.com/InfluxCommunity/influxdb3-js/pull/256): Adds support for named query parameters ## v0.6.0 ### Bugfix 1. [#221](https://github.com/InfluxCommunity/influxdb3-js/issues/221): Client options processing ## v0.5.0 ### Features 1. [#183](https://github.com/InfluxCommunity/influxdb3-js/pull/183): Default Tags for Writes ## v0.4.1 ### Bugfix 1. [#164](https://github.com/InfluxCommunity/influxdb3-js/issues/164): Query infinite wait state ## v0.4.0 ### Features 1. [#157](https://github.com/InfluxCommunity/influxdb3-js/pull/157): Add client instantiation from connection string and environment variables ## v0.3.1 Fixed package distribution files. The distribution files were not being included in the npm package. ## v0.3.0 ### Features 1. [#89](https://github.com/InfluxCommunity/influxdb3-js/pull/89): Add structured query support ### Docs 1. [#89](https://github.com/InfluxCommunity/influxdb3-js/pull/89): Add downsampling example ## v0.2.0 ### Features 1. [#52](https://github.com/InfluxCommunity/influxdb3-js/pull/52): Add support for browser environment ### Docs 1. [#52](https://github.com/InfluxCommunity/influxdb3-js/pull/52): Improve examples ## v0.1.0 * initial release of new client version * write using v2 api * query using FlightSQL * query using InfluxQl --- # Python client library for InfluxDB 3 Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/python/ * influxdb3-python v0.10.0 The InfluxDB 3 [`influxdb3-python` Python client library](https://github.com/InfluxCommunity/influxdb3-python)integrates InfluxDB 3 Enterprise write and query operations with Python scripts and applications. **Latest version:**[v0.21.0](/influxdb3/enterprise/reference/client-libraries/v3/python/release-notes/)(2026-08-27) InfluxDB client libraries provide configurable batch writing of data to InfluxDB 3 Enterprise. Client libraries can be used to construct line protocol data, transform data from other formats to line protocol, and batch write line protocol data to InfluxDB HTTP APIs. InfluxDB 3 client libraries can query InfluxDB 3 Enterprise using SQL or InfluxQL. The `influxdb3-python` Python client library wraps the Apache Arrow `pyarrow.flight` client in a convenient InfluxDB 3 interface for executing SQL and InfluxQL queries, requesting server metadata, and retrieving data from InfluxDB 3 Enterprise using the Flight protocol with gRPC. > [!Note] > Code samples in this page use the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). * [Installation](#installation) * [Importing the module](#importing-the-module) * [API reference](#api-reference) * [Classes](#classes) * [Class InfluxDBClient3](#class-influxdbclient3) * [Parameters](#parameters) * [Writing modes](#writing-modes) * [InfluxDBClient3 instance methods](#influxdbclient3-instance-methods) * [InfluxDBClient3.write](#influxdbclient3write) * [InfluxDBClient3.write\_file](#influxdbclient3write_file) * [InfluxDBClient3.query](#influxdbclient3query) * [InfluxDBClient3.close](#influxdbclient3close) * [Class Point](#class-point) * [Class WriteOptions](#class-writeoptions) * [Parameters](#parameters-4) * [Functions](#functions) * [Function write\_client\_options(\*\*kwargs)](#function-write_client_optionskwargs) * [Function flight\_client\_options(\*\*kwargs)](#function-flight_client_optionskwargs) * [Constants](#constants) * [Exceptions](#exceptions) ## Installation Install the client library and dependencies using `pip`: ```bash pip install influxdb3-python ``` ## Importing the module The `influxdb3-python` client library package provides the `influxdb_client_3`module. Import the module: ```python import influxdb_client_3 ``` Import specific class methods from the module: ```python from influxdb_client_3 import InfluxDBClient3, Point, WriteOptions ``` * [`influxdb_client_3.InfluxDBClient3`](#class-influxdbclient3): a class for interacting with InfluxDB * [`influxdb_client_3.Point`](#class-point): a class for constructing a time series data point * `influxdb_client_3.WriteOptions`: a class for configuring client write options ## API reference The `influxdb_client_3` module includes the following classes and functions. * [Classes](#classes) * [Functions](#functions) * [Constants](#constants) * [Exceptions](#exceptions) ## Classes * [Class InfluxDBClient3](#class-influxdbclient3) * [Parameters](#parameters) * [Writing modes](#writing-modes) * [InfluxDBClient3 instance methods](#influxdbclient3-instance-methods) * [InfluxDBClient3.write](#influxdbclient3write) * [InfluxDBClient3.write\_file](#influxdbclient3write_file) * [InfluxDBClient3.query](#influxdbclient3query) * [InfluxDBClient3.close](#influxdbclient3close) * [Class Point](#class-point) * [Class WriteOptions](#class-writeoptions) * [Parameters](#parameters-4) ## Class InfluxDBClient3 Provides an interface for interacting with InfluxDB APIs for writing and querying data. The `InfluxDBClient3` constructor initializes and returns a client instance with the following: * A singleton *write client* configured for writing to the database. * A singleton *Flight client* configured for querying the database. ### Parameters * **`host`** (string): The host URL of the InfluxDB instance. * **`database`** (string): The database to use for writing and querying. * **`token`** (string): A database token with read/write permissions. * *Optional* **`write_client_options`** (dict): Options to use when writing to InfluxDB. If `None`, writes are [synchronous](#synchronous-writing). * *Optional* **`flight_client_options`** (dict): Options to use when querying InfluxDB. ### Writing modes When writing data, the client uses one of the following modes: * [Synchronous writing](#synchronous-writing) * [Batch writing](#batch-writing) * Asynchronous writing: Deprecated #### Synchronous writing Default. When no `write_client_options` are provided during the initialization of `InfluxDBClient3`, writes are synchronous. When writing data in synchronous mode, the client immediately tries to write the provided data to InfluxDB, doesn’t retry failed requests, and doesn’t invoke response callbacks. ##### Example: initialize a client with synchronous (non-batch) defaults The following example initializes a client for writing and querying data in an InfluxDB 3 Enterprise database. Given that `write_client_options` isn’t specified, the client uses the default [synchronous writing](#synchronous-writing) mode. ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with read/write permissions on the specified database To explicitly specify synchronous mode, create a client with `write_options=SYNCHRONOUS`–for example: ```python from influxdb_client_3 import InfluxDBClient3, write_client_options, SYNCHRONOUS wco = write_client_options(write_options=SYNCHRONOUS) client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN", write_client_options=wco, flight_client_options=None) ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with write permissions on the specified database #### Batch writing Batch writing is particularly useful for efficient bulk data operations. Options include setting batch size, flush intervals, retry intervals, and more. Batch writing groups multiple writes into a single request to InfluxDB. In batching mode, the client adds the record or records to a batch, and then schedules the batch for writing to InfluxDB. The client writes the batch to InfluxDB after reaching `write_client_options.batch_size` or `write_client_options.flush_interval`. If a write fails, the client reschedules the write according to the `write_client_options` retry options. ##### Configuring write client options Use `WriteOptions` and `write_client_options` to configure batch writing and response handling for the client: 1. Instantiate `WriteOptions`. To use batch defaults, call the constructor without specifying parameters. 2. Call `write_client_options` and use the `write_options` parameter to specify the `WriteOptions` instance from the preceding step. Specify callback parameters (success, error, and retry) to invoke functions on success or error. 3. Instantiate `InfluxDBClient3` and use the `write_client_options` parameter to specify the `dict` output from the preceding step. ##### Example: initialize a client using batch defaults and callbacks The following example shows how to use batch mode with defaults and specify callback functions for the response status (success, error, or retryable error). ```python from influxdb_client_3 import(InfluxDBClient3, write_client_options, WriteOptions, InfluxDBError) status = None # Define callbacks for write responses def success(self, data: str): status = "Success writing batch: data: {data}" assert status.startswith('Success'), f"Expected {status} to be success" def error(self, data: str, err: InfluxDBError): status = f"Error writing batch: config: {self}, data: {data}, error: {err}" assert status.startswith('Success'), f"Expected {status} to be success" def retry(self, data: str, err: InfluxDBError): status = f"Retry error writing batch: config: {self}, data: {data}, error: {err}" assert status.startswith('Success'), f"Expected {status} to be success" # Instantiate WriteOptions for batching write_options = WriteOptions() wco = write_client_options(success_callback=success, error_callback=error, retry_callback=retry, write_options=write_options) # Use the with...as statement to ensure the file is properly closed and resources # are released. with InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN", write_client_options=wco) as client: client.write_file(file='./data/home-sensor-data.csv', timestamp_column='time', tag_columns=["room"], write_precision='s') ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with write permissions on the specified database ### InfluxDBClient3 instance methods ### InfluxDBClient3.write Writes a record or a list of records to InfluxDB. #### Parameters * **`record`** (*record* or list): A record or list of records to write. A record can be a `Point` object, a dict that represents a point, a line protocol string, or a `DataFrame`. * **`database`** (string): The database to write to. Default is to write to the database specified for the client. * \*\*`**kwargs`\*\*: Additional write options–for example: * **`write_precision`** (string): *Optional*. Default is `"ns"`. Specifies the [precision](/influxdb3/enterprise/reference/glossary/#precision) (`"ms"`, `"s"`, `"us"`, `"ns"`) for timestamps in `record`. * **`write_client_options`** (dict): *Optional*. Specifies callback functions and options for [batch writing](#batch-writing) mode. To generate the `dict`, use the [`write_client_options` function](#function-write_client_optionskwargs). #### Example: write a line protocol string ```python from influxdb_client_3 import InfluxDBClient3 point = "home,room=Living\\ Room temp=21.1,hum=35.9,co=0i 1641024000" client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") client.write(record=point, write_precision="s") ``` The following sample code executes an SQL query to retrieve the point: ```python # Execute an SQL query table = client.query(query='''SELECT room FROM home WHERE temp=21.1 AND time=from_unixtime(1641024000)''') # table is a pyarrow.Table room = table[0][0] assert f"{room}" == 'Living Room', f"Expected {room} to be Living Room" ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with write permissions on the specified database #### Example: write data using points The `influxdb_client_3.Point` class provides an interface for constructing a data point for a measurement and setting fields, tags, and the timestamp for the point. The following example shows how to create a `Point` object, and then write the data to InfluxDB. ```python from influxdb_client_3 import Point, InfluxDBClient3 point = Point("home").tag("room", "Kitchen").field("temp", 21.5).field("hum", .25) client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") client.write(point) ``` The following sample code executes an InfluxQL query to retrieve the written data: ```python # Execute an InfluxQL query table = client.query(query='''SELECT DISTINCT(temp) as val FROM home WHERE temp > 21.0 AND time >= now() - 10m''', language="influxql") # table is a pyarrow.Table df = table.to_pandas() assert 21.5 in df['val'].values, f"Expected value in {df['val']}" ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with write permissions on the specified database ##### Example: write data using a dict `InfluxDBClient3` can serialize a dictionary object into line protocol. If you pass a `dict` to `InfluxDBClient3.write`, the client expects the `dict` to have the following *point* attributes: * **measurement** (string): the measurement name * **tags** (dict): a dictionary of tag key-value pairs * **fields** (dict): a dictionary of field key-value pairs * **time**: the [timestamp](/influxdb3/enterprise/reference/glossary/#timestamp) for the record The following example shows how to define a `dict` that represents a point, and then write the data to InfluxDB. ```python from influxdb_client_3 import InfluxDBClient3 # Using point dictionary structure points = { "measurement": "home", "tags": {"room": "Kitchen", "sensor": "K001"}, "fields": {"temp": 72.2, "hum": 36.9, "co": 4}, "time": 1641067200 } client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") client.write(record=points, write_precision="s") ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with write permissions on the specified database ### InfluxDBClient3.write\_file Writes data from a file to InfluxDB. Execution is synchronous. #### Parameters * **`file`** (string): A path to a file containing records to write to InfluxDB. The filename must end with one of the following supported extensions. For more information about encoding and formatting data, see the documentation for each supported format: * `.feather`: [Feather](https://arrow.apache.org/docs/python/feather.html) * `.parquet`: [Parquet](https://arrow.apache.org/docs/python/parquet.html) * `.csv`: [Comma-separated values](https://arrow.apache.org/docs/python/csv.html) * `.json`: [JSON](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html) * `.orc`: [ORC](https://arrow.apache.org/docs/python/orc.html) * **`measurement_name`** (string): Defines the measurement name for records in the file. The specified value takes precedence over `measurement` and `iox::measurement` columns in the file. If no value is specified for the parameter, and a `measurement` column exists in the file, the `measurement` column value is used for the measurement name. If no value is specified for the parameter, and no `measurement` column exists, the `iox::measurement` column value is used for the measurement name. * **`tag_columns`** (list): Tag column names. Columns not included in the list and not specified by another parameter are assumed to be fields. * **`timestamp_column`** (string): The name of the column that contains timestamps. Default is `'time'`. * **`database`** (`str`): The database to write to. Default is to write to the database specified for the client. * **`file_parser_options`** (callable): A function for providing additional arguments to the file parser. * **`**kwargs`**: Additional options to pass to the `WriteAPI`–for example: * **`write_precision`** (string): *Optional*. Default is `"ns"`. Specifies the [precision](/influxdb3/enterprise/reference/glossary/#precision) (`"ms"`, `"s"`, `"us"`, `"ns"`) for timestamps in `record`. * **`write_client_options`** (dict): *Optional*. Specifies callback functions and options for [batch writing](#batch-writing) mode. To generate the `dict`, use the [`write_client_options` function](#function-write_client_optionskwargs). #### Example: use batch options when writing file data The following example shows how to specify customized write options for batching, retries, and response callbacks, and how to write data from CSV and JSON files to InfluxDB: ```python from influxdb_client_3 import(InfluxDBClient3, write_client_options, WritePrecision, WriteOptions, InfluxDBError) # Define the result object result = { 'config': None, 'status': None, 'data': None, 'error': None } # Define callbacks for write responses def success_callback(self, data: str): result['config'] = self result['status'] = 'success' result['data'] = data assert result['data'] != None, f"Expected {result['data']}" print("Successfully wrote data: {result['data']}") def error_callback(self, data: str, exception: InfluxDBError): result['config'] = self result['status'] = 'error' result['data'] = data result['error'] = exception assert result['status'] == "success", f"Expected {result['error']} to be success for {result['config']}" def retry_callback(self, data: str, exception: InfluxDBError): result['config'] = self result['status'] = 'retry_error' result['data'] = data result['error'] = exception assert result['status'] == "success", f"Expected {result['status']} to be success for {result['config']}" write_options = WriteOptions(batch_size=500, flush_interval=10_000, jitter_interval=2_000, retry_interval=5_000, max_retries=5, max_retry_delay=30_000, exponential_base=2) wco = write_client_options(success_callback=success_callback, error_callback=error_callback, retry_callback=retry_callback, write_options=write_options) with InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN", write_client_options=wco) as client: client.write_file(file='./data/home-sensor-data.csv', timestamp_column='time', tag_columns=["room"], write_precision='s') client.write_file(file='./data/home-sensor-data.json', timestamp_column='time', tag_columns=["room"], write_precision='s') ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with write permissions on the specified database ### InfluxDBClient3.query Sends a Flight request to execute the specified SQL or InfluxQL query. Returns all data in the query result as an Arrow table ([`pyarrow.Table`](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html) instance). #### Parameters * **`query`** (string): the SQL or InfluxQL to execute. * **`language`** (string): the query language used in the `query` parameter–`"sql"` or `"influxql"`. Default is `"sql"`. * **`mode`** (string): Specifies the output to return from the [`pyarrow.flight.FlightStreamReader`](https://arrow.apache.org/docs/python/generated/pyarrow.flight.FlightStreamReader.html#pyarrow.flight.FlightStreamReader). Default is `"all"`. * `all`: Read the entire contents of the stream and return it as a [`pyarrow.Table`](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). * `chunk`: Read the next message (a `FlightStreamChunk`) and return `data` and `app_metadata`. Returns `null` if there are no more messages. * `pandas`: Read the contents of the stream and return it as a [`pandas.DataFrame`](https://pandas.pydata.org/pandas-docs/stable/reference/frame.html). * `reader`: Convert the `FlightStreamReader` into a [`pyarrow.RecordBatchReader`](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatchReader.html#pyarrow-recordbatchreader). * `schema`: Return the schema for all record batches in the stream. * **`**kwargs`**: [`FlightCallOptions`](https://arrow.apache.org/docs/python/generated/pyarrow.flight.FlightCallOptions.html#pyarrow.flight.FlightCallOptions) #### Example: query using SQL ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") table = client.query("SELECT * from home WHERE time >= now() - INTERVAL '90 days'") # Filter columns. print(table.select(['room', 'temp'])) # Use PyArrow to aggregate data. print(table.group_by('hum').aggregate([])) ``` In the examples, replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with read permission on the specified database #### Example: query using InfluxQL ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") query = "SELECT * from home WHERE time >= -90d" table = client.query(query=query, language="influxql") # Filter columns. print(table.select(['room', 'temp'])) ``` ##### Example: read all data from the stream and return a pandas DataFrame ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") query = "SELECT * from home WHERE time >= now() - INTERVAL '90 days'" pd = client.query(query=query, mode="pandas") # Print the pandas DataFrame formatted as a Markdown table. print(pd.to_markdown()) ``` ##### Example: view the schema for all batches in the stream ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") table = client.query("""SELECT * from home WHERE time >= now() - INTERVAL '90 days'""") # View the table schema. print(table.schema) ``` ##### Example: retrieve the result schema and no data ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") query = "SELECT * from home WHERE time >= now() - INTERVAL '90 days'" schema = client.query(query=query, mode="schema") print(schema) ``` ##### Specify a timeout Pass `timeout=` for [`FlightCallOptions`](https://arrow.apache.org/docs/python/generated/pyarrow.flight.FlightCallOptions.html#pyarrow.flight.FlightCallOptions) to use a custom timeout. ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") query = "SELECT * from home WHERE time >= now() - INTERVAL '90 days'" client.query(query=query, timeout=5) ``` ### InfluxDBClient3.close Sends all remaining records from the batch to InfluxDB, and then closes the underlying write client and Flight client to release resources. #### Example: close a client ```python from influxdb_client_3 import InfluxDBClient3 client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN") client.close() ``` ## Class Point Provides an interface for constructing a time series data point for a measurement, and setting fields, tags, and timestamp. ```python from influxdb_client_3 import Point point = Point("home").tag("room", "Living Room").field("temp", 72) ``` See how to [write data using points](#example-write-data-using-points). ## Class WriteOptions Provides an interface for constructing options that customize batch writing behavior, such as batch size and retry. ```python from influxdb_client_3 import WriteOptions write_options = WriteOptions(batch_size=500, flush_interval=10_000, jitter_interval=2_000, retry_interval=5_000, max_retries=5, max_retry_delay=30_000, exponential_base=2) ``` See how to [use batch options for writing data](#example-use-batch-options-when-writing-file-data). ### Parameters * **`batch_size`**: Default is `1000`. * **`flush_interval`**: Default is `1000`. * **`jitter_interval`**: Default is `0`. * **`retry_interval`**: Default is `5000`. * **`max_retries`**: Default is `5`. * **`max_retry_delay`**: Default is `125000`. * **`max_retry_time`**: Default is `180000`. * **`exponential_base`**: Default is `2`. * **`max_close_wait`**: Default is `300000`. * **`write_scheduler`**: Default is `ThreadPoolScheduler(max_workers=1)`. ## Functions * [influxdb\_client\_3.write\_client\_options](#function-write_client_optionskwargs) * [influxdb\_client\_3.flight\_client\_options](#function-flight_client_optionskwargs) ### Function write\_client\_options(\*\*kwargs) Returns a `dict` with the specified write client options. #### Parameters The function takes the following keyword arguments: * **`write_options`** ([`WriteOptions`](#class-writeoptions)): Specifies whether the client writes data using synchronous mode or batching mode. If using batching mode, the client uses the specified batching options. * **`point_settings`** (dict): Default tags that the client will add to each point when writing the data to InfluxDB. * **`success_callback`** (callable): If using batching mode, a function to call after data is written successfully to InfluxDB (HTTP status `204`) * **`error_callback`** (callable): if using batching mode, a function to call if data is not written successfully (the response has a non-`204` HTTP status) * **`retry_callback`** (callable): if using batching mode, a function to call if the request is a retry (using batching mode) and data is not written successfully #### Example: instantiate options for batch writing ```python from influxdb_client_3 import write_client_options, WriteOptions from influxdb_client_3.write_client.client.write_api import WriteType def success(): print("Success") def error(): print("Error") def retry(): print("Retry error") write_options = WriteOptions() wco = write_client_options(success_callback=success, error_callback=error, retry_callback=retry, write_options=write_options) assert wco['success_callback'] assert wco['error_callback'] assert wco['retry_callback'] assert wco['write_options'].write_type == WriteType.batching ``` #### Example: instantiate options for synchronous writing ```python from influxdb_client_3 import write_client_options, SYNCHRONOUS from influxdb_client_3.write_client.client.write_api import WriteType wco = write_client_options(write_options=SYNCHRONOUS) assert wco['write_options'].write_type == WriteType.synchronous ``` ### Function flight\_client\_options(\*\*kwargs) Returns a `dict` with the specified [FlightClient](https://arrow.apache.org/docs/python/generated/pyarrow.flight.FlightClient.html) parameters. #### Parameters * `kwargs`: keyword arguments for [`pyarrow.flight.FlightClient`](https://arrow.apache.org/docs/python/generated/pyarrow.flight.FlightClient.html) parameters #### Example: specify the root certificate path ```python from influxdb_client_3 import InfluxDBClient3, flight_client_options import certifi fh = open(certifi.where(), "r") cert = fh.read() fh.close() client = InfluxDBClient3(host=f"localhost:8181", database=f"DATABASE_NAME", token=f"DATABASE_TOKEN", flight_client_options=flight_client_options(tls_root_certs=cert)) ``` Replace the following: * `DATABASE_NAME`: the name of your InfluxDB 3 Enterprise [database](/influxdb3/enterprise/admin/databases/) * `DATABASE_TOKEN`: an InfluxDB 3 Enterprise [database token](/influxdb3/enterprise/admin/tokens/#database-tokens)with read permission on the specified database ## Constants * `influxdb_client_3.SYNCHRONOUS`: Represents synchronous write mode * `influxdb_client_3.WritePrecision`: Enum class that represents write precision ## Exceptions * `influxdb_client_3.InfluxDBError`: Exception class raised for InfluxDB-related errors #### Related [Flight API](/influxdb3/enterprise/tags/flight-api/)[python](/influxdb3/enterprise/tags/python/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # influxdb3-python release notes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/python/release-notes/ ## v0.21.0 ### Bug Fixes 1. [#234](https://github.com/InfluxCommunity/influxdb3-python/pull/234): Add support for connecting to InfluxDB servers using IPv6 addresses. * IPv6 addresses in server URLs must be enclosed in square brackets, for example, http://[2001:db8::1]:8086. * IPv6 zone identifiers are not currently supported. ### Breaking Changes 1. [\#217](https://github.com/InfluxCommunity/influxdb3-python/pull/217): Makes the writing API simpler and more consistent with other v3 clients. * Removes unused `InfluxLoggingHandler` class. * `WriteApi` now handles all functions to write to InfluxDB. * The following internal classes have been removed: * `Configuration` * `ApiClient` * `WriteService` * `InfluxDBClient` * The following internal classes have been refactored: * `write_client._sync.RestClient` - this class is now responsible for low-level handling of transport requests. End users should not need to use it directly. * `write_client.client.WriteApi` - all internal settings needed for writing are now encapsulated within this Api. * Refactors Multiprocessing helper class. * **Migration Guidance** * `InfluxDBClient3` constructor now has additional parameters for configuring the internal `WriteApi` and `RestClient`. * `auth_scheme` - string - the token authentation scheme, typically `Token` or `Bearer`. * `enable_gzip` - boolean - whether or not to compress data in writing payloads. * `gzip_threshold` - int - minimum payload size to trigger gzip compression when `enable_gzip` is True. * `point_settings` - PointSettings - default settings used when writing Points, primarily used to wrap default tags to be added to Point data. * `debug` - boolean - toggle debug level logging. * Any clients explicitly using the `Configuration`, `ApiClient`, `WriteService` or *legacy* `InfluxDBClient` classes, will need to migrate their settings to the `InfluxDBClient3` constructor. 2. [\#222](https://github.com/InfluxCommunity/influxdb3-python/pull/222): Refactor `MultiprocessingWriter` class. * New Process will be created by using `DefaultContext.Process(target)` to prevent erratic crashes, handle cross-platform code behavior safely, and coordinate complex resource sharing. * Users can now choose one of the start methods `fork`, `spawn` or `forkserver` when creating new Process. The default will be `spawn`. * Users can set time to live for the child process through `process_ttl` parameter, the default value will be 300 seconds. ## v0.20.0 ### Features 1. [#208](https://github.com/InfluxCommunity/influxdb3-python/pull/208): Add `influx3 query` CLI support for executing SQL/InfluxQL queries with JSON/JSONL/CSV/pretty output, including module execution via `python -m influxdb_client_3`. 2. [#213](https://github.com/InfluxCommunity/influxdb3-python/pull/213): Add partial write support and default writes to the V2 API endpoint. See [Partial writes](https://docs.influxdata.com/influxdb3/core/write-data/http-api/v3-write-lp/#partial-writes) for more.`no_sync` requires `use_v2_api=False`; `accept_partial` applies only to V3 API endpoint writes. ### Others 1. [#215](https://github.com/InfluxCommunity/influxdb3-python/pull/215): * Remove cookie/session-based signin/signout support from the writing client. * Remove basic authentication with a username and password. ## v0.19.0 ### Features 1. [#198](https://github.com/InfluxCommunity/influxdb3-python/pull/198): Support custom tag order via `tag_order` write option. See [Sort tags by priority](https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/schema-design/#sort-tags-by-query-priority) for more. 2. [#202](https://github.com/InfluxCommunity/influxdb3-python/pull/202): Add escape for field keys when serializing to line protocol in `PolarsDataframeSerializer`. ## v0.18.0 ### Features 1. [#196](https://github.com/InfluxCommunity/influxdb3-python/pull/196): Support passing middleware functions to the Flight client. ### Bug Fixes 1. [#194](https://github.com/InfluxCommunity/influxdb3-python/pull/194): Fix `InfluxDBClient3.write_file()` and `InfluxDBClient3.write_dataframe()` fail with batching mode. 2. [#197](https://github.com/InfluxCommunity/influxdb3-python/pull/197): InfluxDB 3 Core/Enterprise write errors details handling. ## v0.17.0 ### Features 1. [#177](https://github.com/InfluxCommunity/influxdb3-python/pull/177): Add dedicated DataFrame methods for improved usability and type safety: * `write_dataframe()`: New method for writing pandas and polars DataFrames with explicit parameters (`measurement`, `timestamp_column`, `tags`, `timestamp_timezone`). * `query_dataframe()`: New method for querying data directly to a pandas or polars DataFrame via the `frame_type` parameter. * Updated README with clear examples for DataFrame operations. 2. [#179](https://github.com/InfluxCommunity/influxdb3-python/pull/179): Add option to disable gRPC response compression for Flight queries: * `disable_grpc_compression` parameter in `InfluxDBClient3` constructor * `INFLUX_DISABLE_GRPC_COMPRESSION` environment variable support in `from_env()` 3. [#180](https://github.com/InfluxCommunity/influxdb3-python/pull/180): Add `flush()` method to `InfluxDBClient3`: * Allows flushing the write buffer without closing the client when using batching mode. * Enables applications to ensure data is written before querying, while keeping the client open for further operations. 4. [#185](https://github.com/InfluxCommunity/influxdb3-python/pull/185): * Add API Reference page on document page * Remove old document link. * Add link to document page in README.md ### Bug Fixes 1. [#177](https://github.com/InfluxCommunity/influxdb3-python/pull/177): Fix `TypeError` when writing DataFrames. Serializer-specific kwargs (e.g., `data_frame_measurement_name`) are now filtered before being passed to the HTTP layer. * CI updates ## v0.16.0 ### Features 1. [#158](https://github.com/InfluxCommunity/influxdb3-python/pull/158) Improved parameters for setting timeouts * `InfluxDB3Client()` constructor now directly specifies * `write_timeout` - timeout in milliseconds to be used when writing data. * `query_timeout` - timeout in milliseconds to be used when querying data. * Timeouts can be specifically overridden in direct method calls. * `client.write()` now propagates the argument `_request_timeout` as an `int` in milliseconds on the call stack even in batching mode. * `client.query()` now propagates the argument `timeout` as a `float` in seconds on the call stack. * CI updates ## v0.15.0 ### Features 1. [#146](https://github.com/InfluxCommunity/influxdb3-python/pull/146): Add function to get InfluxDB version. 2. [#149](https://github.com/InfluxCommunity/influxdb3-python/pull/149): Run integration tests against a locally started InfluxDB 3 Core server. ## v0.14.0 ### Features 1. [#141](https://github.com/InfluxCommunity/influxdb3-python/pull/141) Move “setuptools” package to build dependency. 2. [#142](https://github.com/InfluxCommunity/influxdb3-python/pull/142): Support fast writes without waiting for WAL persistence: * New write option (`WriteOptions.no_sync`) added: `True` value means faster write but without the confirmation that the data was persisted. Default value: `False`. * **Supported by self-managed InfluxDB 3 Core and Enterprise servers only!** * Also configurable via environment variable (`INFLUX_WRITE_NO_SYNC`). * Long precision string values added from v3 HTTP API: `"nanosecond"`, `"microsecond"`, `"millisecond"`,`"second"` ( in addition to the existing `"ns"`, `"us"`, `"ms"`, `"s"`). 3. [#145](https://github.com/InfluxCommunity/influxdb3-python/pull/145): Improve the document wording for README.md ## v0.13.0 ### Features 1. [#130](https://github.com/InfluxCommunity/influxdb3-python/pull/130): Remove org parameters from the example code because It is not mandatory in Influxdb3 2. [#139](https://github.com/InfluxCommunity/influxdb3-python/pull/139): Supports environment variables with the same name as other clients 3. [#140](https://github.com/InfluxCommunity/influxdb3-python/pull/140): Query api will throw `InfluxdbClientQueryError` when receiving `ArrowException` from gRPC servers ## v0.12.0 ### Features 1. [#123](https://github.com/InfluxCommunity/influxdb3-python/pull/123): Introduces `query_async()` method. From this release the client now has a `query_async()` method that takes advantage of asyncio’s event loop to run query calls in their own executor. For example: ``` table = await client.query_async(query) ``` ### Bug Fixes 1. [#121](https://github.com/InfluxCommunity/influxdb3-python/pull/121): Fix use of arguments `verify_ssl` and `ssl_ca_cert` in `QueryApi`. ## v0.11.0 ### Bug Fixes 1. [#119](https://github.com/InfluxCommunity/influxdb3-python/pull/119): Fix use of `proxy` argument in client and query\_api to use in channel solution for GRPC proxy. ## v0.10.0 ### Bug Fixes 1. [#113](https://github.com/InfluxCommunity/influxdb3-python/pull/113): Fix import error of `PolarsDataframeSerializer` in batching mode ## v0.9.0 ### Features 1. [#108](https://github.com/InfluxCommunity/influxdb3-python/pull/108): Better expose access to response headers in `InfluxDBError`. Example `handle_http_error` added. 2. [#112](https://github.com/InfluxCommunity/influxdb3-python/pull/112): Update batching examples, add integration tests of batching. ### Bug Fixes 1. [#107](https://github.com/InfluxCommunity/influxdb3-python/pull/107): Missing `py.typed` in distribution package 2. [#111](https://github.com/InfluxCommunity/influxdb3-python/pull/111): Reduce log level of disposal of batch processor to DEBUG ## v0.8.0 ### Features 1. [#101](https://github.com/InfluxCommunity/influxdb3-python/pull/101): Add support for InfluxDB Edge (OSS) authentication ### Bug Fixes 1. [#100](https://github.com/InfluxCommunity/influxdb3-python/pull/100): InfluxDB Edge (OSS) error handling 2. [#105](https://github.com/InfluxCommunity/influxdb3-python/pull/105): Importing Polars serialization module ## v0.7.0 ### Bug Fixes 1. [#95](https://github.com/InfluxCommunity/influxdb3-python/pull/95): `Polars` is optional dependency 2. [#99](https://github.com/InfluxCommunity/influxdb3-python/pull/99): Skip infinite values during serialization to line protocol ## v0.6.1 ### Bug Fixes 1. [#98](https://github.com/InfluxCommunity/influxdb3-python/pull/98): Missing declaration for `query` module ## v0.6.0 ### Features 1. [#89](https://github.com/InfluxCommunity/influxdb3-python/pull/89): Use `datetime.fromisoformat` over `dateutil.parse` in Python 3.11+ 2. [#92](https://github.com/InfluxCommunity/influxdb3-python/pull/92): Update `user-agent` header value to `influxdb3-python/{VERSION}` and add it to queries as well. ### Bug Fixes 1. [#86](https://github.com/InfluxCommunity/influxdb3-python/pull/86): Refactor to `timezone` specific `datetime` helpers to avoid use deprecated functions ## v0.5.0 ### Features 1. [#88](https://github.com/InfluxCommunity/influxdb3-python/pull/88): Add support for named query parameters: ``` from influxdb_client_3 import InfluxDBClient3 with InfluxDBClient3(host="https://us-east-1-1.aws.cloud2.influxdata.com", token="my-token", database="my-database") as client: table = client.query("select * from cpu where host=$host", query_parameters={"host": "server01"}) print(table.to_pandas()) ``` ### Bug Fixes 1. [#87](https://github.com/InfluxCommunity/influxdb3-python/pull/87): Fix examples to use `write_options` instead of the object name `WriteOptions` ### Others 1. [#84](https://github.com/InfluxCommunity/influxdb3-python/pull/84): Enable packaging type information - `py.typed` ## v0.4.0 ### Bugfix 1. [#77](https://github.com/InfluxCommunity/influxdb3-python/pull/77): Support using pandas nullable types ### Others 1. [#80](https://github.com/InfluxCommunity/influxdb3-python/pull/80): Integrate code style check into CI --- # Rust client library for InfluxDB 3 Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/rust/ The InfluxDB 3 [`influxdb3-rust` Rust client library](https://github.com/InfluxCommunity/influxdb3-rust) integrates with Rust applications to write and query data stored in InfluxDB 3 Enterprise. **Latest version:**[v0.3.0](/influxdb3/enterprise/reference/client-libraries/v3/rust/release-notes/)(2026-08-27) The documentation for this client library is available on [docs.rs](https://docs.rs/influxdb3-client/latest/influxdb3_client/). [InfluxDB 3 Rust client library](https://github.com/InfluxCommunity/influxdb3-rust) [Flight client](/influxdb3/enterprise/tags/flight-client/)[Rust](/influxdb3/enterprise/tags/rust/)[gRPC](/influxdb3/enterprise/tags/grpc/)[SQL](/influxdb3/enterprise/tags/sql/)[Flight SQL](/influxdb3/enterprise/tags/flight-sql/)[client libraries](/influxdb3/enterprise/tags/client-libraries/) --- # influxdb3-rust release notes Source: https://docs.influxdata.com/influxdb3/enterprise/reference/client-libraries/v3/rust/release-notes/ ## v0.3.0 > > > ⚠️ This release requires Rust 1.91 or later. > > ### Performance 1. [#30](https://github.com/InfluxCommunity/influxdb3-rust/pull/30): Rewrite the polars DataFrame serializer with typed column iterators and the shared ryu/itoa line-protocol writers; \~2.5x faster on wide numeric frames. Floats on the DataFrame path now format via ryu, matching `Point`, so extreme magnitudes emit exponent notation instead of expanded decimals. ### Dependencies 1. [#20](https://github.com/InfluxCommunity/influxdb3-rust/pull/20): Upgrade Arrow dependencies from version 58 to 59. 2. [#22](https://github.com/InfluxCommunity/influxdb3-rust/pull/22): Upgrade Polars from version 0.53 to 0.54. * CI updates ## v0.2.0 ### Features 1. [#12](https://github.com/InfluxCommunity/influxdb3-rust/pull/12): Expand client configuration support for environment variables and connection strings. Add auth scheme and write option support. Preserve explicit ports and strip userinfo from normalized hosts. Remove the legacy `bucket` and `INFLUX_BUCKET` aliases in favor of `database` and `INFLUX_DATABASE`. 2. [#19](https://github.com/InfluxCommunity/influxdb3-rust/pull/19): Default writes to the V2 API endpoint. Add builder methods for write defaults.`no_sync` requires `use_v2_api=false` to write to the V3 API endpoint.`accept_partial` applies only when writes are sent to the V3 API endpoint and is ignored otherwise. ### Bug Fixes 1. [#13](https://github.com/InfluxCommunity/influxdb3-rust/pull/13): Improve Arrow query result type support. Unsupported Arrow types now produce `Error::UnsupportedArrowType` instead of `Null`. ### Dependencies 1. [#18](https://github.com/InfluxCommunity/influxdb3-rust/pull/18): Upgrade Arrow dependencies to version 58 and require Rust 1.89 or later. ## v0.1.0 Initial release. ### Features * Async client for InfluxDB 3 Core and Enterprise over HTTP (writes) and Arrow Flight (queries). * Write API: a builder accepting line-protocol strings, `Vec`, and (with the `polars` feature) a DataFrame. Options for timestamp precision, batching, in-flight concurrency, default tags, gzip, and WAL no-sync. * Query API: SQL and InfluxQL, parameterised queries, row iteration, and streaming of results too large to hold in memory. * Automatic retries with exponential backoff and full jitter for transient failures (transport errors, `429`, `5xx`), honouring `Retry-After`. Configurable per client or per request. * Partial-write error reporting with per-line detail. * Optional `polars` feature: DataFrame writes and query-to-DataFrame conversion. ### Notes * Retries are enabled by default (`max_retries = 3`). Use `.no_retry()` or a custom `RetryConfig` to change this. --- # InfluxDB 3 Enterprise configuration options Source: https://docs.influxdata.com/influxdb3/enterprise/reference/config-options/ InfluxDB 3 Enterprise lets you customize your configuration by using`influxdb3` command options or by setting environment variables. ## Configure your server Pass configuration options using either command options or environment variables. Command options take precedence over environment variables. ### TOML configuration files DEB and RPM installs include a TOML configuration file at`/etc/influxdb3/influxdb3-enterprise.conf`. The systemd launcher reads this file and converts each entry to an`INFLUXDB3_*` environment variable (the pattern is `INFLUXDB3_`with hyphens replaced by underscores and uppercased) before invoking`influxdb3 serve`. `influxdb3 serve` does not yet natively read TOML; the file’s effect is applied through the environment variables the launcher sets. CLI flags passed to `influxdb3 serve` still override values defined in the TOML file because they override the environment variables that file becomes. For day-to-day systemd operation — including the edit-and-restart workflow for applying changes — see[Run as a system service (Linux)](/influxdb3/enterprise/install/#run-as-a-system-service-linux). ### Global vs serve-specific options Some options are **global** (specified before the command) while others are **serve-specific** (specified after `serve`): * **Global options**: Apply to the `influxdb3` CLI itself (for example, `--num-io-threads`) * **Serve options**: Apply only to the `serve` command (for example, `--node-id`, `--object-store`, `--verbose`) #### Example command with global and serve-specific options ```sh influxdb3 --num-io-threads=4 serve \ --node-id node0 \ --cluster-id cluster0 \ --license-email example@email.com \ --object-store file \ --data-dir ~/.influxdb3 \ --verbose \ --log-filter info ``` ##### Example environment variables ```sh export INFLUXDB3_LICENSE_EMAIL=example@email.com export INFLUXDB3_CLUSTER_ID=cluster0 export INFLUXDB3_NODE_ID=my-node export INFLUXDB3_OBJECT_STORE=file export INFLUXDB3_DATA_DIR=~/.influxdb3 export INFLUXDB3_LOG_FILTER=info influxdb3 serve ``` ### Deprecated names Many options and environment variables were renamed in InfluxDB 3 Enterprise 3.11.0, with the old names kept as deprecated aliases. Option tables on this page list the pre-3.11 names where they exist. For the full old-to-new tables and compatibility rules, see[Configuration name changes in 3.11](#name-changes-in-3-11). ### Size option values Options that accept a size value require an explicit unit: * Append a unit suffix `b`, `kb`, `mb`, `gb`, or `tb`(case-insensitive)—for example, `10mb` or `8GB`. * Where noted, specify a percentage of total available memory—for example, `20%`. Bare numbers are rejected to avoid a silent change in meaning—historically, a bare number meant megabytes for some options and bytes for others. For compatibility with pre-3.11 configurations: * The deprecated `--parquet-mem-cache-size`, `--exec-mem-pool-bytes`, and`--force-snapshot-mem-threshold` options (and their environment variables) accept their pre-3.11 value format—a bare number means megabytes—with a startup deprecation warning. * `--max-http-request-size` (name unchanged) accepts a bare number as bytes, its pre-3.11 meaning, with a startup warning. ## Global configuration options The following options apply to the `influxdb3` CLI globally and must be specified **before** any subcommand (for example, `serve`): ### num-io-threads Sets the number of threads allocated to the IO runtime thread pool. IO threads handle HTTP request serving, line protocol parsing, and file operations. > [!Important] > `--num-io-threads` is a **global option** that must be specified before the `serve` command. **Default:** `2` on Parquet-engine clusters; the licensed core count on clusters running the upgraded storage engine (the default for new clusters). Values above the licensed core count are capped with a startup warning. ```bash # Set IO threads (global option before serve) influxdb3 --num-io-threads=8 serve --node-id=node0 --object-store=file ``` For detailed information about thread allocation, see the [Resource Limits](#resource-limits) section. | influxdb3 option | Environment variable | |------------------|--------------------------| |`--num-io-threads`|`INFLUXDB3_NUM_IO_THREADS`| ## Server configuration options > [!Note] > #### Storage engines and option visibility > > New InfluxDB 3 Enterprise clusters default to the upgraded storage engine. > Clusters that started on 3.10 or earlier keep the Parquet engine until you > run the storage engine upgrade by restarting the cluster with[`--upgrade-pacha-tree`](#upgrade-pacha-tree). > > The default `influxdb3 serve --help` output shows a single unified > storage-engine section. > Parquet-specific tuning options (for example, `compaction-*`, `gen1-*`,`wal-files-per-snapshot`, and the `num-*` limits) no longer appear in the > default help output but remain functional for clusters that have not run > the storage engine upgrade. > Use `influxdb3 serve --help-all` to list every option. > > If you explicitly set an option (on the command line or through an > environment variable) that has no effect on the active storage engine, > the server logs one aggregated startup warning naming the ignored options. > After the storage engine upgrade completes, this includes long-standing > Parquet options that no longer apply. The warnings appear on the next > server start following completion of the upgrade–use them as a checklist > for cleaning up your configuration. * [General](#general) * [Licensing](#licensing) * [Security](#security) * [AWS](#aws) * [Google Cloud Service](#google-cloud-service) * [Microsoft Azure](#microsoft-azure) * [Object Storage](#object-storage) * [Logs](#logs) * [Traces](#traces) * [DataFusion](#datafusion) * [HTTP](#http) * [Memory](#memory) * [Write-Ahead Log (WAL)](#write-ahead-log-wal) * [Compaction](#compaction) * [Caching](#caching) * [Processing Engine](#processing-engine) * [Bulk Import](#bulk-import) * [Cluster Management](#cluster-management) * [Web UI](#web-ui) * [Resource Limits](#resource-limits) * [Data Lifecycle Management](#data-lifecycle-management) * [Telemetry](#telemetry) * [TCP Listeners](#tcp-listeners) ### General * [cluster-id](#cluster-id) * [data-dir](#data-dir) * [mode](#mode) * [node-id](#node-id) * [node-id-from-env](#node-id-from-env) * [object-store](#object-store) * [query-file-limit](#query-file-limit) * [upgrade-pacha-tree](#upgrade-pacha-tree) #### cluster-id Specifies the cluster identifier that prefixes the object store path for the Enterprise Catalog. This value must be different than the [`--node-id`](#node-id) value. |influxdb3 serve option| Environment variables | |----------------------|-------------------------------------------------------------------------------------------------------| | `--cluster-id` |`INFLUXDB3_CLUSTER_ID`
`INFLUXDB3_ENTERPRISE_CLUSTER_ID` ([pre-3.11 name](#name-changes-in-3-11))| #### data-dir For the `file` object store, defines the location InfluxDB 3 Enterprise uses to store files locally. Required when using the `file` [object store](#object-store). Use a separate `--data-dir` for each `influxdb3 serve` process. A multi-node cluster requires a shared object store such as Amazon S3; don’t use the `file` object store. See [Create a multi-node cluster](/influxdb3/enterprise/get-started/multi-server/#create-an-object-store). |influxdb3 serve option| Environment variables | |----------------------|--------------------------------------------------------------------------------------| | `--data-dir` |`INFLUXDB3_DATA_DIR`
`INFLUXDB3_DB_DIR` ([pre-3.11 name](#name-changes-in-3-11))| #### mode Sets the mode to start the server in, allowing you to create specialized nodes in a distributed cluster. This option supports the following values: * `all` *(default)*: Enables all server modes * `ingest`: Enables only data ingest capabilities * `query`: Enables only query capabilities * `compact`: Enables only compaction processes * `process`: Runs the [Processing Engine](/influxdb3/enterprise/reference/processing-engine/), which runs plugins for triggers. A node in `process` mode alone doesn’t accept writes or answer data queries. Setting [`--plugin-dir`](#plugin-dir) adds `process` mode regardless of `--mode`. Conversely, `--mode=process` requires `--plugin-dir`. In a cluster, run `process` alone on a dedicated node. See [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). * `webui` *(3.11+)*: Serves the [InfluxDB 3 Explorer](/influxdb3/enterprise/admin/explorer-ui/) web UI from the server process as a WebAssembly (WASM) guest. `all` doesn’t include `webui`, so name `webui` explicitly–for example, `--mode all,webui`. `webui` mode requires [`--webui-session-secret`](#webui-session-secret). Set [`--plugin-dir`](#plugin-dir) as well to use the plugin features in Explorer. You can specify multiple modes using a comma-delimited list (for example, `ingest,query`). **Default:** `all` > [!Important] > **Thread allocation for different modes:** > > * **Ingest mode**: Benefits from additional IO threads for line protocol parsing. For high-throughput > scenarios with multiple concurrent writers, consider increasing [`--num-io-threads`](#num-io-threads) (global option) > to 8-16+ to optimize performance. DataFusion threads are still needed for snapshot operations. > > * **Query mode**: Benefits from maximizing DataFusion threads. Use most available cores for DataFusion > with minimal IO threads (2-4). > > * **Compact mode**: Primarily uses DataFusion threads for sort/dedupe operations. > > * **All mode**: Requires balanced thread allocation based on your workload mix. **Example configurations:** ```bash # High-throughput ingest node (32 cores) influxdb3 --num-io-threads=12 serve --mode=ingest --datafusion-num-threads=20 # Query-optimized node (32 cores) influxdb3 --num-io-threads=4 serve --mode=query --datafusion-num-threads=28 # Balanced all-in-one (32 cores) influxdb3 --num-io-threads=6 serve --mode=all --datafusion-num-threads=26 ``` |influxdb3 serve option| Environment variables | |----------------------|-------------------------------------------------------------------------------------------| | `--mode` |`INFLUXDB3_MODE`
`INFLUXDB3_ENTERPRISE_MODE` ([pre-3.11 name](#name-changes-in-3-11))| #### node-id Specifies the node identifier used as a prefix in all object store file paths. This should be unique for any hosts sharing the same object store configuration–for example, the same bucket. |influxdb3 serve option| Environment variables | |----------------------|-----------------------------------------------------------------------------------------------------| | `--node-id` |`INFLUXDB3_NODE_ID`
`INFLUXDB3_NODE_IDENTIFIER_PREFIX` ([pre-3.11 name](#name-changes-in-3-11))| #### node-id-from-env Specifies the node identifier used as a prefix in all object store file paths. Takes the name of an environment variable as an argument and uses the value of that environment variable as the node identifier. This option cannot be used with the `--node-id` option. |influxdb3 serve option| Environment variables | |----------------------|----------------------------------------------------------------------------------------------------------------| | `--node-id-from-env` |`INFLUXDB3_NODE_ID_FROM_ENV`
`INFLUXDB3_NODE_IDENTIFIER_FROM_ENV` ([pre-3.11 name](#name-changes-in-3-11))| ##### Example using –node-id-from-env ```bash export DATABASE_NODE=node0 && influxdb3 serve \ --node-id-from-env DATABASE_NODE \ --cluster-id cluster0 \ --object-store file \ --data-dir ~/.influxdb3/data ``` #### object-store Specifies which object storage to use to store Parquet files. This option supports the following values: * `memory`: Effectively no object persistence * `memory-throttled`: Like `memory` but with latency and throughput that somewhat resembles a cloud object store * `file`: Stores objects in the local filesystem (must also set `--data-dir`) * `s3`: Amazon S3 (must also set `--bucket`, `--aws-access-key-id`, `--aws-secret-access-key`, and possibly `--aws-default-region`) * `google`: Google Cloud Storage (must also set `--bucket` and `--google-service-account`) * `azure`: Microsoft Azure blob storage (must also set `--bucket`, `--azure-storage-account`, and `--azure-storage-access-key`) |influxdb3 serve option| Environment variable | |----------------------|------------------------| | `--object-store` |`INFLUXDB3_OBJECT_STORE`| #### upgrade-pacha-tree Migrates the cluster’s existing Parquet data to the upgraded storage engine. New clusters default to the upgraded storage engine. Clusters that started on 3.10 or earlier keep the Parquet engine until you run the storage engine upgrade by restarting the cluster with`--upgrade-pacha-tree`. New clusters do not need this flag. For the upgrade procedure, see[Upgrade from Parquet](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet). **Default:** `false` > [!Note] > `--use-pacha-tree` (environment variable `INFLUXDB3_USE_PACHA_TREE`; legacy`INFLUXDB3_ENTERPRISE_USE_PACHA_TREE`) is deprecated. > It is still accepted and keeps its previous behavior—on an existing Parquet > cluster it starts the same migration as `--upgrade-pacha-tree`—but the > server logs a deprecation warning at startup. |influxdb3 serve option| Environment variable | |----------------------|------------------------------| |`--upgrade-pacha-tree`|`INFLUXDB3_UPGRADE_PACHA_TREE`| ### Licensing #### license-email Specifies the email address to associate with your InfluxDB 3 Enterprise license and automatically responds to the interactive email prompt when the server starts. This option is mutually exclusive with [license-file](#license-file). |influxdb3 serve option| Environment variables | |----------------------|-------------------------------------------------------------------------------------------------------------| | `--license-email` |`INFLUXDB3_LICENSE_EMAIL`
`INFLUXDB3_ENTERPRISE_LICENSE_EMAIL` ([pre-3.11 name](#name-changes-in-3-11))| #### license-file Specifies the path to a license file for InfluxDB 3 Enterprise. When provided, the license file’s contents are used instead of requesting a new license. This option is mutually exclusive with [license-email](#license-email). |influxdb3 serve option| Environment variables | |----------------------|-----------------------------------------------------------------------------------------------------------| | `--license-file` |`INFLUXDB3_LICENSE_FILE`
`INFLUXDB3_ENTERPRISE_LICENSE_FILE` ([pre-3.11 name](#name-changes-in-3-11))| #### license-type Specifies the type of InfluxDB 3 Enterprise license to use and bypasses the interactive license prompt. Provide one of the following license types: * `home` * `trial` * `commercial` |influxdb3 serve option| Environment variables | |----------------------|-----------------------------------------------------------------------------------------------------------| | `--license-type` |`INFLUXDB3_LICENSE_TYPE`
`INFLUXDB3_ENTERPRISE_LICENSE_TYPE` ([pre-3.11 name](#name-changes-in-3-11))| ### Security * [tls-key](#tls-key) * [tls-cert](#tls-cert) * [tls-minimum-versions](#tls-minimum-version) * [without-auth](#without-auth) * [disable-authz](#disable-authz) * [admin-token-recovery-http-bind](#admin-token-recovery-http-bind) * [admin-token-file](#admin-token-file) * [permission-tokens-file](#permission-tokens-file) * [user-auth-type](#user-auth-type) * [without-user-auth](#without-user-auth) * [jwt-key-id](#jwt-key-id) * [jwt-private-key](#jwt-private-key) * [jwt-issuer](#jwt-issuer) * [jwt-default-ttl-seconds](#jwt-default-ttl-seconds) * [oauth-issuer](#oauth-issuer) * [oauth-audience](#oauth-audience) * [oauth-client-id](#oauth-client-id) * [oauth-scopes](#oauth-scopes) * [rbac-authoring-disabled](#rbac-authoring-disabled) #### tls-key The path to a key file for TLS to be enabled. |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--tls-key` |`INFLUXDB3_TLS_KEY` | #### tls-cert The path to a cert file for TLS to be enabled. |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--tls-cert` |`INFLUXDB3_TLS_CERT`| #### tls-minimum-version The minimum version for TLS. Valid values are `tls-1.2` or `tls-1.3`. Default is `tls-1.2`. |influxdb3 serve option | Environment variable | |-----------------------|-------------------------------| |`--tls-minimum-version`|`INFLUXDB3_TLS_MINIMUM_VERSION`| #### without-auth Disables authentication for all server actions (CLI commands and API requests). The server processes all requests without requiring tokens or authentication. |influxdb3 serve option| Environment variables | |----------------------|------------------------------------------------------------------------------------------------------| | `--without-auth` |`INFLUXDB3_WITHOUT_AUTH`
`INFLUXDB3_START_WITHOUT_AUTH` ([pre-3.11 name](#name-changes-in-3-11))| #### disable-authz Optionally disable authz by passing in a comma separated list of resources. Valid values are `health`, `ping`, and `metrics`. |influxdb3 serve option| Environment variable | |----------------------|-------------------------| | `--disable-authz` |`INFLUXDB3_DISABLE_AUTHZ`| #### admin-token-recovery-http-bind Enables an admin token recovery HTTP server on a separate port. This server allows regenerating lost admin tokens without existing authentication. The server automatically shuts down after a successful token regeneration. > [!Warning] > This option creates an unauthenticated endpoint that can regenerate admin tokens. > Only use this when you have lost access to your admin token and ensure the > server is only accessible from trusted networks. **Default:** `127.0.0.1:8182` (when enabled) | influxdb3 serve option | Environment variable | |----------------------------------|-----------------------------------------------| |`--admin-token-recovery-http-bind`|`INFLUXDB3_ADMIN_TOKEN_RECOVERY_HTTP_BIND_ADDR`| ##### Example usage ```bash # Start server with recovery endpoint influxdb3 serve --admin-token-recovery-http-bind # In another terminal, regenerate the admin token influxdb3 create token --admin --regenerate --host http://127.0.0.1:8182 ``` #### admin-token-file Specifies an offline admin token file to use if no tokens exist when the server starts. Once started, you can interact with the server using the provided token. Offline admin tokens are designed to help with automated deployments. |influxdb3 serve option| Environment variable | |----------------------|----------------------------| | `--admin-token-file` |`INFLUXDB3_ADMIN_TOKEN_FILE`| Offline admin tokens are defined in a JSON-formatted file. Use the following command to generate an offline admin token file: ```bash influxdb3 create token --admin \ --name "example-admin-token" \ --expiry 1d \ --offline \ --output-file ./path/to/admin-token.json ``` [](#view-example-offline-admin-token-file) View example offline admin token file ```json { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "example-admin-token", "expiry_millis": 1756400061529 } ``` ##### Example usage ```bash # Generate an admin token offline influxdb3 create token \ --admin \ --name "example-admin-token" \ --expiry 1d \ --offline \ --output-file ./path/to/admin-token.json # Start InfluxDB 3 Enterprise using the generated token influxdb3 serve --admin-token-file ./path/to/admin-token.json ``` #### permission-tokens-file Specifies an offline permission (resource) tokens file to use if no resource tokens exist when the server starts. Once started, you can interact with the server using the provided tokens. Offline permission tokens are designed to help with automated deployments. | influxdb3 serve option | Environment variable | |--------------------------|----------------------------------| |`--permission-tokens-file`|`INFLUXDB3_PERMISSION_TOKENS_FILE`| Multiple tokens with database-level permissions can be defined. You can also specify databases to create at startup. Use the a command similar to the following to generate an offline permission token file: ```bash influxdb3 create token \ --name "example-token" \ --permission "db:db1,db2:read,write" \ --permission "db:db3:read" \ --expiry 1d \ --offline \ --create-databases db1,db2 \ --output-file ./path/to/tokens.json ``` [](#view-example-offline-permission-tokens-file) View example offline permission tokens file ```json { "create_databases": [ "db1", "db2", "d3" ], "tokens": [ { "token": "apiv3_0XXXX-xxxXxXxxxXX_OxxxX...", "name": "example-token", "expiry_millis": 1756400061529, "permissions": [ "db:db1,db2:read,write", "db:db3:read" ] } ] } ``` > [!Note] > If you write a new offline permission token to an existing permission token file, > the command appends the new token to the existing output file. ##### Example usage ```bash # Generate an admin token offline influxdb3 create token \ --name "example-token" \ --permission "db:db1,db2:read,write" \ --permission "db:db3:read" \ --expiry 1d \ --offline \ --create-databases db1,db2 \ --output-file ./path/to/tokens.json # Start InfluxDB 3 Enterprise using the generated token influxdb3 serve --permission-tokens-file ./path/to/tokens.json ``` #### user-auth-type Specifies which user authentication methods InfluxDB 3 Enterprise enables, as a comma-separated list of `basic` (username and password) and/or `oauth`, or`none` to disable user authentication.`none` can’t be combined with other values.[`--without-auth`](#without-auth) overrides this option and disables all authentication. This option supports the following values: * `none` *(default)*: Disable user authentication. * `basic`: Enable username and password authentication. Requires [`--jwt-key-id`](#jwt-key-id) and[`--jwt-private-key`](#jwt-private-key). * `oauth`: Enable OAuth authentication. Requires [`--oauth-issuer`](#oauth-issuer) and[`--oauth-audience`](#oauth-audience). **Default:** `none` |influxdb3 serve option| Environment variable | |----------------------|--------------------------| | `--user-auth-type` |`INFLUXDB3_USER_AUTH_TYPE`| #### without-user-auth Deprecated. When set, overrides [`--user-auth-type`](#user-auth-type):`true` disables user authentication (the same as `--user-auth-type none`), and `false` enables both `basic` and `oauth` user authentication. The server logs a deprecation warning when this option is set. > [!Warning] > #### Deprecated > > `--without-user-auth` is deprecated. > Use [`--user-auth-type`](#user-auth-type) instead. **Default:** not set (`--user-auth-type` applies) |influxdb3 serve option| Environment variable | |----------------------|-----------------------------| |`--without-user-auth` |`INFLUXDB3_WITHOUT_USER_AUTH`| #### jwt-key-id RSA key ID used for signing JWTs for user authentication. Required together with [jwt-private-key](#jwt-private-key) to enable JWT authentication. |influxdb3 serve option| Environment variable | |----------------------|----------------------| | `--jwt-key-id` |`INFLUXDB3_JWT_KEY_ID`| #### jwt-private-key RSA private key (PEM) used for signing JWTs for user authentication. Required together with [jwt-key-id](#jwt-key-id) to enable JWT authentication. |influxdb3 serve option| Environment variable | |----------------------|---------------------------| | `--jwt-private-key` |`INFLUXDB3_JWT_PRIVATE_KEY`| #### jwt-issuer Issuer claim for user authentication JWTs. **Default:** `influxdb3-enterprise` |influxdb3 serve option| Environment variable | |----------------------|----------------------| | `--jwt-issuer` |`INFLUXDB3_JWT_ISSUER`| #### jwt-default-ttl-seconds Default TTL (in seconds) for user authentication JWTs. | influxdb3 serve option | Environment variable | |---------------------------|-----------------------------------| |`--jwt-default-ttl-seconds`|`INFLUXDB3_JWT_DEFAULT_TTL_SECONDS`| #### oauth-issuer OAuth issuer URL for validating OAuth tokens. Required together with [oauth-audience](#oauth-audience) to enable OAuth authentication. |influxdb3 serve option| Environment variable | |----------------------|------------------------| | `--oauth-issuer` |`INFLUXDB3_OAUTH_ISSUER`| #### oauth-audience OAuth audience for validating OAuth tokens. Required together with [oauth-issuer](#oauth-issuer) to enable OAuth authentication. |influxdb3 serve option| Environment variable | |----------------------|--------------------------| | `--oauth-audience` |`INFLUXDB3_OAUTH_AUDIENCE`| #### oauth-client-id OAuth client ID for the device-code login flow. Required to enable `influxdb3 auth login --oauth`. |influxdb3 serve option| Environment variable | |----------------------|---------------------------| | `--oauth-client-id` |`INFLUXDB3_OAUTH_CLIENT_ID`| #### oauth-scopes OAuth scopes to request during device-code login (comma-separated). **Default:** `openid,offline_access` |influxdb3 serve option| Environment variable | |----------------------|------------------------| | `--oauth-scopes` |`INFLUXDB3_OAUTH_SCOPES`| #### rbac-authoring-disabled Disables RBAC authoring endpoints and CLI commands. When `true`, InfluxDB 3 Enterprise doesn’t expose endpoints or commands that create roles, change role permissions, look up roles by name, or read role permissions. Listing roles and assigning roles to users remain available. Accepts `true` or `false`. This option has no effect when user authentication is disabled (see [`--user-auth-type`](#user-auth-type)). **Default:** `true` | influxdb3 serve option | Environment variable | |---------------------------|-----------------------------------| |`--rbac-authoring-disabled`|`INFLUXDB3_RBAC_AUTHORING_DISABLED`| ### AWS * [aws-access-key-id](#aws-access-key-id) * [aws-secret-access-key](#aws-secret-access-key) * [aws-default-region](#aws-default-region) * [aws-endpoint](#aws-endpoint) * [aws-session-token](#aws-session-token) * [aws-allow-http](#aws-allow-http) * [aws-skip-signature](#aws-skip-signature) * [aws-credentials-file](#aws-credentials-file) #### aws-access-key-id When using Amazon S3 as the object store, set this to an access key that has permission to read from and write to the specified S3 bucket. |influxdb3 serve option|Environment variable| |----------------------|--------------------| |`--aws-access-key-id` |`AWS_ACCESS_KEY_ID` | #### aws-secret-access-key When using Amazon S3 as the object store, set this to the secret access key that goes with the specified access key ID. | influxdb3 serve option | Environment variable | |-------------------------|-----------------------| |`--aws-secret-access-key`|`AWS_SECRET_ACCESS_KEY`| #### aws-default-region When using Amazon S3 as the object store, set this to the region that goes with the specified bucket if different from the fallback value. **Default:** `us-east-1` |influxdb3 serve option|Environment variable| |----------------------|--------------------| |`--aws-default-region`|`AWS_DEFAULT_REGION`| #### aws-endpoint When using an Amazon S3 compatibility storage service, set this to the endpoint. |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--aws-endpoint` | `AWS_ENDPOINT` | #### aws-session-token When using Amazon S3 as an object store, set this to the session token. This is handy when using a federated login or SSO and fetching credentials via the UI. |influxdb3 serve option|Environment variable| |----------------------|--------------------| |`--aws-session-token` |`AWS_SESSION_TOKEN` | #### aws-allow-http Allows unencrypted HTTP connections to AWS. |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--aws-allow-http` | `AWS_ALLOW_HTTP` | #### aws-skip-signature If enabled, S3 object stores do not fetch credentials and do not sign requests. |influxdb3 serve option|Environment variable| |----------------------|--------------------| |`--aws-skip-signature`|`AWS_SKIP_SIGNATURE`| #### aws-credentials-file Specifies the path to your S3 credentials file. When using a credentials file, settings in the file override the corresponding CLI flags. S3 credential files are JSON-formatted and should contain the following: ```json { "aws_access_key_id": "AWS_ACCESS_KEY_ID", "aws_secret_access_key": "AWS_SECRET_ACCESS_KEY", "aws_session_token": "AWS_SESSION_TOKEN", "expiry": "UNIX_SECONDS_TIMESTAMP" } ``` The `aws_session_token` and `expiry` fields are optional. The file is automatically checked for updates at the expiry time or at 1-hour intervals. If the object store returns an “Unauthenticated” error, InfluxDB will attempt to update its in-memory credentials from this file and then retry the object store request. | influxdb3 serve option | Environment variable | |------------------------|----------------------| |`--aws-credentials-file`|`AWS_CREDENTIALS_FILE`| ### Google Cloud Service * [google-service-account](#google-service-account) #### google-service-account When using Google Cloud Storage as the object store, set this to the path to the JSON file that contains the Google credentials. | influxdb3 serve option | Environment variable | |--------------------------|------------------------| |`--google-service-account`|`GOOGLE_SERVICE_ACCOUNT`| ### Microsoft Azure * [azure-storage-account](#azure-storage-account) * [azure-storage-access-key](#azure-storage-access-key) * [azure-endpoint](#azure-endpoint) * [azure-allow-http](#azure-allow-http) #### azure-storage-account When using Microsoft Azure as the object store, set this to the name you see when navigating to **All Services \> Storage accounts \> `[name]`**. | influxdb3 serve option | Environment variable | |-------------------------|-----------------------| |`--azure-storage-account`|`AZURE_STORAGE_ACCOUNT`| #### azure-storage-access-key When using Microsoft Azure as the object store, set this to one of the Key values in the Storage account’s **Settings \> Access keys**. | influxdb3 serve option | Environment variable | |----------------------------|--------------------------| |`--azure-storage-access-key`|`AZURE_STORAGE_ACCESS_KEY`| #### azure-endpoint When using Microsoft Azure as the object store, set this to the Azure Blob Storage endpoint. |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--azure-endpoint` | `AZURE_ENDPOINT` | #### azure-allow-http When using Microsoft Azure as the object store, allow unencrypted HTTP requests to Azure Blob Storage. **Default:** `false` |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--azure-allow-http` | `AZURE_ALLOW_HTTP` | ### Object Storage * [bucket](#bucket) * [object-store-connection-limit](#object-store-connection-limit) * [object-store-http2-only](#object-store-http2-only) * [object-store-http2-max-frame-size](#object-store-http2-max-frame-size) * [object-store-request-timeout](#object-store-request-timeout) * [object-store-max-retries](#object-store-max-retries) * [object-store-retry-timeout](#object-store-retry-timeout) * [object-store-tls-allow-insecure](#object-store-tls-allow-insecure) * [object-store-tls-ca](#object-store-tls-ca) #### bucket Sets the name of the object storage bucket to use. Must also set`--object-store` to a cloud object storage for this option to take effect. |influxdb3 serve option|Environment variable| |----------------------|--------------------| | `--bucket` | `INFLUXDB3_BUCKET` | #### object-store-connection-limit When using a network-based object store, limits the number of connections to this value. **Default:** `16` | influxdb3 serve option | Environment variables | |---------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| |`--object-store-connection-limit`|`INFLUXDB3_OBJECT_STORE_CONNECTION_LIMIT` (preferred)
`OBJECT_STORE_CONNECTION_LIMIT` (deprecated; supported for backward compatibility)| #### object-store-http2-only Forces HTTP/2 connections to network-based object stores. | influxdb3 serve option | Environment variables | |---------------------------|----------------------------------------------------------------------------------------------------------------------------------| |`--object-store-http2-only`|`INFLUXDB3_OBJECT_STORE_HTTP2_ONLY` (preferred)
`OBJECT_STORE_HTTP2_ONLY` (deprecated; supported for backward compatibility)| #### object-store-http2-max-frame-size Sets the maximum frame size (in bytes/octets) for HTTP/2 connections. | influxdb3 serve option | Environment variables | |-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| |`--object-store-http2-max-frame-size`|`INFLUXDB3_OBJECT_STORE_HTTP2_MAX_FRAME_SIZE` (preferred)
`OBJECT_STORE_HTTP2_MAX_FRAME_SIZE` (deprecated; supported for backward compatibility)| #### object-store-request-timeout Sets the HTTP request timeout for object store requests. **Default:** `30s` | influxdb3 serve option | Environment variables | |--------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------| |`--object-store-request-timeout`|`INFLUXDB3_OBJECT_STORE_REQUEST_TIMEOUT` (preferred)
`OBJECT_STORE_REQUEST_TIMEOUT` (deprecated; supported for backward compatibility)| #### object-store-max-retries Defines the maximum number of times to retry a request. | influxdb3 serve option | Environment variables | |----------------------------|------------------------------------------------------------------------------------------------------------------------------------| |`--object-store-max-retries`|`INFLUXDB3_OBJECT_STORE_MAX_RETRIES` (preferred)
`OBJECT_STORE_MAX_RETRIES` (deprecated; supported for backward compatibility)| #### object-store-retry-timeout Specifies the maximum length of time from the initial request after which no further retries are be attempted. | influxdb3 serve option | Environment variables | |------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| |`--object-store-retry-timeout`|`INFLUXDB3_OBJECT_STORE_RETRY_TIMEOUT` (preferred)
`OBJECT_STORE_RETRY_TIMEOUT` (deprecated; supported for backward compatibility)| #### object-store-tls-allow-insecure Allows invalid TLS certificates when connecting to object storage. This disables TLS certificate verification and should only be used for testing. | influxdb3 serve option | Environment variables | |-----------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------| |`--object-store-tls-allow-insecure`|`INFLUXDB3_OBJECT_STORE_TLS_ALLOW_INSECURE` (preferred)
`OBJECT_STORE_TLS_ALLOW_INSECURE` (deprecated; supported for backward compatibility)| #### object-store-tls-ca Specifies the path to a custom CA certificate file (PEM format) for verifying object store connections. Use this when your object store uses a certificate signed by a private CA. |influxdb3 serve option | Environment variables | |-----------------------|--------------------------------------------------------------------------------------------------------------------------| |`--object-store-tls-ca`|`INFLUXDB3_OBJECT_STORE_TLS_CA` (preferred)
`OBJECT_STORE_TLS_CA` (deprecated; supported for backward compatibility)| ### Logs * [log-filter](#log-filter) * [log-destination](#log-destination) * [log-format](#log-format) * [query-log-max-entries](#query-log-max-entries) (`--query-log-size` before 3.11) #### log-filter Sets the filter directive for logs. Use this option to control the verbosity of server logs globally or for specific components. ##### Log levels The following log levels are available (from least to most verbose): | Level | Description | |-------|-----------------------------------------------------------------------------------| |`error`| Only errors | |`warn` | Warnings and errors | |`info` | Informational messages, warnings, and errors *(default)* | |`debug`| Debug information for troubleshooting, plus all above levels | |`trace`|Very detailed tracing information, plus all above levels (produces high log volume)| ##### Basic usage To set the log level globally, pass one of the log levels: ```sh influxdb3 serve --log-filter debug ``` ##### Targeted filtering Globally enabling `debug` or `trace` produces a high volume of log output. For more targeted debugging, you can set different log levels for specific components using the format `,=`. ###### Debug write buffer operations ```sh influxdb3 serve --log-filter info,influxdb3_write_buffer=debug ``` ###### Trace WAL operations ```sh influxdb3 serve --log-filter info,influxdb3_wal=trace ``` ###### Multiple targeted filters ```sh influxdb3 serve --log-filter info,influxdb3_write_buffer=debug,influxdb3_wal=debug ``` ###### Debug Enterprise storage engine operations ```sh influxdb3 serve --log-filter info,influxdb3_enterprise=debug ``` ##### Common component names The following are common component names you can use for targeted filtering: | Component | Description | |------------------------|-----------------------------| |`influxdb3_write_buffer`| Write buffer operations | | `influxdb3_wal` | Write-ahead log operations | | `influxdb3_catalog` |Catalog and schema operations| | `influxdb3_cache` | Caching operations | | Component | Description | |----------------------|----------------------------| |`influxdb3_enterprise`|Enterprise-specific features| > [!Note] > Targeted filtering requires knowledge of the codebase component names. > The component names correspond to Rust package names in the InfluxDB 3 source > code. Use `debug` or `trace` sparingly on specific components to avoid > excessive log output. |influxdb3 serve option| Environment variables | |----------------------|--------------------------------------------------------------------------------------------------------| | `--log-filter` |`INFLUXDB3_LOG_FILTER` (preferred)
`LOG_FILTER` (deprecated; supported for backward compatibility)| #### log-destination Specifies the destination for logs. This option supports the following values: * `stdout` *(default)* * `stderr` **Default:** `stdout` |influxdb3 serve option| Environment variables | |----------------------|------------------------------------------------------------------------------------------------------------------| | `--log-destination` |`INFLUXDB3_LOG_DESTINATION` (preferred)
`LOG_DESTINATION` (deprecated; supported for backward compatibility)| #### log-format Defines the message format for logs. This option supports the following values: * `full` *(default)* **Default:** `full` |influxdb3 serve option| Environment variables | |----------------------|--------------------------------------------------------------------------------------------------------| | `--log-format` |`INFLUXDB3_LOG_FORMAT` (preferred)
`LOG_FORMAT` (deprecated; supported for backward compatibility)| #### query-log-max-entries Defines the maximum number of entries in the query log. Up to this many queries remain in the log before older queries are evicted to make room for new ones. **Default:** `1000` > [!Note] > `--query-log-max-entries` was [renamed in 3.11](#name-changes-in-3-11)from `--query-log-size`. > The legacy option and environment variable names are deprecated aliases. | influxdb3 serve option | Environment variables | |------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------| |`--query-log-max-entries`
`--query-log-size` (pre-3.11 name)|`INFLUXDB3_QUERY_LOG_MAX_ENTRIES`
`INFLUXDB3_QUERY_LOG_SIZE` ([pre-3.11 name](#name-changes-in-3-11))| ### Traces * [traces-exporter](#traces-exporter) * [traces-exporter-jaeger-agent-host](#traces-exporter-jaeger-agent-host) * [traces-exporter-jaeger-agent-port](#traces-exporter-jaeger-agent-port) * [traces-exporter-jaeger-service-name](#traces-exporter-jaeger-service-name) * [traces-exporter-jaeger-trace-context-header-name](#traces-exporter-jaeger-trace-context-header-name) * [traces-jaeger-debug-name](#traces-jaeger-debug-name) * [traces-jaeger-tags](#traces-jaeger-tags) * [traces-jaeger-max-msgs-per-second](#traces-jaeger-max-msgs-per-second) #### traces-exporter Sets the type of tracing exporter. **Default:** `none` |influxdb3 serve option| Environment variables | |----------------------|------------------------------------------------------------------------------------------------------------------| | `--traces-exporter` |`INFLUXDB3_TRACES_EXPORTER` (preferred)
`TRACES_EXPORTER` (deprecated; supported for backward compatibility)| #### traces-exporter-jaeger-agent-host Specifies the Jaeger agent network hostname for tracing. **Default:** `0.0.0.0` | influxdb3 serve option | Environment variables | |-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-exporter-jaeger-agent-host`|`INFLUXDB3_TRACES_EXPORTER_JAEGER_AGENT_HOST` (preferred)
`TRACES_EXPORTER_JAEGER_AGENT_HOST` (deprecated; supported for backward compatibility)| #### traces-exporter-jaeger-agent-port Defines the Jaeger agent network port for tracing. **Default:** `6831` | influxdb3 serve option | Environment variables | |-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-exporter-jaeger-agent-port`|`INFLUXDB3_TRACES_EXPORTER_JAEGER_AGENT_PORT` (preferred)
`TRACES_EXPORTER_JAEGER_AGENT_PORT` (deprecated; supported for backward compatibility)| #### traces-exporter-jaeger-service-name Sets the Jaeger service name for tracing. **Default:** `iox-conductor` | influxdb3 serve option | Environment variables | |---------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-exporter-jaeger-service-name`|`INFLUXDB3_TRACES_EXPORTER_JAEGER_SERVICE_NAME` (preferred)
`TRACES_EXPORTER_JAEGER_SERVICE_NAME` (deprecated; supported for backward compatibility)| #### traces-exporter-jaeger-trace-context-header-name Specifies the header name used for passing trace context. **Default:** `uber-trace-id` | influxdb3 serve option | Environment variables | |----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-exporter-jaeger-trace-context-header-name`|`INFLUXDB3_TRACES_EXPORTER_JAEGER_TRACE_CONTEXT_HEADER_NAME` (preferred)
`TRACES_EXPORTER_JAEGER_TRACE_CONTEXT_HEADER_NAME` (deprecated; supported for backward compatibility)| #### traces-jaeger-debug-name Specifies the header name used for force sampling in tracing. **Default:** `jaeger-debug-id` | influxdb3 serve option | Environment variables | |----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-jaeger-debug-name`|`INFLUXDB3_TRACES_EXPORTER_JAEGER_DEBUG_NAME` (preferred)
`TRACES_EXPORTER_JAEGER_DEBUG_NAME` (deprecated; supported for backward compatibility)| #### traces-jaeger-tags Defines a set of `key=value` pairs to annotate tracing spans with. |influxdb3 serve option| Environment variables | |----------------------|------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-jaeger-tags`|`INFLUXDB3_TRACES_EXPORTER_JAEGER_TAGS` (preferred)
`TRACES_EXPORTER_JAEGER_TAGS` (deprecated; supported for backward compatibility)| #### traces-jaeger-max-msgs-per-second Specifies the maximum number of messages sent to a Jaeger service per second. **Default:** `1000` | influxdb3 serve option | Environment variables | |-------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| |`--traces-jaeger-max-msgs-per-second`|`INFLUXDB3_TRACES_JAEGER_MAX_MSGS_PER_SECOND` (preferred)
`TRACES_JAEGER_MAX_MSGS_PER_SECOND` (deprecated; supported for backward compatibility)| ### DataFusion * [datafusion-num-threads](#datafusion-num-threads) * [datafusion-max-parquet-fanout](#datafusion-max-parquet-fanout) * [datafusion-use-cached-parquet-loader](#datafusion-use-cached-parquet-loader) * [datafusion-config](#datafusion-config) #### datafusion-num-threads Sets the maximum number of DataFusion runtime threads to use. | influxdb3 serve option | Environment variable | |--------------------------|----------------------------------| |`--datafusion-num-threads`|`INFLUXDB3_DATAFUSION_NUM_THREADS`| #### datafusion-max-parquet-fanout When multiple Parquet files are required in a sorted way (deduplication for example), specifies the maximum fanout. **Default:** `1000` | influxdb3 serve option | Environment variable | |---------------------------------|-----------------------------------------| |`--datafusion-max-parquet-fanout`|`INFLUXDB3_DATAFUSION_MAX_PARQUET_FANOUT`| #### datafusion-use-cached-parquet-loader Uses a cached Parquet loader when reading Parquet files from the object store. | influxdb3 serve option | Environment variable | |----------------------------------------|------------------------------------------------| |`--datafusion-use-cached-parquet-loader`|`INFLUXDB3_DATAFUSION_USE_CACHED_PARQUET_LOADER`| #### datafusion-config Provides custom configuration to DataFusion as a comma-separated list of`key:value` pairs. |influxdb3 serve option| Environment variable | |----------------------|-----------------------------| |`--datafusion-config` |`INFLUXDB3_DATAFUSION_CONFIG`| ### HTTP * [max-http-request-size](#max-http-request-size) * [http-bind](#http-bind) #### max-http-request-size Specifies the maximum size of HTTP requests. Prefer a [unit suffix](#size-option-values)—for example, `10mb`. A bare number is accepted as bytes (its pre-3.11 meaning) with a startup warning. **Default:** `10mb` | influxdb3 serve option | Environment variable | |-------------------------|---------------------------------| |`--max-http-request-size`|`INFLUXDB3_MAX_HTTP_REQUEST_SIZE`| #### http-bind Defines the address on which InfluxDB serves HTTP API requests. **Default:** `0.0.0.0:8181` |influxdb3 serve option| Environment variable | |----------------------|--------------------------| | `--http-bind` |`INFLUXDB3_HTTP_BIND_ADDR`| ### Memory * [exec-mem-pool-size](#exec-mem-pool-size) (`--exec-mem-pool-bytes` before 3.11) * [force-snapshot-mem-size](#force-snapshot-mem-size) (`--force-snapshot-mem-threshold` before 3.11) * [force-snapshot-max-age](#force-snapshot-max-age) #### exec-mem-pool-size Specifies the size of the memory pool used for query processing and data operations. This memory pool is used when InfluxDB 3 Enterprise processes queries and performs internal data management tasks. Provide a value with a [unit suffix](#size-option-values) or as a percentage of the total available memory—for example: `8gb` or `10%`. **Default:** `20%` > [!Note] > `--exec-mem-pool-size` was [renamed in 3.11](#name-changes-in-3-11)from `--exec-mem-pool-bytes`. > The legacy option and environment variable names are still > accepted–including the pre-3.11 bare-number format, meaning > megabytes–with a startup deprecation warning. | influxdb3 serve option | Environment variables | |--------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| |`--exec-mem-pool-size`
`--exec-mem-pool-bytes` (pre-3.11 name)|`INFLUXDB3_EXEC_MEM_POOL_SIZE`
`INFLUXDB3_EXEC_MEM_POOL_BYTES` ([pre-3.11 name](#name-changes-in-3-11))| #### force-snapshot-mem-size Specifies the threshold for the internal memory buffer. Supports either a percentage (portion of available memory) or a value with a[unit suffix](#size-option-values)—for example: `70%` or `1000mb`. **Default:** `50%` > [!Note] > `--force-snapshot-mem-size` was [renamed in 3.11](#name-changes-in-3-11)from `--force-snapshot-mem-threshold`. > The legacy option and environment variable names are still > accepted–including the pre-3.11 bare-number format, meaning > megabytes–with a startup deprecation warning. | influxdb3 serve option | Environment variables | |----------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------| |`--force-snapshot-mem-size`
`--force-snapshot-mem-threshold` (pre-3.11 name)|`INFLUXDB3_FORCE_SNAPSHOT_MEM_SIZE`
`INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD` ([pre-3.11 name](#name-changes-in-3-11))| #### force-snapshot-max-age Specifies how long WAL data can wait for a snapshot. When the oldest WAL data that isn’t in a snapshot reaches this age, InfluxDB forces a snapshot, regardless of how many WAL files have accumulated. WAL data replayed after a restart ages from the time it’s replayed. InfluxDB checks the age every 10 seconds. Snapshots otherwise trigger on the WAL file count or on[force-snapshot-mem-size](#force-snapshot-mem-size), so a node that receives few or no writes can hold hours of data in memory and replay it all on restart. This option bounds that wait. This option applies to the Parquet engine only (clusters that started on 3.10 or earlier that have not run the[storage engine upgrade](#upgrade-pacha-tree)). **Default:** `1h` | influxdb3 serve option | Environment variable | |--------------------------|----------------------------------| |`--force-snapshot-max-age`|`INFLUXDB3_FORCE_SNAPSHOT_MAX_AGE`| ### Write-Ahead Log (WAL) * [checkpoint-interval](#checkpoint-interval) * [wal-flush-interval](#wal-flush-interval) * [wal-files-per-snapshot](#wal-files-per-snapshot) (`--wal-snapshot-size` before 3.11) * [wal-max-buffered-writes](#wal-max-buffered-writes) (`--wal-max-write-buffer-size` before 3.11) * [snapshotted-wal-files-to-keep](#snapshotted-wal-files-to-keep) * [wal-replay-fail-on-error](#wal-replay-fail-on-error) * [wal-replay-concurrency-limit](#wal-replay-concurrency-limit) * [write-timestamp-max-past](#write-timestamp-max-past) * [write-timestamp-max-future](#write-timestamp-max-future) > [!Note] > `wal-files-per-snapshot`, `wal-max-buffered-writes`, and`snapshotted-wal-files-to-keep` apply to the Parquet engine only > (clusters that started on 3.10 or earlier that have not run the[storage engine upgrade](#upgrade-pacha-tree)).`wal-flush-interval` applies to both engines. #### checkpoint-interval Sets the interval for consolidating[snapshots](/influxdb3/enterprise/admin/backup-restore/#file-structure) into monthly checkpoints for faster server startup. Snapshots accumulate in object storage over time and are not automatically deleted. Without checkpointing, the server loads individual snapshots on startup. The number of snapshots is determined by the lookback window ([`gen1-lookback-duration`](#gen1-lookback-duration), default 1 month) divided by [`gen1-duration`](#gen1-duration) (default 10 minutes), with a minimum of 100. With default settings, that can be up to \~4,320 snapshots. With checkpointing enabled, the server periodically consolidates snapshot metadata into checkpoints in object storage. On startup, the server loads one to two checkpoints per calendar month, then loads only snapshots created since the last checkpoint. Enabling checkpointing does not delete old snapshots. Up to 10 checkpoints load concurrently during startup. The server retains two checkpoints per calendar month and handles month rollovers automatically. Accepts a [duration](/influxdb3/enterprise/reference/glossary/#duration) value–for example: `1h`, `30m`, `10m`. **Default:** *Not set (disabled)* |influxdb3 serve option | Environment variable | |-----------------------|-------------------------------| |`--checkpoint-interval`|`INFLUXDB3_CHECKPOINT_INTERVAL`| ##### Example ```bash influxdb3 serve --checkpoint-interval 1h ``` #### wal-flush-interval Specifies the interval to flush buffered data to a WAL file. Writes that wait for WAL confirmation take up to this interval to complete. Use `s` for seconds or `ms` for milliseconds. For local disks, `100 ms` is recommended. **Default:** `1s` |influxdb3 serve option| Environment variable | |----------------------|------------------------------| |`--wal-flush-interval`|`INFLUXDB3_WAL_FLUSH_INTERVAL`| #### wal-files-per-snapshot Defines the number of WAL files to attempt to remove in a snapshot. This, multiplied by the interval, determines how often snapshots are taken. **Default:** `600` > [!Note] > `--wal-files-per-snapshot` was [renamed in 3.11](#name-changes-in-3-11)from `--wal-snapshot-size`. > The legacy option and environment variable names are deprecated aliases. | influxdb3 serve option | Environment variables | |----------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------| |`--wal-files-per-snapshot`
`--wal-snapshot-size` (pre-3.11 name)|`INFLUXDB3_WAL_FILES_PER_SNAPSHOT`
`INFLUXDB3_WAL_SNAPSHOT_SIZE` ([pre-3.11 name](#name-changes-in-3-11))| #### wal-max-buffered-writes Specifies the maximum number of write requests that can be buffered awaiting a WAL flush. **Default:** `100000` > [!Note] > #### In v3.12+: enforced > > InfluxDB 3 Enterprise enforces this limit. > Writes that arrive while the buffer is at this cap are rejected with an > HTTP `429` response and the message `wal buffer is full; writes are temporarily rejected, retry shortly` until a flush drains the buffer.`/api/v2/write` returns the same condition as JSON with`code: "too many requests"`. > Retry the write after a short delay. > [!Note] > `--wal-max-buffered-writes` was [renamed in 3.11](#name-changes-in-3-11)from `--wal-max-write-buffer-size`. > The legacy option and environment variable names are deprecated aliases. | influxdb3 serve option | Environment variables | |-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| |`--wal-max-buffered-writes`
`--wal-max-write-buffer-size` (pre-3.11 name)|`INFLUXDB3_WAL_MAX_BUFFERED_WRITES`
`INFLUXDB3_WAL_MAX_WRITE_BUFFER_SIZE` ([pre-3.11 name](#name-changes-in-3-11))| #### snapshotted-wal-files-to-keep Specifies the number of snapshotted WAL files to retain in the object store. Flushing the WAL files does not clear the WAL files immediately; they are deleted when the number of snapshotted WAL files exceeds this number. **Default:** `300` | influxdb3 serve option | Environment variables | |---------------------------------|--------------------------------------------------------------------------------------------------------------------------| |`--snapshotted-wal-files-to-keep`|`INFLUXDB3_SNAPSHOTTED_WAL_FILES_TO_KEEP`
`INFLUXDB3_NUM_WAL_FILES_TO_KEEP` ([pre-3.11 name](#name-changes-in-3-11))| #### wal-replay-fail-on-error Determines whether WAL replay should fail when encountering errors. **Default:** `false` | influxdb3 serve option | Environment variable | |----------------------------|------------------------------------| |`--wal-replay-fail-on-error`|`INFLUXDB3_WAL_REPLAY_FAIL_ON_ERROR`| #### wal-replay-concurrency-limit Concurrency limit during WAL replay. Setting this number too high can lead to OOM. The default is dynamically determined. **Default:** `max(num_cpus, 10)` | influxdb3 serve option | Environment variable | |--------------------------------|----------------------------------------| |`--wal-replay-concurrency-limit`|`INFLUXDB3_WAL_REPLAY_CONCURRENCY_LIMIT`| #### write-timestamp-max-past Rejects explicit point timestamps older than this duration relative to the time the server received the write request. Lines that don’t specify an explicit timestamp are exempt. Applies to every database except `_internal`. Each ingest node evaluates the bound against its own clock, so rejections are per line, and the usual partial-write rules apply to the rest of the request. Disabled when unset. This option applies to the upgraded storage engine only. **Default:** *Not set (disabled)* | influxdb3 serve option | Environment variable | |----------------------------|------------------------------------| |`--write-timestamp-max-past`|`INFLUXDB3_WRITE_TIMESTAMP_MAX_PAST`| #### write-timestamp-max-future Rejects explicit point timestamps newer than this duration relative to the time the server received the write request. Lines that don’t specify an explicit timestamp are exempt. Applies to every database except `_internal`. Each ingest node evaluates the bound against its own clock, so rejections are per line, and the usual partial-write rules apply to the rest of the request. Disabled when unset. This option applies to the upgraded storage engine only. **Default:** *Not set (disabled)* | influxdb3 serve option | Environment variable | |------------------------------|--------------------------------------| |`--write-timestamp-max-future`|`INFLUXDB3_WRITE_TIMESTAMP_MAX_FUTURE`| ### Compaction > [!Note] > The `compaction-*` and `gen1-*` options apply to the Parquet engine only > (clusters that started on 3.10 or earlier that have not run the[storage engine upgrade](#upgrade-pacha-tree)). > For the upgraded storage engine’s compaction options, see the[storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/). * [compaction-row-limit](#compaction-row-limit) * [compaction-max-num-files-per-plan](#compaction-max-num-files-per-plan) * [compaction-gen2-duration](#compaction-gen2-duration) * [compaction-multipliers](#compaction-multipliers) * [compaction-cleanup-wait](#compaction-cleanup-wait) * [compaction-check-interval](#compaction-check-interval) * [compacted-data-load-concurrency-limit](#compacted-data-load-concurrency-limit) * [compacted-data-skip-file-index](#compacted-data-skip-file-index) > [!Note] > The following distributed-compaction and orphan-sweep options apply to > the upgraded storage engine only. > For more information, see[Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/). * [compactor-dispatch-target](#compactor-dispatch-target) * [compactor-dispatch-ack-timeout](#compactor-dispatch-ack-timeout) * [compactor-sweep-interval](#compactor-sweep-interval) * [compactor-sweep-mode](#compactor-sweep-mode) * [compactor-sweep-grace](#compactor-sweep-grace) * [compactor-sweep-auto-resume](#compactor-sweep-auto-resume) * [compactor-sweep-audit-retention](#compactor-sweep-audit-retention) * [gen1-duration](#gen1-duration) #### compaction-row-limit Specifies the soft limit for the number of rows per file that the compactor writes. The compactor may write more rows than this limit. **Default:** `1000000` | influxdb3 serve option | Environment variable | |------------------------|---------------------------------------------------------------------------------------------------------------------------| |`--compaction-row-limit`|`INFLUXDB3_COMPACTION_ROW_LIMIT`
`INFLUXDB3_ENTERPRISE_COMPACTION_ROW_LIMIT` ([pre-3.11 name](#name-changes-in-3-11))| #### compaction-max-num-files-per-plan > [!Warning] > #### Deprecated in v3.12: has no effect > > In InfluxDB 3 Enterprise 3.12, `--compaction-max-num-files-per-plan` no > longer has any effect: the compactor no longer limits compaction plans by > input file count. > The option is still accepted so existing configurations start, but the > server logs a startup warning–remove it from your configuration. | influxdb3 serve option | Environment variable | |-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| |`--compaction-max-num-files-per-plan`|`INFLUXDB3_COMPACTION_MAX_NUM_FILES_PER_PLAN`
`INFLUXDB3_ENTERPRISE_COMPACTION_MAX_NUM_FILES_PER_PLAN` ([pre-3.11 name](#name-changes-in-3-11))| #### compaction-gen2-duration Specifies the duration of the first level of compaction (gen2). Later levels of compaction are multiples of this duration. This value should be equal to or greater than the gen1 duration. **Default:** `20m` | influxdb3 serve option | Environment variable | |----------------------------|-----------------------------------------------------------------------------------------------------------------------------------| |`--compaction-gen2-duration`|`INFLUXDB3_COMPACTION_GEN2_DURATION`
`INFLUXDB3_ENTERPRISE_COMPACTION_GEN2_DURATION` ([pre-3.11 name](#name-changes-in-3-11))| #### compaction-multipliers Specifies a comma-separated list of multiples defining the duration of each level of compaction. The number of elements in the list determines the number of compaction levels. The first element specifies the duration of the first level (gen3); subsequent levels are multiples of the previous level. **Default:** `3,4,6,5` | influxdb3 serve option | Environment variable | |--------------------------|-------------------------------------------------------------------------------------------------------------------------------| |`--compaction-multipliers`|`INFLUXDB3_COMPACTION_MULTIPLIERS`
`INFLUXDB3_ENTERPRISE_COMPACTION_MULTIPLIERS` ([pre-3.11 name](#name-changes-in-3-11))| #### compaction-cleanup-wait Specifies the amount of time that the compactor waits after finishing a compaction run to delete files marked as needing deletion during that compaction run. On the legacy Parquet storage engine, compaction doesn’t mark Gen1 files for deletion; see [Gen1 file cleanup](/influxdb3/enterprise/admin/gen1-file-cleanup/). **Default:** `1h` | influxdb3 serve option | Environment variable | |---------------------------|---------------------------------------------------------------------------------------------------------------------------------| |`--compaction-cleanup-wait`|`INFLUXDB3_COMPACTION_CLEANUP_WAIT`
`INFLUXDB3_ENTERPRISE_COMPACTION_CLEANUP_WAIT` ([pre-3.11 name](#name-changes-in-3-11))| #### compaction-check-interval Specifies how often the compactor checks for new compaction work to perform. **Default:** `10s` | influxdb3 serve option | Environment variable | |-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------| |`--compaction-check-interval`|`INFLUXDB3_COMPACTION_CHECK_INTERVAL`
`INFLUXDB3_ENTERPRISE_COMPACTION_CHECK_INTERVAL` ([pre-3.11 name](#name-changes-in-3-11))| #### compacted-data-load-concurrency-limit Specifies the maximum number of concurrent object store fetches while a node loads compacted data (compaction detail and generation detail files), both at startup and when a consumer picks up new compaction summaries. The default bounds the transient memory used by in-flight downloads and is sized so the load saturates neither a 10 GbE network interface nor the shared object store connection pool (see[object-store-connection-limit](#object-store-connection-limit)). Increase the limit for faster startup when the compaction index is large and the host has network headroom; decrease it if the object store throttles the load or memory is tight during startup. This option isn’t supported in the TOML configuration file; use the command option or environment variable. **Default:** `20` | influxdb3 serve option | Environment variable | |-----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| |`--compacted-data-load-concurrency-limit`|`INFLUXDB3_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT`
`INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT` ([pre-3.11 name](#name-changes-in-3-11))| #### compacted-data-skip-file-index Loads compacted data without building the in-memory file index. Use this option when the file index has grown too large to fit in host memory and the node can’t start. Without the index, queries can’t prune to a subset of files and instead scan all files in the matching time range, so queries are correct but slower. Pruning generations by time range is unaffected. The setting applies to the node as a whole: a node started with this option also skips index merges for newly compacted generations, and continues to persist full indexes to the object store for other nodes to use. Starting in v3.11.5, nodes that run only in ingest mode (`--mode ingest`) skip the index regardless of this option, because they don’t prune files at query time. In v3.12.1 and later, only nodes with query mode enabled load the index. This includes nodes started with `--mode query` or `--mode all`. Other nodes skip the index regardless of this option and log the following message at startup: ```text compacted-data file index not loaded ``` This option isn’t supported in the TOML configuration file; use the command option or environment variable. **Default:** `false` | influxdb3 serve option | Environment variable | |----------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------| |`--compacted-data-skip-file-index`|`INFLUXDB3_COMPACTED_DATA_SKIP_FILE_INDEX`
`INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX` ([pre-3.11 name](#name-changes-in-3-11))| #### compactor-dispatch-target Selects which nodes run compaction work. This option supports the following values: * `local` *(default)*: Every compaction runs on the node that holds the compactor lease (the primary). * `remote`: Every compaction runs on other nodes running in `compact`[mode](#mode); work waits until one of those nodes is available. * `all`: Compactions run on both the lease holder and the other compact nodes. `remote` and `all` require every `compact`-mode node to advertise its internode address: set `--internode-bind-addr` (and[`--conn-info`](#conn-info) if the node isn’t otherwise reachable). A compact node that hasn’t advertised a `conn_info` catalog entry can’t receive dispatched work; the primary logs a warning naming the peer. > [!Important] > #### Distributed compaction is a beta feature > > Running compaction across multiple nodes with `remote` or `all` is a beta > feature in InfluxDB 3 Enterprise 3.12. > For more information, see[Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/). This option applies to the upgraded storage engine only. **Default:** `local` | influxdb3 serve option | Environment variable | |-----------------------------|-------------------------------------| |`--compactor-dispatch-target`|`INFLUXDB3_COMPACTOR_DISPATCH_TARGET`| #### compactor-dispatch-ack-timeout Specifies how long a remote output-group dispatch can sit unacknowledged before the compactor primary declares the worker unresponsive and redispatches its in-flight work elsewhere. This option applies to the upgraded storage engine only. **Default:** `5s` | influxdb3 serve option | Environment variable | |----------------------------------|------------------------------------------| |`--compactor-dispatch-ack-timeout`|`INFLUXDB3_COMPACTOR_DISPATCH_ACK_TIMEOUT`| #### compactor-sweep-interval Specifies how often the compactor’s primary node sweeps its output prefixes for orphaned files (objects that no published state references) and records them in a durable audit trail. In [`armed`](#compactor-sweep-mode) mode, the sweep also deletes orphaned files once they’re past the [`--compactor-sweep-grace`](#compactor-sweep-grace)window. Set to `off` to disable the scheduled sweep. This option applies to the upgraded storage engine only. **Default:** `7d` | influxdb3 serve option | Environment variable | |----------------------------|------------------------------------| |`--compactor-sweep-interval`|`INFLUXDB3_COMPACTOR_SWEEP_INTERVAL`| #### compactor-sweep-mode Sets the mode for scheduled orphan-sweep passes. This option supports the following values: * `armed` *(default)*: Enqueue sweep candidates into the cooldown-protected delete queue, so confirmed orphaned files are deleted. * `dry-run`: Only report candidates in the audit trail; delete nothing. An interrupted `armed` pass resumes at the weaker of its recorded mode and this setting. This option applies to the upgraded storage engine only. **Default:** `armed` | influxdb3 serve option | Environment variable | |------------------------|--------------------------------| |`--compactor-sweep-mode`|`INFLUXDB3_COMPACTOR_SWEEP_MODE`| #### compactor-sweep-grace Specifies the orphan-sweep grace window. Objects modified within this duration before a pass’s capture instant are never deletion candidates. Values below `6h` are raised to `6h`. This option applies to the upgraded storage engine only. **Default:** `24h` | influxdb3 serve option | Environment variable | |-------------------------|---------------------------------| |`--compactor-sweep-grace`|`INFLUXDB3_COMPACTOR_SWEEP_GRACE`| #### compactor-sweep-auto-resume Determines whether an unfinished scheduled sweep pass resumes after a restart instead of waiting for a full interval to elapse. Set to `false` only when the sweep itself is misbehaving (for example, crashing mid-pass); every scheduled fire then starts a fresh pass. This option applies to the upgraded storage engine only. **Default:** `true` | influxdb3 serve option | Environment variable | |-------------------------------|---------------------------------------| |`--compactor-sweep-auto-resume`|`INFLUXDB3_COMPACTOR_SWEEP_AUTO_RESUME`| #### compactor-sweep-audit-retention Specifies how long the orphan sweep retains its audit-trail objects. InfluxDB 3 Enterprise expires older audit objects at the start of each pass. This option applies to the upgraded storage engine only. **Default:** `7d` | influxdb3 serve option | Environment variable | |-----------------------------------|-------------------------------------------| |`--compactor-sweep-audit-retention`|`INFLUXDB3_COMPACTOR_SWEEP_AUDIT_RETENTION`| #### gen1-duration Specifies the duration that Parquet files are arranged into. Data timestamps land each row into a file of this duration. Supported durations are `1m`,`5m`, and `10m`. These files are known as “generation 1” files, which the compactor can merge into larger generations . **Default:** `10m` |influxdb3 serve option| Environment variable | |----------------------|-------------------------| | `--gen1-duration` |`INFLUXDB3_GEN1_DURATION`| ### Caching * [preemptive-cache-age](#preemptive-cache-age) * [file-cache-size](#file-cache-size) (`--parquet-mem-cache-size` before 3.11) * [parquet-mem-cache-prune-percentage](#parquet-mem-cache-prune-percentage) * [parquet-mem-cache-prune-interval](#parquet-mem-cache-prune-interval) * [file-cache-recency](#file-cache-recency) (`--parquet-mem-cache-query-path-duration` before 3.11) * [disable-file-cache](#disable-file-cache) (`--disable-parquet-mem-cache`, `--disable-data-file-cache` before 3.11) * [table-index-cache-max-entries](#table-index-cache-max-entries) * [table-index-cache-concurrency-limit](#table-index-cache-concurrency-limit) * [last-value-cache-disable-from-history](#last-value-cache-disable-from-history) * [last-cache-eviction-interval](#last-cache-eviction-interval) * [distinct-value-cache-disable-from-history](#distinct-value-cache-disable-from-history) * [distinct-cache-eviction-interval](#distinct-cache-eviction-interval) #### preemptive-cache-age Specifies the interval to prefetch into the Parquet cache during compaction. **Default:** `3d` | influxdb3 serve option | Environment variable | |------------------------|--------------------------------| |`--preemptive-cache-age`|`INFLUXDB3_PREEMPTIVE_CACHE_AGE`| #### file-cache-size Specifies the size of the in-memory data file cache. Provide a value with a [unit suffix](#size-option-values) or as a percentage of total available memory—for example, `4gb` or `20%`. This is a total budget. With the Parquet engine, the budget is used entirely by the in-memory Parquet cache. During the storage engine upgrade with hybrid query enabled, the budget is split 50/50 between the hybrid-query Parquet cache and the upgraded engine’s file cache; otherwise, the single active cache receives the full budget. > [!Note] > In v3.11.6 and later, the upgraded storage engine enforces a hard cache limit. > The budget includes cached files and files open in running queries. When file access would exceed the budget, InfluxDB 3 Enterprise fails the query. The error identifies whether the budget is exhausted or a file exceeds the total budget: ```text file access cache budget exhausted: ``` ```text file larger than the file access cache budget: ``` Flight (gRPC) clients receive a `RESOURCE_EXHAUSTED` error. For HTTP queries in CSV, JSON, or JSON Lines format: * Before v3.12.1, the response returns HTTP 200 and ends early. * In v3.12.1 and later, a failure before the first row returns HTTP 429. A failure after the response starts still ends the response early. Retry the query after other queries finish. Set the cache size to accommodate the files accessed by concurrent queries. ##### Adjusting the file cache size Because the cache limit is a hard limit, a cache that’s too small causes failures even when the node has free memory. This can look like: * On query nodes, queries fail with `file access cache budget exhausted`. * On compactor nodes, compaction falls behind. If queries fail with cache budget errors, address in the following order: 1. **Check compactor health.**When compaction falls behind, queries read more uncompacted files, which uses more of the query node cache. Signs that compaction is falling behind include the following: * Compactor logs contain `File cache budget exhausted` or`non-admitting run set index read failure, skipping plan this pass`. * The number of L1 run sets in `system.pt_compaction_run_sets` keeps growing. * `deferred_snapshot_count` in `system.pt_compaction_ingest_nodes` is greater than zero. ``` SELECT level, count(*) AS run_sets FROM system.pt_compaction_run_sets GROUP BY level ORDER BY level ``` For more information, see[Query storage engine tables](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables). 2. **If compaction is falling behind, increase the compactor cache.**Increase `--file-cache-size` on the compactor node, and consider adding memory to the compactor node. 3. **If compaction keeps up and queries still fail, increase the query node cache.**Increase `--file-cache-size` on query nodes only after compaction keeps up with ingest and has no backlog. Query nodes reserve memory for three budgets: * [`--exec-mem-pool-size`](#exec-mem-pool-size) (default `20%`) * `--file-cache-size` (default `20%`) * [`--replica-max-buffer-size`](/influxdb3/enterprise/reference/storage-engine-config-options/#replica-buffer-size)(default 50%) When you increase `--file-cache-size`, lower `--exec-mem-pool-size` or`--replica-max-buffer-size` by the same amount to keep the total unchanged. **Default:** `20%` > [!Note] > `--file-cache-size` was [renamed in 3.11](#name-changes-in-3-11)from `--parquet-mem-cache-size`. > The legacy option and environment variable names are still > accepted–including the pre-3.11 bare-number format, meaning > megabytes–with a startup deprecation warning. | influxdb3 serve option | Environment variables | |--------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| |`--file-cache-size`
`--parquet-mem-cache-size` (pre-3.11 name)|`INFLUXDB3_FILE_CACHE_SIZE`
`INFLUXDB3_PARQUET_MEM_CACHE_SIZE` ([pre-3.11 name](#name-changes-in-3-11))| #### parquet-mem-cache-prune-percentage Specifies the percentage of entries to prune during a prune operation on the in-memory Parquet cache. **Default:** `0.1` | influxdb3 serve option | Environment variable | |--------------------------------------|----------------------------------------------| |`--parquet-mem-cache-prune-percentage`|`INFLUXDB3_PARQUET_MEM_CACHE_PRUNE_PERCENTAGE`| #### parquet-mem-cache-prune-interval Sets the interval to check if the in-memory Parquet cache needs to be pruned. **Default:** `1s` | influxdb3 serve option | Environment variable | |------------------------------------|--------------------------------------------| |`--parquet-mem-cache-prune-interval`|`INFLUXDB3_PARQUET_MEM_CACHE_PRUNE_INTERVAL`| #### file-cache-recency A [duration](/influxdb3/enterprise/reference/glossary/#duration) that specifies the time window for caching recent data files in memory. **Default:** `3d` Only files containing data with a timestamp between `now` and `now - duration`are cached when accessed during queries—for example, with a `5h` setting: * Current time: `2024-06-10 15:00:00` * Cache window: Last 5 hours (`2024-06-10 10:00:00` to now) If a query requests data from `2024-06-09` (old) and `2024-06-10 14:00` (recent): * **Cached**: data files with data from `2024-06-10 14:00` (within 5-hour window) * **Not cached**: data files with data from `2024-06-09` (outside 5-hour window) > [!Note] > `--file-cache-recency` was [renamed in 3.11](#name-changes-in-3-11)from `--parquet-mem-cache-query-path-duration`. > The legacy option and environment variable names are deprecated aliases. | influxdb3 serve option | Environment variables | |--------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| |`--file-cache-recency`
`--parquet-mem-cache-query-path-duration` (pre-3.11 name)|`INFLUXDB3_FILE_CACHE_RECENCY`
`INFLUXDB3_PARQUET_MEM_CACHE_QUERY_PATH_DURATION` ([pre-3.11 name](#name-changes-in-3-11))| #### disable-file-cache Disables the in-memory data file cache. By default, the cache is enabled. This disables data file caching in both the Parquet and upgraded storage engines. > [!Note] > `--disable-file-cache` was [renamed in 3.11](#name-changes-in-3-11)from `--disable-parquet-mem-cache`. > The legacy option and environment variable names (including`disable-data-file-cache`) are deprecated aliases. | influxdb3 serve option | Environment variables | |----------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------| |`--disable-file-cache`
`--disable-parquet-mem-cache` (pre-3.11 name)
`--disable-data-file-cache` (pre-3.11 name)|`INFLUXDB3_DISABLE_FILE_CACHE`
`INFLUXDB3_DISABLE_PARQUET_MEM_CACHE`, `INFLUXDB3_DISABLE_DATA_FILE_CACHE` ([pre-3.11 names](#name-changes-in-3-11))| #### table-index-cache-max-entries Specifies the maximum number of entries in the table index cache. **Default:** `1000` | influxdb3 serve option | Environment variable | |---------------------------------|-----------------------------------------| |`--table-index-cache-max-entries`|`INFLUXDB3_TABLE_INDEX_CACHE_MAX_ENTRIES`| #### table-index-cache-concurrency-limit Limits the concurrency level for table index cache operations. **Default:** `8` | influxdb3 serve option | Environment variable | |---------------------------------------|-----------------------------------------------| |`--table-index-cache-concurrency-limit`|`INFLUXDB3_TABLE_INDEX_CACHE_CONCURRENCY_LIMIT`| #### last-value-cache-disable-from-history Disables populating the last-N-value cache from historical data. If disabled, the cache is still populated with data from the write-ahead log (WAL). | influxdb3 serve option | Environment variable | |-----------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------| |`--last-value-cache-disable-from-history`|`INFLUXDB3_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY`
`INFLUXDB3_ENTERPRISE_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY` ([pre-3.11 name](#name-changes-in-3-11))| #### last-cache-eviction-interval Specifies the interval to evict expired entries from the Last-N-Value cache, expressed as a human-readable duration–for example: `20s`, `1m`, `1h`. **Default:** `10s` | influxdb3 serve option | Environment variable | |--------------------------------|----------------------------------------| |`--last-cache-eviction-interval`|`INFLUXDB3_LAST_CACHE_EVICTION_INTERVAL`| ---------- #### distinct-value-cache-disable-from-history Disables populating the distinct value cache from historical data. If disabled, the cache is still populated with data from the write-ahead log (WAL). | influxdb3 serve option | Environment variable | |---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`--distinct-value-cache-disable-from-history`|`INFLUXDB3_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY`
`INFLUXDB3_ENTERPRISE_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY` ([pre-3.11 name](#name-changes-in-3-11))| #### distinct-cache-eviction-interval Specifies the interval to evict expired entries from the distinct value cache, expressed as a human-readable duration–for example: `20s`, `1m`, `1h`. **Default:** `10s` | influxdb3 serve option | Environment variable | |------------------------------------|--------------------------------------------| |`--distinct-cache-eviction-interval`|`INFLUXDB3_DISTINCT_CACHE_EVICTION_INTERVAL`| #### query-file-limit Limits the number of Parquet files a query can access. If a query attempts to read more than this limit, InfluxDB 3 Enterprise returns an error. |influxdb3 serve option| Environment variable | |----------------------|----------------------------| | `--query-file-limit` |`INFLUXDB3_QUERY_FILE_LIMIT`| ### Processing Engine * [plugin-dir](#plugin-dir) * [plugin-repo](#plugin-repo) * [virtual-env-location](#virtual-env-location) * [package-manager](#package-manager) * [restrict-plugin-triggers-to](#restrict-plugin-triggers-to) * [trigger-retry-max-attempts](#trigger-retry-max-attempts) * [trigger-work-silence-timeout](#trigger-work-silence-timeout) * [processing-engine-restart-state-snapshot-interval](#processing-engine-restart-state-snapshot-interval) * [plugin-dir-only](#plugin-dir-only) #### plugin-dir Specifies the local directory that contains Python plugins and their test files. |influxdb3 serve option| Environment variable | |----------------------|----------------------| | `--plugin-dir` |`INFLUXDB3_PLUGIN_DIR`| ##### Default behavior by deployment type | Deployment |Default state| Configuration | |----------------|-------------|-----------------------------------------| | Docker images | **Enabled** | `INFLUXDB3_PLUGIN_DIR=/plugins` | |DEB/RPM packages| **Enabled** |`plugin-dir="/var/lib/influxdb3/plugins"`| | Binary/source | Disabled | No `plugin-dir` configured | ##### Disable the Processing Engine To disable the Processing Engine, ensure `plugin-dir` is not configured. > [!Warning] > Setting `plugin-dir=""` or `INFLUXDB3_PLUGIN_DIR=""` (empty string) does **not** disable the Processing Engine. > You must comment out, remove, or unset the configuration — not set it to empty. **Docker:** Use `INFLUXDB3_UNSET_VARS` to unset default environment variables that are preconfigured in the container image. `INFLUXDB3_UNSET_VARS` accepts a comma-separated list of environment variable names to unset in the container entrypoint before InfluxDB 3 Enterprise starts. This lets you disable or override image defaults (for example, `INFLUXDB3_PLUGIN_DIR`, logging, or other configuration variables) without modifying the container image itself. To disable the default plugin directory, unset `INFLUXDB3_PLUGIN_DIR`: ```bash docker run -e INFLUXDB3_UNSET_VARS="INFLUXDB3_PLUGIN_DIR" influxdb:3-enterprise ``` **systemd (DEB/RPM):** Comment out or remove `plugin-dir` in the configuration file: ```bash sudo nano /etc/influxdb3/influxdb3-enterprise.conf ``` ```toml # plugin-dir="/var/lib/influxdb3/plugins" ``` Then restart the service: ```bash sudo systemctl restart influxdb3-enterprise ``` When the Processing Engine is disabled: * The Python environment and PyO3 bindings are not initialized * Plugin-related operations return a “No plugin directory configured” error * The server runs with reduced resource usage #### plugin-repo Specifies the base URL of the remote repository used when referencing plugins with the `gh:` prefix. When you create a trigger with a plugin filename starting with `gh:`, InfluxDB fetches the plugin code from this repository URL. The URL construction automatically handles trailing slashes—both formats work identically: * `https://example.com/plugins/` (with trailing slash) * `https://example.com/plugins` (without trailing slash) **Default:** The official InfluxDB 3 plugins repository at `https://raw.githubusercontent.com/influxdata/influxdb3_plugins/main/` |influxdb3 serve option| Environment variable | |----------------------|-----------------------| | `--plugin-repo` |`INFLUXDB3_PLUGIN_REPO`| ##### Example usage ```bash # Use a custom organization repository influxdb3 serve \ --plugin-dir ~/.plugins \ --plugin-repo "https://raw.githubusercontent.com/myorg/influxdb-plugins/main/" # Use an internal mirror influxdb3 serve \ --plugin-dir ~/.plugins \ --plugin-repo "https://internal.company.com/influxdb-plugins/" # Set via environment variable export INFLUXDB3_PLUGIN_REPO="https://custom-repo.example.com/plugins/" influxdb3 serve --plugin-dir ~/.plugins ``` #### virtual-env-location Specifies the location of the Python virtual environment that the processing engine uses. | influxdb3 serve option |Environment variable| |------------------------|--------------------| |`--virtual-env-location`| `VIRTUAL_ENV` | #### package-manager > [!Caution] > #### Deprecated in InfluxDB 3 Enterprise 3.10 > > `--package-manager` is deprecated. > Python and `pip` are bundled with InfluxDB 3 Enterprise, and `pip` is always > used for plugin dependency installation. > The server still starts if you set this option, but prints a deprecation > warning.`disabled` continues to block plugin package installation API calls for > compatibility. Specifies the Python package manager that the Processing Engine uses to install plugin dependencies. This option supports the following values: * `discover` *(default)*: Automatically detect and use available package manager (`uv` or `pip`) * `pip`: Use pip package manager exclusively * `uv`: Use uv package manager exclusively * `disabled`: Disable automatic package installation (all dependencies must be pre-installed) **Default:** `discover` ##### Security mode (disabled) When set to `disabled`, InfluxDB 3 Enterprise blocks all package installation attempts for security and compliance requirements. The Processing Engine and plugins continue to function normally, but package installation requests (via CLI or API) return a `403 Forbidden` error. This mode is designed for: * **Enterprise security requirements**: Prevent arbitrary package installation * **Compliance environments**: Control exactly which packages are available * **Air-gapped deployments**: Pre-install all dependencies before deployment * **Multi-tenant scenarios**: Prevent tenants from installing potentially malicious packages > [!Important] > Before using `--package-manager disabled`, administrators must pre-install all required Python packages into the virtual environment that plugins will use. **Example:** ```bash # Start InfluxDB 3 with disabled package manager influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins \ --package-manager disabled ``` For more information about plugins and package management, see [Processing Engine plugins](/influxdb3/enterprise/plugins/). |influxdb3 serve option| Environment variable | |----------------------|---------------------------| | `--package-manager` |`INFLUXDB3_PACKAGE_MANAGER`| #### restrict-plugin-triggers-to Restrict plugin triggers to one or more trigger types. Provide one or more of `wal`, `schedule`, or `request`. | influxdb3 serve option | Environment variable | |-------------------------------|---------------------------------------| |`--restrict-plugin-triggers-to`|`INFLUXDB3_RESTRICT_PLUGIN_TRIGGERS_TO`| #### trigger-retry-max-attempts Sets the maximum number of attempts for a failed trigger invocation before it stops retrying. Higher values keep retrying transient failures longer, but can retain failing work and delay newer invocations. Must be greater than zero. **Default:** `5` | influxdb3 serve option | Environment variable | |------------------------------|--------------------------------------| |`--trigger-retry-max-attempts`|`INFLUXDB3_TRIGGER_RETRY_MAX_ATTEMPTS`| #### trigger-work-silence-timeout Sets how long a trigger worker can go without reporting progress before its work becomes eligible for recovery by another process node. Smaller values reduce recovery time after a hard restart, but increase the chance of duplicate executions during slow or partitioned operation. Configure the same value on every [process](#mode) node. Must be at least `1s`. **Default:** `60s` | influxdb3 serve option | Environment variable | |--------------------------------|----------------------------------------| |`--trigger-work-silence-timeout`|`INFLUXDB3_TRIGGER_WORK_SILENCE_TIMEOUT`| #### processing-engine-restart-state-snapshot-interval Sets the interval between Processing Engine restart-state checkpoints. Smaller values reduce the amount of scheduled work that a hard restart must replay, at the cost of more object store writes. Must be greater than zero. **Default:** `1s` | influxdb3 serve option | Environment variable | |-----------------------------------------------------|-------------------------------------------------------------| |`--processing-engine-restart-state-snapshot-interval`|`INFLUXDB3_PROCESSING_ENGINE_RESTART_STATE_SNAPSHOT_INTERVAL`| #### plugin-dir-only Only allow plugins that already exist in the configured plugin directory. Blocks plugin installation from any other source. |influxdb3 serve option| Environment variable | |----------------------|---------------------------| | `--plugin-dir-only` |`INFLUXDB3_PLUGIN_DIR_ONLY`| ### Bulk Import * [import-attempt-server-side-copy](#import-attempt-server-side-copy) #### import-attempt-server-side-copy Applies to `influxdb3 import from-object-store` bulk imports that copy Parquet files from a bucket other than the server’s own object store. When enabled, InfluxDB 3 Enterprise first attempts a true S3 server-side copy, which works only when all of the following hold: * The source and destination object stores are both S3 (or an S3-compatible endpoint set with [`--aws-endpoint`](#aws-endpoint)). * This server’s credentials can read the source bucket and write the destination bucket. This is true within one AWS account, or across accounts only when a bucket policy grants that access. * The source file is under 5 GiB. When any condition doesn’t hold, or when this option is `false`, the import falls back to a streamed copy through this server, which still succeeds but moves the file’s bytes through the server. This option applies to the upgraded storage engine only. **Default:** `true` | influxdb3 serve option | Environment variable | |-----------------------------------|-------------------------------------------| |`--import-attempt-server-side-copy`|`INFLUXDB3_IMPORT_ATTEMPT_SERVER_SIDE_COPY`| ### Cluster Management * [replication-interval](#replication-interval) * [catalog-sync-interval](#catalog-sync-interval) * [wait-for-running-ingester](#wait-for-running-ingester) (`--wait-for-running-ingestor` before 3.11) * [conn-info](#conn-info) #### replication-interval Specifies the interval at which data replication occurs between cluster nodes. **Default:** `250ms` | influxdb3 serve option | Environment variable | |------------------------|---------------------------------------------------------------------------------------------------------------------------| |`--replication-interval`|`INFLUXDB3_REPLICATION_INTERVAL`
`INFLUXDB3_ENTERPRISE_REPLICATION_INTERVAL` ([pre-3.11 name](#name-changes-in-3-11))| #### catalog-sync-interval Defines how often the catalog synchronizes across cluster nodes. **Default:** `1s` | influxdb3 serve option | Environment variable | |-------------------------|-----------------------------------------------------------------------------------------------------------------------------| |`--catalog-sync-interval`|`INFLUXDB3_CATALOG_SYNC_INTERVAL`
`INFLUXDB3_ENTERPRISE_CATALOG_SYNC_INTERVAL` ([pre-3.11 name](#name-changes-in-3-11))| #### wait-for-running-ingester Specifies how long to wait for a running ingester during startup. **Default:** `10s` > [!Note] > `--wait-for-running-ingester` was [renamed in 3.11](#name-changes-in-3-11)from `--wait-for-running-ingestor` (misspelled). > The legacy option and environment variable names are deprecated aliases. | influxdb3 serve option | Environment variables | |---------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`--wait-for-running-ingester`
`--wait-for-running-ingestor` (pre-3.11 name)|`INFLUXDB3_WAIT_FOR_RUNNING_INGESTER`
`INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR`, `INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTER`, `INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTOR` ([pre-3.11 names](#name-changes-in-3-11))| #### conn-info In multi-node deployments, specifies the connection information used to reach the ingester over the internode gRPC port (not the HTTP port). Required for Processing Engine plugin writes from non-ingester nodes, and used together with the `--internode-bind-addr` option. |influxdb3 serve option| Environment variables | |----------------------|-----------------------------------------------------------------------------------------------------| | `--conn-info` |`INFLUXDB3_CONN_INFO`
`INFLUXDB3_ENTERPRISE_CONN_INFO` ([pre-3.11 name](#name-changes-in-3-11))| ### Web UI Configure the [integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/), which the server hosts in-process as a WebAssembly (WASM) guest. The web UI is off unless you add `webui` to [`--mode`](#mode). * [webui-session-secret](#webui-session-secret) * [webui-public-uri](#webui-public-uri) * [webui-cookie-secure](#webui-cookie-secure) * [webui-openai-base-url](#webui-openai-base-url) #### webui-session-secret Specifies the secret that signs web UI session cookies. The secret doesn’t encrypt the cookies. Keep it private: anyone who has it can forge session cookies. Required whenever [`--mode`](#mode) includes `webui`–the server doesn’t start without it. Use a secret that’s unique to the cluster. Generate a secret with `openssl rand -base64 24`, then pass the same value on every start and on every node that runs `webui` mode. A secret that changes between restarts signs out every user. | influxdb3 serve option | Environment variable | |------------------------|--------------------------------| |`--webui-session-secret`|`INFLUXDB3_WEBUI_SESSION_SECRET`| #### webui-public-uri Sets the browser-reachable base URI of the embedded Web UI. InfluxDB 3 Enterprise derives the OAuth callback URI as`/auth/callback`. Requires [`--oauth-client-id`](#oauth-client-id). |influxdb3 serve option| Environment variable | |----------------------|----------------------------| | `--webui-public-uri` |`INFLUXDB3_WEBUI_PUBLIC_URI`| #### webui-cookie-secure Sets the `Secure` attribute on web UI session cookies, which tells browsers to send the cookies only over HTTPS. **Default:** `false` Enable this option when browsers reach Explorer over HTTPS–for example, through a TLS-terminating reverse proxy. Leave it disabled when you reach Explorer over HTTP, such as through`http://localhost:8181/`; browsers withhold `Secure` cookies from HTTP requests, so sessions don’t persist. |influxdb3 serve option | Environment variable | |-----------------------|-------------------------------| |`--webui-cookie-secure`|`INFLUXDB3_WEBUI_COOKIE_SECURE`| #### webui-openai-base-url Specifies the base URL of the OpenAI-compatible endpoint that the web UI AI chat sends requests to. Chat prompts, and any query results included with them, go to this endpoint. | influxdb3 serve option | Environment variable | |-------------------------|---------------------------------| |`--webui-openai-base-url`|`INFLUXDB3_WEBUI_OPENAI_BASE_URL`| ### Resource Limits * [num-cores](#num-cores) * [max-concurrent-queries](#max-concurrent-queries) * [datafusion-num-threads](#datafusion-num-threads) * *[num-io-threads](#num-io-threads) - See [Global configuration options](#global-configuration-options)* * [num-database-limit](#num-database-limit) * [num-table-limit](#num-table-limit) * [num-total-columns-per-table-limit](#num-total-columns-per-table-limit) #### num-cores Limits the number of CPU cores that the InfluxDB 3 Enterprise process can use when running on systems where resources are shared. **Default:** All available cores on the system Maximum cores allowed is determined by your InfluxDB 3 Enterprise license: * **Trial**: up to 256 cores * **At-Home**: 2 cores * **Commercial**: per contract When specified, InfluxDB automatically assigns the number of DataFusion threads and IO threads based on the core count. **Default thread assignment logic when `num-cores` is set:** * **1-2 cores**: 1 IO thread, 1 DataFusion thread * **3 cores**: 1 IO thread, 2 DataFusion threads * **4+ cores**: 2 IO threads, (n-2) DataFusion threads This automatic allocation applies when you don’t explicitly set [`--num-io-threads`](#num-io-threads) and [`--datafusion-num-threads`](#datafusion-num-threads). > [!Note] > You can override the automatic thread assignment by explicitly setting [`--num-io-threads`](#num-io-threads)and [`--datafusion-num-threads`](#datafusion-num-threads). > This is particularly important for specialized > workloads like [ingest mode](#mode) where you may need more IO threads than the default allocation. > [!Note] > #### Thread defaults on the upgraded storage engine > > The default thread assignment logic above applies to Parquet-engine > clusters. > On clusters running the upgraded storage engine (the default for new > clusters), the IO and DataFusion > runtimes each default to the licensed core count, and the node consumes the > licensed core count regardless of thread configuration. > Thread counts set above the licensed core count are capped with a startup > warning instead of rejected, and the combined IO + DataFusion total is not > checked against `num-cores`. > An at-home license always runs 1 IO thread and 1 DataFusion thread. **Constraints (Parquet-engine clusters):** * Must be at least 2 * Cannot exceed the number of cores available on the system * Total thread count from `--num-io-threads` (global option) and `--datafusion-num-threads` cannot exceed the `num-cores` value |influxdb3 serve option| Environment variable | |----------------------|-----------------------------------------------------------------------------------------------------| | `--num-cores` |`INFLUXDB3_NUM_CORES`
`INFLUXDB3_ENTERPRISE_NUM_CORES` ([pre-3.11 name](#name-changes-in-3-11))| #### max-concurrent-queries Limits the number of queries that can run concurrently. Queries beyond the limit wait in a queue until a running query completes. To read, change, or reset the limit at runtime, send a `GET`, `PUT`, or `DELETE`request to `/api/v3/configure/query_concurrency_limit`.`DELETE` restores the limit set at startup. **Default:** `max(50, 4 x P)`, where `P` is the effective query parallelism: the smaller of the number of CPU cores (the licensed core count on InfluxDB 3 Enterprise) and [`--datafusion-num-threads`](#datafusion-num-threads). > [!Note] > #### In v3.12+: finite default, and queuing > > `--max-concurrent-queries` now has a finite default instead of being > effectively unlimited, and queries beyond the limit queue for admission > instead of always running immediately. > If you set an explicit value below `max(16, P)`, the server logs a > startup warning recommending you raise it. Limits that low mostly > serialize query execution and suit testing, not production. | influxdb3 serve option | Environment variable | |--------------------------|----------------------------------| |`--max-concurrent-queries`|`INFLUXDB3_MAX_CONCURRENT_QUERIES`| #### datafusion-num-threads Sets the number of threads allocated to the DataFusion runtime thread pool. DataFusion threads handle: * Query execution and processing * Data aggregation and transformation * Snapshot creation (sort/dedupe operations) * Parquet file generation **Default:** * Clusters running the upgraded storage engine (the default for new clusters): the licensed core count * Parquet-engine clusters, if `--num-cores` is not set: all available cores minus IO threads * Parquet-engine clusters, if `--num-cores` is set: automatically determined based on core count (see [`--num-cores`](#num-cores)) > [!Note] > DataFusion threads are used for both query processing and snapshot operations. > Even ingest-only nodes use DataFusion threads during WAL snapshot creation. **Constraints:** On Parquet-engine clusters, when used with `--num-cores`, the sum of `--num-io-threads` and `--datafusion-num-threads` cannot exceed the `num-cores` value. On clusters running the upgraded storage engine, values above the licensed core count are capped with a startup warning. | influxdb3 serve option | Environment variable | |--------------------------|----------------------------------| |`--datafusion-num-threads`|`INFLUXDB3_DATAFUSION_NUM_THREADS`| > [!Note] > [`--num-io-threads`](#num-io-threads) is a [global configuration option](#global-configuration-options). ---------- > [!Note] > #### Parquet-engine limits > > `num-database-limit`, `num-table-limit`, and`num-total-columns-per-table-limit` apply to the Parquet engine only > (clusters that started on 3.10 or earlier that have not run the[storage engine upgrade](#upgrade-pacha-tree)). > On clusters running the upgraded storage engine, explicitly setting these > options logs a startup > warning and has no effect.`--max-total-columns` (documented in the[storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/)) > is the upgraded storage engine’s counterpart of`--num-total-columns-per-table-limit`; > the database and table limits have no equivalent on the upgraded engine. #### num-database-limit Limits the total number of active databases. Default is 100. |influxdb3 serve option| Environment variable | |----------------------|-----------------------------------------------------------------------------------------------------------------------| |`--num-database-limit`|`INFLUXDB3_NUM_DATABASE_LIMIT`
`INFLUXDB3_ENTERPRISE_NUM_DATABASE_LIMIT` ([pre-3.11 name](#name-changes-in-3-11))| #### num-table-limit Limits the total number of active tables across all databases. Default is 10000. |influxdb3 serve option| Environment variable | |----------------------|-----------------------------------------------------------------------------------------------------------------| | `--num-table-limit` |`INFLUXDB3_NUM_TABLE_LIMIT`
`INFLUXDB3_ENTERPRISE_NUM_TABLE_LIMIT` ([pre-3.11 name](#name-changes-in-3-11))| #### num-total-columns-per-table-limit Limits the total number of columns per table. Default is 500. | influxdb3 serve option | Environment variable | |-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------| |`--num-total-columns-per-table-limit`|`INFLUXDB3_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT`
`INFLUXDB3_ENTERPRISE_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT` ([pre-3.11 name](#name-changes-in-3-11))| ### Data Lifecycle Management * [gen1-lookback-duration](#gen1-lookback-duration) * [retention-check-interval](#retention-check-interval) * [delete-grace-period](#delete-grace-period) * [hard-delete-default-duration](#hard-delete-default-duration) #### gen1-lookback-duration Specifies how far back to look when creating generation 1 Parquet files. **Default:** `24h` | influxdb3 serve option | Environment variable | |--------------------------|----------------------------------| |`--gen1-lookback-duration`|`INFLUXDB3_GEN1_LOOKBACK_DURATION`| #### retention-check-interval The interval at which retention policies are checked and enforced. Enter as a human-readable time–for example: `30m` or `1h`. **Default:** `30m` | influxdb3 serve option | Environment variable | |----------------------------|------------------------------------| |`--retention-check-interval`|`INFLUXDB3_RETENTION_CHECK_INTERVAL`| #### delete-grace-period Specifies the grace period before permanently deleting data. **Default:** `24h` |influxdb3 serve option | Environment variable | |-----------------------|-------------------------------| |`--delete-grace-period`|`INFLUXDB3_DELETE_GRACE_PERIOD`| #### hard-delete-default-duration > [!Warning] > #### Deprecated: has no effect > > `--hard-delete-default-duration` has never had an effect in any release; > the server always uses the built-in default hard-delete duration. > The option is still accepted so existing configurations start, but the > server logs a startup warning–remove it from your configuration. | influxdb3 serve option | Environment variable | |--------------------------------|----------------------------------------| |`--hard-delete-default-duration`|`INFLUXDB3_HARD_DELETE_DEFAULT_DURATION`| ### Telemetry * [disable-telemetry-upload](#disable-telemetry-upload) * [telemetry-endpoint](#telemetry-endpoint) #### disable-telemetry-upload Disables the upload of telemetry data to InfluxData. **Default:** `false` | influxdb3 serve option | Environment variables | |----------------------------|------------------------------------------------------------------------------------------------------------------------| |`--disable-telemetry-upload`|`INFLUXDB3_DISABLE_TELEMETRY_UPLOAD`
`INFLUXDB3_TELEMETRY_DISABLE_UPLOAD` ([pre-3.11 name](#name-changes-in-3-11))| #### telemetry-endpoint Specifies the endpoint for telemetry data uploads. |influxdb3 serve option| Environment variable | |----------------------|------------------------------| |`--telemetry-endpoint`|`INFLUXDB3_TELEMETRY_ENDPOINT`| ### TCP Listeners * [tcp-listener-file-path](#tcp-listener-file-path) * [admin-token-recovery-tcp-listener-file-path](#admin-token-recovery-tcp-listener-file-path) #### tcp-listener-file-path Specifies the file path for the TCP listener configuration. | influxdb3 serve option | Environment variables | |--------------------------|-------------------------------------------------------------------------------------------------------------------------------| |`--tcp-listener-file-path`|`INFLUXDB3_TCP_LISTENER_FILE_PATH`
`INFLUXDB3_TCP_LISTINER_FILE_PATH` ([misspelled pre-3.11 name](#name-changes-in-3-11))| #### admin-token-recovery-tcp-listener-file-path Specifies the TCP listener file path for admin token recovery operations. | influxdb3 serve option | Environment variable | |-----------------------------------------------|-------------------------------------------------------| |`--admin-token-recovery-tcp-listener-file-path`|`INFLUXDB3_ADMIN_TOKEN_RECOVERY_TCP_LISTENER_FILE_PATH`| ## Configuration name changes in 3.11 InfluxDB 3 Enterprise 3.11.0 renamed many `influxdb3 serve` options and environment variables so that each variable matches its command option. In addition, Enterprise-specific variables dropped the `ENTERPRISE_` segment—for example, `INFLUXDB3_ENTERPRISE_LICENSE_EMAIL`is now `INFLUXDB3_LICENSE_EMAIL`. On 3.11.0 and later, legacy names remain supported as deprecated aliases: the server logs a deprecation warning at startup when it detects one, and if both the new and the legacy name are set, the new name takes precedence. > [!Warning] > #### New names don’t work on releases earlier than 3.11.0 > > A pre-3.11 server doesn’t recognize names introduced by these 3.11 > renames: it exits with an error on an unknown command line option, but it**silently ignores** an unknown environment variable. > If you run a release earlier than 3.11.0, use the legacy names, and verify > in your startup logs that settings take effect. Use the following tables to migrate a deployment configuration—for example, Helm values or a systemd environment file—in one pass. ### Renamed options (legacy names aliased) The following `influxdb3 serve` options and their environment variables were renamed. Legacy names are still accepted; the server logs a deprecation warning at startup, and values in the old format are honored. | Legacy name | New name | |-----------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| |`--disable-parquet-mem-cache`
`INFLUXDB3_DISABLE_PARQUET_MEM_CACHE`
`--disable-data-file-cache`
`INFLUXDB3_DISABLE_DATA_FILE_CACHE`| `--disable-file-cache`
`INFLUXDB3_DISABLE_FILE_CACHE` | | `--exec-mem-pool-bytes`
`INFLUXDB3_EXEC_MEM_POOL_BYTES` | `--exec-mem-pool-size`
`INFLUXDB3_EXEC_MEM_POOL_SIZE` | | `--force-snapshot-mem-threshold`
`INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD` | `--force-snapshot-mem-size`
`INFLUXDB3_FORCE_SNAPSHOT_MEM_SIZE` | | `--parquet-mem-cache-query-path-duration`
`INFLUXDB3_PARQUET_MEM_CACHE_QUERY_PATH_DURATION` | `--file-cache-recency`
`INFLUXDB3_FILE_CACHE_RECENCY` | | `--parquet-mem-cache-size`
`INFLUXDB3_PARQUET_MEM_CACHE_SIZE` | `--file-cache-size`
`INFLUXDB3_FILE_CACHE_SIZE` | | `--query-log-size`
`INFLUXDB3_QUERY_LOG_SIZE` | `--query-log-max-entries`
`INFLUXDB3_QUERY_LOG_MAX_ENTRIES` | | `--wal-max-write-buffer-size`
`INFLUXDB3_WAL_MAX_WRITE_BUFFER_SIZE` | `--wal-max-buffered-writes`
`INFLUXDB3_WAL_MAX_BUFFERED_WRITES` | | `--wal-snapshot-size`
`INFLUXDB3_WAL_SNAPSHOT_SIZE` | `--wal-files-per-snapshot`
`INFLUXDB3_WAL_FILES_PER_SNAPSHOT` | | `--wait-for-running-ingestor`
`INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR` |`--wait-for-running-ingester`
`INFLUXDB3_WAIT_FOR_RUNNING_INGESTER`| ### Renamed environment variables (legacy names aliased) The following environment variables were renamed to match their command options; the option names are unchanged. Legacy names remain supported as deprecated aliases. | Legacy name | New name | |------------------------------------|-----------------------------------------| | `INFLUXDB3_DB_DIR` | `INFLUXDB3_DATA_DIR` | |`INFLUXDB3_NODE_IDENTIFIER_FROM_ENV`| `INFLUXDB3_NODE_ID_FROM_ENV` | | `INFLUXDB3_NODE_IDENTIFIER_PREFIX` | `INFLUXDB3_NODE_ID` | | `INFLUXDB3_NUM_WAL_FILES_TO_KEEP` |`INFLUXDB3_SNAPSHOTTED_WAL_FILES_TO_KEEP`| | `INFLUXDB3_START_WITHOUT_AUTH` | `INFLUXDB3_WITHOUT_AUTH` | | `INFLUXDB3_TCP_LISTINER_FILE_PATH` | `INFLUXDB3_TCP_LISTENER_FILE_PATH` | |`INFLUXDB3_TELEMETRY_DISABLE_UPLOAD`| `INFLUXDB3_DISABLE_TELEMETRY_UPLOAD` | ### Renamed Enterprise environment variables (legacy names aliased) Enterprise-specific environment variables dropped the `ENTERPRISE_` segment. Legacy names remain supported as deprecated aliases. | Legacy name | New name | |----------------------------------------------------------------|-----------------------------------------------------| | `INFLUXDB3_ENTERPRISE_API_UPLOAD_CHECK_INTERVAL` | `INFLUXDB3_API_UPLOAD_CHECK_INTERVAL` | | `INFLUXDB3_ENTERPRISE_API_UPLOAD_CONCURRENT_STATUS_READS` | `INFLUXDB3_API_UPLOAD_CONCURRENT_STATUS_READS` | | `INFLUXDB3_ENTERPRISE_CATALOG_SYNC_INTERVAL` | `INFLUXDB3_CATALOG_SYNC_INTERVAL` | | `INFLUXDB3_ENTERPRISE_CLUSTER_ID` | `INFLUXDB3_CLUSTER_ID` | | `INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT` | `INFLUXDB3_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT` | | `INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX` | `INFLUXDB3_COMPACTED_DATA_SKIP_FILE_INDEX` | | `INFLUXDB3_ENTERPRISE_COMPACTION_CHECK_INTERVAL` | `INFLUXDB3_COMPACTION_CHECK_INTERVAL` | | `INFLUXDB3_ENTERPRISE_COMPACTION_CLEANUP_WAIT` | `INFLUXDB3_COMPACTION_CLEANUP_WAIT` | | `INFLUXDB3_ENTERPRISE_COMPACTION_GEN2_DURATION` | `INFLUXDB3_COMPACTION_GEN2_DURATION` | | `INFLUXDB3_ENTERPRISE_COMPACTION_MAX_NUM_FILES_PER_PLAN` | `INFLUXDB3_COMPACTION_MAX_NUM_FILES_PER_PLAN` | | `INFLUXDB3_ENTERPRISE_COMPACTION_MULTIPLIERS` | `INFLUXDB3_COMPACTION_MULTIPLIERS` | | `INFLUXDB3_ENTERPRISE_COMPACTION_ROW_LIMIT` | `INFLUXDB3_COMPACTION_ROW_LIMIT` | | `INFLUXDB3_ENTERPRISE_COMPACTION_SNAPSHOTS_LIMIT` | `INFLUXDB3_COMPACTION_SNAPSHOTS_LIMIT` | | `INFLUXDB3_ENTERPRISE_CONN_INFO` | `INFLUXDB3_CONN_INFO` | |`INFLUXDB3_ENTERPRISE_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY`|`INFLUXDB3_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY`| | `INFLUXDB3_ENTERPRISE_INTERNODE_BIND_ADDR` | `INFLUXDB3_INTERNODE_BIND_ADDR` | | `INFLUXDB3_ENTERPRISE_INTERNODE_JWT_PRIMARY` | `INFLUXDB3_INTERNODE_JWT_PRIMARY` | | `INFLUXDB3_ENTERPRISE_INTERNODE_JWT_SECONDARY` | `INFLUXDB3_INTERNODE_JWT_SECONDARY` | | `INFLUXDB3_ENTERPRISE_INTERNODE_TLS_CA` | `INFLUXDB3_INTERNODE_TLS_CA` | | `INFLUXDB3_ENTERPRISE_INTERNODE_TLS_CERT` | `INFLUXDB3_INTERNODE_TLS_CERT` | | `INFLUXDB3_ENTERPRISE_INTERNODE_TLS_KEY` | `INFLUXDB3_INTERNODE_TLS_KEY` | | `INFLUXDB3_ENTERPRISE_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY` | `INFLUXDB3_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY` | | `INFLUXDB3_ENTERPRISE_LICENSE_EMAIL` | `INFLUXDB3_LICENSE_EMAIL` | | `INFLUXDB3_ENTERPRISE_LICENSE_FILE` | `INFLUXDB3_LICENSE_FILE` | | `INFLUXDB3_ENTERPRISE_LICENSE_TYPE` | `INFLUXDB3_LICENSE_TYPE` | | `INFLUXDB3_ENTERPRISE_MODE` | `INFLUXDB3_MODE` | | `INFLUXDB3_ENTERPRISE_NUM_CORES` | `INFLUXDB3_NUM_CORES` | | `INFLUXDB3_ENTERPRISE_NUM_DATABASE_LIMIT` | `INFLUXDB3_NUM_DATABASE_LIMIT` | | `INFLUXDB3_ENTERPRISE_NUM_TABLE_LIMIT` | `INFLUXDB3_NUM_TABLE_LIMIT` | | `INFLUXDB3_ENTERPRISE_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT` | `INFLUXDB3_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT` | | `INFLUXDB3_ENTERPRISE_REPLICATION_INTERVAL` | `INFLUXDB3_REPLICATION_INTERVAL` | | `INFLUXDB3_ENTERPRISE_USE_PACHA_TREE` | `INFLUXDB3_USE_PACHA_TREE` | | `INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_COMPACTOR` | `INFLUXDB3_WAIT_FOR_RUNNING_COMPACTOR` | | `INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTER` | `INFLUXDB3_WAIT_FOR_RUNNING_INGESTER` | | `INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTOR` | `INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR` | ### Removed pt- option names (no aliases) In InfluxDB 3 Enterprise 3.11+, options for the upgraded storage engine dropped the `pt-`prefix without backward compatibility: old `--pt-*` flags cause a startup error, and legacy `INFLUXDB3_PT_*` and `INFLUXDB3_ENTERPRISE_PT_*`environment variables are ignored (the server logs a warning at startup for each one that is still set). For the complete old-to-new name table, see[Migrate from `--pt-*` option names](/influxdb3/enterprise/reference/storage-engine-config-options/#migrate-from-pt-option-names). | influxdb3 option | Environment variable | | --- | --- | | influxdb3 option | Environment variable | | --num-io-threads | INFLUXDB3_NUM_IO_THREADS | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --cluster-id | INFLUXDB3_CLUSTER_ID INFLUXDB3_ENTERPRISE_CLUSTER_ID ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --data-dir | INFLUXDB3_DATA_DIR INFLUXDB3_DB_DIR ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --mode | INFLUXDB3_MODE INFLUXDB3_ENTERPRISE_MODE ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --node-id | INFLUXDB3_NODE_ID INFLUXDB3_NODE_IDENTIFIER_PREFIX ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --node-id-from-env | INFLUXDB3_NODE_ID_FROM_ENV INFLUXDB3_NODE_IDENTIFIER_FROM_ENV ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --object-store | INFLUXDB3_OBJECT_STORE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --upgrade-pacha-tree | INFLUXDB3_UPGRADE_PACHA_TREE | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --license-email | INFLUXDB3_LICENSE_EMAIL INFLUXDB3_ENTERPRISE_LICENSE_EMAIL ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --license-file | INFLUXDB3_LICENSE_FILE INFLUXDB3_ENTERPRISE_LICENSE_FILE ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --license-type | INFLUXDB3_LICENSE_TYPE INFLUXDB3_ENTERPRISE_LICENSE_TYPE ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --tls-key | INFLUXDB3_TLS_KEY | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --tls-cert | INFLUXDB3_TLS_CERT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --tls-minimum-version | INFLUXDB3_TLS_MINIMUM_VERSION | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --without-auth | INFLUXDB3_WITHOUT_AUTH INFLUXDB3_START_WITHOUT_AUTH ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --disable-authz | INFLUXDB3_DISABLE_AUTHZ | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --admin-token-recovery-http-bind | INFLUXDB3_ADMIN_TOKEN_RECOVERY_HTTP_BIND_ADDR | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --admin-token-file | INFLUXDB3_ADMIN_TOKEN_FILE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --permission-tokens-file | INFLUXDB3_PERMISSION_TOKENS_FILE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --user-auth-type | INFLUXDB3_USER_AUTH_TYPE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --without-user-auth | INFLUXDB3_WITHOUT_USER_AUTH | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --jwt-key-id | INFLUXDB3_JWT_KEY_ID | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --jwt-private-key | INFLUXDB3_JWT_PRIVATE_KEY | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --jwt-issuer | INFLUXDB3_JWT_ISSUER | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --jwt-default-ttl-seconds | INFLUXDB3_JWT_DEFAULT_TTL_SECONDS | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --oauth-issuer | INFLUXDB3_OAUTH_ISSUER | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --oauth-audience | INFLUXDB3_OAUTH_AUDIENCE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --oauth-client-id | INFLUXDB3_OAUTH_CLIENT_ID | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --oauth-scopes | INFLUXDB3_OAUTH_SCOPES | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --rbac-authoring-disabled | INFLUXDB3_RBAC_AUTHORING_DISABLED | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-access-key-id | AWS_ACCESS_KEY_ID | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-secret-access-key | AWS_SECRET_ACCESS_KEY | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-default-region | AWS_DEFAULT_REGION | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-endpoint | AWS_ENDPOINT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-session-token | AWS_SESSION_TOKEN | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-allow-http | AWS_ALLOW_HTTP | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-skip-signature | AWS_SKIP_SIGNATURE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --aws-credentials-file | AWS_CREDENTIALS_FILE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --google-service-account | GOOGLE_SERVICE_ACCOUNT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --azure-storage-account | AZURE_STORAGE_ACCOUNT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --azure-storage-access-key | AZURE_STORAGE_ACCESS_KEY | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --azure-endpoint | AZURE_ENDPOINT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --azure-allow-http | AZURE_ALLOW_HTTP | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --bucket | INFLUXDB3_BUCKET | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-connection-limit | INFLUXDB3_OBJECT_STORE_CONNECTION_LIMIT (preferred) OBJECT_STORE_CONNECTION_LIMIT (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-http2-only | INFLUXDB3_OBJECT_STORE_HTTP2_ONLY (preferred) OBJECT_STORE_HTTP2_ONLY (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-http2-max-frame-size | INFLUXDB3_OBJECT_STORE_HTTP2_MAX_FRAME_SIZE (preferred) OBJECT_STORE_HTTP2_MAX_FRAME_SIZE (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-request-timeout | INFLUXDB3_OBJECT_STORE_REQUEST_TIMEOUT (preferred) OBJECT_STORE_REQUEST_TIMEOUT (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-max-retries | INFLUXDB3_OBJECT_STORE_MAX_RETRIES (preferred) OBJECT_STORE_MAX_RETRIES (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-retry-timeout | INFLUXDB3_OBJECT_STORE_RETRY_TIMEOUT (preferred) OBJECT_STORE_RETRY_TIMEOUT (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-tls-allow-insecure | INFLUXDB3_OBJECT_STORE_TLS_ALLOW_INSECURE (preferred) OBJECT_STORE_TLS_ALLOW_INSECURE (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --object-store-tls-ca | INFLUXDB3_OBJECT_STORE_TLS_CA (preferred) OBJECT_STORE_TLS_CA (deprecated; supported for backward compatibility) | | Level | Description | | --- | --- | | Level | Description | | error | Only errors | | warn | Warnings and errors | | info | Informational messages, warnings, and errors (default) | | debug | Debug information for troubleshooting, plus all above levels | | trace | Very detailed tracing information, plus all above levels (produces high log volume) | | Component | Description | | --- | --- | | Component | Description | | influxdb3_write_buffer | Write buffer operations | | influxdb3_wal | Write-ahead log operations | | influxdb3_catalog | Catalog and schema operations | | influxdb3_cache | Caching operations | | Component | Description | | --- | --- | | Component | Description | | influxdb3_enterprise | Enterprise-specific features | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --log-filter | INFLUXDB3_LOG_FILTER (preferred) LOG_FILTER (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --log-destination | INFLUXDB3_LOG_DESTINATION (preferred) LOG_DESTINATION (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --log-format | INFLUXDB3_LOG_FORMAT (preferred) LOG_FORMAT (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --query-log-max-entries --query-log-size (pre-3.11 name) | INFLUXDB3_QUERY_LOG_MAX_ENTRIES INFLUXDB3_QUERY_LOG_SIZE ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-exporter | INFLUXDB3_TRACES_EXPORTER (preferred) TRACES_EXPORTER (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-exporter-jaeger-agent-host | INFLUXDB3_TRACES_EXPORTER_JAEGER_AGENT_HOST (preferred) TRACES_EXPORTER_JAEGER_AGENT_HOST (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-exporter-jaeger-agent-port | INFLUXDB3_TRACES_EXPORTER_JAEGER_AGENT_PORT (preferred) TRACES_EXPORTER_JAEGER_AGENT_PORT (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-exporter-jaeger-service-name | INFLUXDB3_TRACES_EXPORTER_JAEGER_SERVICE_NAME (preferred) TRACES_EXPORTER_JAEGER_SERVICE_NAME (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-exporter-jaeger-trace-context-header-name | INFLUXDB3_TRACES_EXPORTER_JAEGER_TRACE_CONTEXT_HEADER_NAME (preferred) TRACES_EXPORTER_JAEGER_TRACE_CONTEXT_HEADER_NAME (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-jaeger-debug-name | INFLUXDB3_TRACES_EXPORTER_JAEGER_DEBUG_NAME (preferred) TRACES_EXPORTER_JAEGER_DEBUG_NAME (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-jaeger-tags | INFLUXDB3_TRACES_EXPORTER_JAEGER_TAGS (preferred) TRACES_EXPORTER_JAEGER_TAGS (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --traces-jaeger-max-msgs-per-second | INFLUXDB3_TRACES_JAEGER_MAX_MSGS_PER_SECOND (preferred) TRACES_JAEGER_MAX_MSGS_PER_SECOND (deprecated; supported for backward compatibility) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --datafusion-num-threads | INFLUXDB3_DATAFUSION_NUM_THREADS | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --datafusion-max-parquet-fanout | INFLUXDB3_DATAFUSION_MAX_PARQUET_FANOUT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --datafusion-use-cached-parquet-loader | INFLUXDB3_DATAFUSION_USE_CACHED_PARQUET_LOADER | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --datafusion-config | INFLUXDB3_DATAFUSION_CONFIG | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --max-http-request-size | INFLUXDB3_MAX_HTTP_REQUEST_SIZE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --http-bind | INFLUXDB3_HTTP_BIND_ADDR | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --exec-mem-pool-size --exec-mem-pool-bytes (pre-3.11 name) | INFLUXDB3_EXEC_MEM_POOL_SIZE INFLUXDB3_EXEC_MEM_POOL_BYTES ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --force-snapshot-mem-size --force-snapshot-mem-threshold (pre-3.11 name) | INFLUXDB3_FORCE_SNAPSHOT_MEM_SIZE INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --force-snapshot-max-age | INFLUXDB3_FORCE_SNAPSHOT_MAX_AGE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --checkpoint-interval | INFLUXDB3_CHECKPOINT_INTERVAL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --wal-flush-interval | INFLUXDB3_WAL_FLUSH_INTERVAL | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --wal-files-per-snapshot --wal-snapshot-size (pre-3.11 name) | INFLUXDB3_WAL_FILES_PER_SNAPSHOT INFLUXDB3_WAL_SNAPSHOT_SIZE ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --wal-max-buffered-writes --wal-max-write-buffer-size (pre-3.11 name) | INFLUXDB3_WAL_MAX_BUFFERED_WRITES INFLUXDB3_WAL_MAX_WRITE_BUFFER_SIZE ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --snapshotted-wal-files-to-keep | INFLUXDB3_SNAPSHOTTED_WAL_FILES_TO_KEEP INFLUXDB3_NUM_WAL_FILES_TO_KEEP ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --wal-replay-fail-on-error | INFLUXDB3_WAL_REPLAY_FAIL_ON_ERROR | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --wal-replay-concurrency-limit | INFLUXDB3_WAL_REPLAY_CONCURRENCY_LIMIT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --write-timestamp-max-past | INFLUXDB3_WRITE_TIMESTAMP_MAX_PAST | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --write-timestamp-max-future | INFLUXDB3_WRITE_TIMESTAMP_MAX_FUTURE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compaction-row-limit | INFLUXDB3_COMPACTION_ROW_LIMIT INFLUXDB3_ENTERPRISE_COMPACTION_ROW_LIMIT ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compaction-max-num-files-per-plan | INFLUXDB3_COMPACTION_MAX_NUM_FILES_PER_PLAN INFLUXDB3_ENTERPRISE_COMPACTION_MAX_NUM_FILES_PER_PLAN ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compaction-gen2-duration | INFLUXDB3_COMPACTION_GEN2_DURATION INFLUXDB3_ENTERPRISE_COMPACTION_GEN2_DURATION ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compaction-multipliers | INFLUXDB3_COMPACTION_MULTIPLIERS INFLUXDB3_ENTERPRISE_COMPACTION_MULTIPLIERS ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compaction-cleanup-wait | INFLUXDB3_COMPACTION_CLEANUP_WAIT INFLUXDB3_ENTERPRISE_COMPACTION_CLEANUP_WAIT ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compaction-check-interval | INFLUXDB3_COMPACTION_CHECK_INTERVAL INFLUXDB3_ENTERPRISE_COMPACTION_CHECK_INTERVAL ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compacted-data-load-concurrency-limit | INFLUXDB3_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compacted-data-skip-file-index | INFLUXDB3_COMPACTED_DATA_SKIP_FILE_INDEX INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-dispatch-target | INFLUXDB3_COMPACTOR_DISPATCH_TARGET | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-dispatch-ack-timeout | INFLUXDB3_COMPACTOR_DISPATCH_ACK_TIMEOUT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-sweep-interval | INFLUXDB3_COMPACTOR_SWEEP_INTERVAL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-sweep-mode | INFLUXDB3_COMPACTOR_SWEEP_MODE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-sweep-grace | INFLUXDB3_COMPACTOR_SWEEP_GRACE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-sweep-auto-resume | INFLUXDB3_COMPACTOR_SWEEP_AUTO_RESUME | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --compactor-sweep-audit-retention | INFLUXDB3_COMPACTOR_SWEEP_AUDIT_RETENTION | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --gen1-duration | INFLUXDB3_GEN1_DURATION | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --preemptive-cache-age | INFLUXDB3_PREEMPTIVE_CACHE_AGE | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --file-cache-size --parquet-mem-cache-size (pre-3.11 name) | INFLUXDB3_FILE_CACHE_SIZE INFLUXDB3_PARQUET_MEM_CACHE_SIZE ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --parquet-mem-cache-prune-percentage | INFLUXDB3_PARQUET_MEM_CACHE_PRUNE_PERCENTAGE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --parquet-mem-cache-prune-interval | INFLUXDB3_PARQUET_MEM_CACHE_PRUNE_INTERVAL | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --file-cache-recency --parquet-mem-cache-query-path-duration (pre-3.11 name) | INFLUXDB3_FILE_CACHE_RECENCY INFLUXDB3_PARQUET_MEM_CACHE_QUERY_PATH_DURATION ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --disable-file-cache --disable-parquet-mem-cache (pre-3.11 name) --disable-data-file-cache (pre-3.11 name) | INFLUXDB3_DISABLE_FILE_CACHE INFLUXDB3_DISABLE_PARQUET_MEM_CACHE , INFLUXDB3_DISABLE_DATA_FILE_CACHE ( pre-3.11 names ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --table-index-cache-max-entries | INFLUXDB3_TABLE_INDEX_CACHE_MAX_ENTRIES | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --table-index-cache-concurrency-limit | INFLUXDB3_TABLE_INDEX_CACHE_CONCURRENCY_LIMIT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --last-value-cache-disable-from-history | INFLUXDB3_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY INFLUXDB3_ENTERPRISE_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --last-cache-eviction-interval | INFLUXDB3_LAST_CACHE_EVICTION_INTERVAL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --distinct-value-cache-disable-from-history | INFLUXDB3_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY INFLUXDB3_ENTERPRISE_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --distinct-cache-eviction-interval | INFLUXDB3_DISTINCT_CACHE_EVICTION_INTERVAL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --query-file-limit | INFLUXDB3_QUERY_FILE_LIMIT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --plugin-dir | INFLUXDB3_PLUGIN_DIR | | Deployment | Default state | Configuration | | --- | --- | --- | | Deployment | Default state | Configuration | | Docker images | Enabled | INFLUXDB3_PLUGIN_DIR=/plugins | | DEB/RPM packages | Enabled | plugin-dir="/var/lib/influxdb3/plugins" | | Binary/source | Disabled | No plugin-dir configured | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --plugin-repo | INFLUXDB3_PLUGIN_REPO | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --virtual-env-location | VIRTUAL_ENV | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --package-manager | INFLUXDB3_PACKAGE_MANAGER | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --restrict-plugin-triggers-to | INFLUXDB3_RESTRICT_PLUGIN_TRIGGERS_TO | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --trigger-retry-max-attempts | INFLUXDB3_TRIGGER_RETRY_MAX_ATTEMPTS | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --trigger-work-silence-timeout | INFLUXDB3_TRIGGER_WORK_SILENCE_TIMEOUT | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --processing-engine-restart-state-snapshot-interval | INFLUXDB3_PROCESSING_ENGINE_RESTART_STATE_SNAPSHOT_INTERVAL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --plugin-dir-only | INFLUXDB3_PLUGIN_DIR_ONLY | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --import-attempt-server-side-copy | INFLUXDB3_IMPORT_ATTEMPT_SERVER_SIDE_COPY | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --replication-interval | INFLUXDB3_REPLICATION_INTERVAL INFLUXDB3_ENTERPRISE_REPLICATION_INTERVAL ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --catalog-sync-interval | INFLUXDB3_CATALOG_SYNC_INTERVAL INFLUXDB3_ENTERPRISE_CATALOG_SYNC_INTERVAL ( pre-3.11 name ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --wait-for-running-ingester --wait-for-running-ingestor (pre-3.11 name) | INFLUXDB3_WAIT_FOR_RUNNING_INGESTER INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR , INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTER , INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTOR ( pre-3.11 names ) | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --conn-info | INFLUXDB3_CONN_INFO INFLUXDB3_ENTERPRISE_CONN_INFO ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --webui-session-secret | INFLUXDB3_WEBUI_SESSION_SECRET | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --webui-public-uri | INFLUXDB3_WEBUI_PUBLIC_URI | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --webui-cookie-secure | INFLUXDB3_WEBUI_COOKIE_SECURE | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --webui-openai-base-url | INFLUXDB3_WEBUI_OPENAI_BASE_URL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --num-cores | INFLUXDB3_NUM_CORES INFLUXDB3_ENTERPRISE_NUM_CORES ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --max-concurrent-queries | INFLUXDB3_MAX_CONCURRENT_QUERIES | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --datafusion-num-threads | INFLUXDB3_DATAFUSION_NUM_THREADS | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --num-database-limit | INFLUXDB3_NUM_DATABASE_LIMIT INFLUXDB3_ENTERPRISE_NUM_DATABASE_LIMIT ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --num-table-limit | INFLUXDB3_NUM_TABLE_LIMIT INFLUXDB3_ENTERPRISE_NUM_TABLE_LIMIT ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --num-total-columns-per-table-limit | INFLUXDB3_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT INFLUXDB3_ENTERPRISE_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --gen1-lookback-duration | INFLUXDB3_GEN1_LOOKBACK_DURATION | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --retention-check-interval | INFLUXDB3_RETENTION_CHECK_INTERVAL | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --delete-grace-period | INFLUXDB3_DELETE_GRACE_PERIOD | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --hard-delete-default-duration | INFLUXDB3_HARD_DELETE_DEFAULT_DURATION | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --disable-telemetry-upload | INFLUXDB3_DISABLE_TELEMETRY_UPLOAD INFLUXDB3_TELEMETRY_DISABLE_UPLOAD ( pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --telemetry-endpoint | INFLUXDB3_TELEMETRY_ENDPOINT | | influxdb3 serve option | Environment variables | | --- | --- | | influxdb3 serve option | Environment variables | | --tcp-listener-file-path | INFLUXDB3_TCP_LISTENER_FILE_PATH INFLUXDB3_TCP_LISTINER_FILE_PATH ( misspelled pre-3.11 name ) | | influxdb3 serve option | Environment variable | | --- | --- | | influxdb3 serve option | Environment variable | | --admin-token-recovery-tcp-listener-file-path | INFLUXDB3_ADMIN_TOKEN_RECOVERY_TCP_LISTENER_FILE_PATH | | Legacy name | New name | | --- | --- | | Legacy name | New name | | --disable-parquet-mem-cache INFLUXDB3_DISABLE_PARQUET_MEM_CACHE --disable-data-file-cache INFLUXDB3_DISABLE_DATA_FILE_CACHE | --disable-file-cache INFLUXDB3_DISABLE_FILE_CACHE | | --exec-mem-pool-bytes INFLUXDB3_EXEC_MEM_POOL_BYTES | --exec-mem-pool-size INFLUXDB3_EXEC_MEM_POOL_SIZE | | --force-snapshot-mem-threshold INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD | --force-snapshot-mem-size INFLUXDB3_FORCE_SNAPSHOT_MEM_SIZE | | --parquet-mem-cache-query-path-duration INFLUXDB3_PARQUET_MEM_CACHE_QUERY_PATH_DURATION | --file-cache-recency INFLUXDB3_FILE_CACHE_RECENCY | | --parquet-mem-cache-size INFLUXDB3_PARQUET_MEM_CACHE_SIZE | --file-cache-size INFLUXDB3_FILE_CACHE_SIZE | | --query-log-size INFLUXDB3_QUERY_LOG_SIZE | --query-log-max-entries INFLUXDB3_QUERY_LOG_MAX_ENTRIES | | --wal-max-write-buffer-size INFLUXDB3_WAL_MAX_WRITE_BUFFER_SIZE | --wal-max-buffered-writes INFLUXDB3_WAL_MAX_BUFFERED_WRITES | | --wal-snapshot-size INFLUXDB3_WAL_SNAPSHOT_SIZE | --wal-files-per-snapshot INFLUXDB3_WAL_FILES_PER_SNAPSHOT | | --wait-for-running-ingestor INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR | --wait-for-running-ingester INFLUXDB3_WAIT_FOR_RUNNING_INGESTER | | Legacy name | New name | | --- | --- | | Legacy name | New name | | INFLUXDB3_DB_DIR | INFLUXDB3_DATA_DIR | | INFLUXDB3_NODE_IDENTIFIER_FROM_ENV | INFLUXDB3_NODE_ID_FROM_ENV | | INFLUXDB3_NODE_IDENTIFIER_PREFIX | INFLUXDB3_NODE_ID | | INFLUXDB3_NUM_WAL_FILES_TO_KEEP | INFLUXDB3_SNAPSHOTTED_WAL_FILES_TO_KEEP | | INFLUXDB3_START_WITHOUT_AUTH | INFLUXDB3_WITHOUT_AUTH | | INFLUXDB3_TCP_LISTINER_FILE_PATH | INFLUXDB3_TCP_LISTENER_FILE_PATH | | INFLUXDB3_TELEMETRY_DISABLE_UPLOAD | INFLUXDB3_DISABLE_TELEMETRY_UPLOAD | | Legacy name | New name | | --- | --- | | Legacy name | New name | | INFLUXDB3_ENTERPRISE_API_UPLOAD_CHECK_INTERVAL | INFLUXDB3_API_UPLOAD_CHECK_INTERVAL | | INFLUXDB3_ENTERPRISE_API_UPLOAD_CONCURRENT_STATUS_READS | INFLUXDB3_API_UPLOAD_CONCURRENT_STATUS_READS | | INFLUXDB3_ENTERPRISE_CATALOG_SYNC_INTERVAL | INFLUXDB3_CATALOG_SYNC_INTERVAL | | INFLUXDB3_ENTERPRISE_CLUSTER_ID | INFLUXDB3_CLUSTER_ID | | INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT | INFLUXDB3_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT | | INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX | INFLUXDB3_COMPACTED_DATA_SKIP_FILE_INDEX | | INFLUXDB3_ENTERPRISE_COMPACTION_CHECK_INTERVAL | INFLUXDB3_COMPACTION_CHECK_INTERVAL | | INFLUXDB3_ENTERPRISE_COMPACTION_CLEANUP_WAIT | INFLUXDB3_COMPACTION_CLEANUP_WAIT | | INFLUXDB3_ENTERPRISE_COMPACTION_GEN2_DURATION | INFLUXDB3_COMPACTION_GEN2_DURATION | | INFLUXDB3_ENTERPRISE_COMPACTION_MAX_NUM_FILES_PER_PLAN | INFLUXDB3_COMPACTION_MAX_NUM_FILES_PER_PLAN | | INFLUXDB3_ENTERPRISE_COMPACTION_MULTIPLIERS | INFLUXDB3_COMPACTION_MULTIPLIERS | | INFLUXDB3_ENTERPRISE_COMPACTION_ROW_LIMIT | INFLUXDB3_COMPACTION_ROW_LIMIT | | INFLUXDB3_ENTERPRISE_COMPACTION_SNAPSHOTS_LIMIT | INFLUXDB3_COMPACTION_SNAPSHOTS_LIMIT | | INFLUXDB3_ENTERPRISE_CONN_INFO | INFLUXDB3_CONN_INFO | | INFLUXDB3_ENTERPRISE_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY | INFLUXDB3_DISTINCT_VALUE_CACHE_DISABLE_FROM_HISTORY | | INFLUXDB3_ENTERPRISE_INTERNODE_BIND_ADDR | INFLUXDB3_INTERNODE_BIND_ADDR | | INFLUXDB3_ENTERPRISE_INTERNODE_JWT_PRIMARY | INFLUXDB3_INTERNODE_JWT_PRIMARY | | INFLUXDB3_ENTERPRISE_INTERNODE_JWT_SECONDARY | INFLUXDB3_INTERNODE_JWT_SECONDARY | | INFLUXDB3_ENTERPRISE_INTERNODE_TLS_CA | INFLUXDB3_INTERNODE_TLS_CA | | INFLUXDB3_ENTERPRISE_INTERNODE_TLS_CERT | INFLUXDB3_INTERNODE_TLS_CERT | | INFLUXDB3_ENTERPRISE_INTERNODE_TLS_KEY | INFLUXDB3_INTERNODE_TLS_KEY | | INFLUXDB3_ENTERPRISE_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY | INFLUXDB3_LAST_VALUE_CACHE_DISABLE_FROM_HISTORY | | INFLUXDB3_ENTERPRISE_LICENSE_EMAIL | INFLUXDB3_LICENSE_EMAIL | | INFLUXDB3_ENTERPRISE_LICENSE_FILE | INFLUXDB3_LICENSE_FILE | | INFLUXDB3_ENTERPRISE_LICENSE_TYPE | INFLUXDB3_LICENSE_TYPE | | INFLUXDB3_ENTERPRISE_MODE | INFLUXDB3_MODE | | INFLUXDB3_ENTERPRISE_NUM_CORES | INFLUXDB3_NUM_CORES | | INFLUXDB3_ENTERPRISE_NUM_DATABASE_LIMIT | INFLUXDB3_NUM_DATABASE_LIMIT | | INFLUXDB3_ENTERPRISE_NUM_TABLE_LIMIT | INFLUXDB3_NUM_TABLE_LIMIT | | INFLUXDB3_ENTERPRISE_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT | INFLUXDB3_NUM_TOTAL_COLUMNS_PER_TABLE_LIMIT | | INFLUXDB3_ENTERPRISE_REPLICATION_INTERVAL | INFLUXDB3_REPLICATION_INTERVAL | | INFLUXDB3_ENTERPRISE_USE_PACHA_TREE | INFLUXDB3_USE_PACHA_TREE | | INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_COMPACTOR | INFLUXDB3_WAIT_FOR_RUNNING_COMPACTOR | | INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTER | INFLUXDB3_WAIT_FOR_RUNNING_INGESTER | | INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTOR | INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR | --- # Glossary Source: https://docs.influxdata.com/influxdb3/enterprise/reference/glossary/ [A](#a) | [B](#b) | [C](#c) | [D](#d) | [E](#e) | [F](#f) | [G](#g) | [H](#h) | [I](#i) | [J](#j) | [K](#k) | [L](#l) | [M](#m) | [N](#n) | [O](#o) | [P](#p) | [Q](#q) | [R](#r) | [S](#s) | [T](#t) | [U](#u) | [V](#v) | [W](#w) | X | Y | Z A ---------- ### abstract syntax tree (AST) Tree representation of source code that shows the structure, content, and rules of programming statements and discards additional syntax elements. The tree is hierarchical, with elements of program statements broken down into their parts. For more information about AST design, see [Abstract Syntax Tree on Wikipedia](https://en.wikipedia.org/wiki/Abstract_syntax_tree). ### agent A background process started by (or on behalf of) a user that typically requires user input. [Telegraf](/telegraf/v1/) is an agent that requires user input (a configuration file) to gather metrics from declared input plugins and sends metrics to declared output plugins, based on the plugins enabled for a configuration. Related entries:[input plugin](#input-plugin),[output plugin](#output-plugin),[daemon](#daemon) ### aggregator plugin Receives metrics from input plugins, creates aggregate metrics, and then passes aggregate metrics to configured output plugins. Related entries:[input plugin](#input-plugin),[output plugin](#output-plugin),[processor plugin](#processor-plugin) ### aggregate A function that returns an aggregated value across a set of points. For a list of available aggregation functions, see [SQL aggregate functions](/influxdb3/enterprise/reference/sql/functions/aggregate/). Related entries:[function](#function),[selector](#selector) ### API Application programming interface that facilitates and standardizes communication between two or more computer programs. ### argument A value passed to a function or command that determines how the process operates. Related entries:[parameter](#parameter) B ---------- ### batch A collection of points in line protocol format, separated by newlines (`0x0A`). Submitting a batch of points using a single HTTP request to the write endpoints drastically increases performance by reducing the HTTP overhead. InfluxData typically recommends batch sizes of 5,000-10,000 points. In some use cases, performance may improve with significantly smaller or larger batches. Related entries:[line protocol](#line-protocol-lp),[point](#point) ### batch size The number of lines or individual data points in a line protocol batch. The Telegraf agent sends metrics to output plugins in batches rather than individually. Batch size controls the size of each write batch that Telegraf sends to the output plugins. Related entries:[output plugin](#output-plugin) ### bin In a cumulative histogram, a bin includes all data points less than or equal to a specified upper bound. In a normal histogram, a bin includes all data points between the upper and lower bounds. Histogram bins are also sometimes referred to as “buckets.” ### boolean A data type with two possible values: true or false. By convention, you can express `true` as the integer `1` and false as the integer `0` (zero). ### bucket “Bucket” is the term used in InfluxDB 2.x and *InfluxDB Cloud Serverless* to refer to a named location where time series data is stored. Bucket is synonymous with “database” when using InfluxDB 3 Enterprise. Related entries:[database](#database) C ---------- ### CSV Comma-separated values (CSV) delimits text between commas to separate values. A CSV file stores tabular data (numbers and text) in plain text. Each line of the file is a data row. Each row consists of one or more columns, separated by commas. CSV file format is not fully standardized. ### cardinality Cardinality is the number of unique values in a set. Series cardinality is the number of unique [series](#series) in a database as a whole. With the InfluxDB 3 storage engine, high series cardinality *does not* affect performance. ### cluster A collection of servers or processes that work together as a single unit. An InfluxDB cluster is a collection of InfluxDB servers dedicated to the workload of a single customer. ### collect Collect and write time series data to InfluxDB using line protocol and any of the following tools: * Telegraf * the InfluxDB v1, v2, or v3 HTTP APIs * `influxdb3` command line interface (CLI) for InfluxDB 3 Core and Enterprise * InfluxDB 3 Processing engine with the `LineBuilder` and `influxdb3_local` shared API. * v1 or v2 `influx` command line interface (CLI) * InfluxDB v1, v2, or v3 client libraries ### collection interval The default global interval for collecting data from each Telegraf input plugin. The collection interval can be overridden by each individual input plugin’s configuration. Related entries:[input plugin](#input-plugin) ### collection jitter Collection jitter prevents every input plugin from collecting metrics simultaneously, which can have a measurable effect on the system. For each collection interval, every Telegraf input plugin sleeps for a random time between zero and the collection jitter before collecting the metrics. Related entries:[collection interval](#collection-interval),[input plugin](#input-plugin) ### column InfluxDB data is stored in tables within rows and columns. Columns store tag sets and fields sets, and time values. The only required column is *time*, which stores timestamps and is included in all InfluxDB tables. ### common log format (CLF) A standardized text file format used by the InfluxDB server to create log entries when generating server log files. ### compaction Compressing time series data to optimize disk usage. ### continuous query (CQ) Continuous queries are a feature of InfluxDB 1.x used to regularly downsample or process time series data. D ---------- ### daemon A background process that runs without user input. ### dashboard A collection of data visualizations, charts, and panels organized in a single view to monitor and analyze time series data. Dashboards provide at-a-glance visualization of metrics and allow users to track trends, spot anomalies, and understand relationships between different data points over time. Some versions of InfluxDB include built-in dashboarding features. InfluxDB can integrate with third-party visualization and dashboard tools, such as[Grafana](https://grafana.com), [Apache Superset](https://superset.apache.org/),[Tableau](https://www.tableau.com/). ### data model A data model organizes elements of data and standardizes how they relate to one another and to properties of the real world entities. For more information, see the [InfluxDB 3 Enterprise data model](/influxdb3/enterprise/get-started/#data-model) ### data service Stores time series data and handles writes and queries. ### data source A source of data that InfluxDB collects or queries data from. Related entries:[database](#database) ### data type A data type is defined by the values it can take, the programming language used, or the operations that can be performed on it. InfluxDB supports the following data types: * string * boolean * float (64-bit) * integer (64-bit) * unsigned integer (64-bit) * time For more information about different data types, see: * [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/#data-types-and-format) * [InfluxQL](/influxdb3/enterprise/reference/influxql/#literals) * [SQL](/influxdb3/enterprise/reference/sql/data-types/#sql-and-arrow-data-types) #### database A named location where time series data is stored. In InfluxDB 1.x, a database represented a logical container for users, retention policies, continuous queries, and time series data. In InfluxDB 2.x and InfluxDB Cloud Serverless, the equivalent of this concept is an InfluxDB [bucket](#bucket). Related entries:[bucket](#bucket),[retention policy](#retention-policy-rp) ### date-time InfluxDB stores the date-time format for each data point in a timestamp with nanosecond-precision Unix time. Specifying a timestamp is optional. If a timestamp isn’t specified for a data point, InfluxDB uses the server’s local nanosecond timestamp in UTC. ### downsample Aggregating high resolution data into lower resolution data to preserve disk space. ### downstream (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the node an [upstream](#upstream-edr) node sends replicated data to. A node can be downstream-only (a sink), or both upstream and downstream at once (a regional relay). Related entries:[upstream (EDR)](#upstream-edr),[sidecar agent](#sidecar-agent) ### duration A data type that represents a duration of time–for example, `1s`, `1m`, `1h`, `1d`. Retention periods are set using durations. Related entries:[retention period](#retention-period) E ---------- ### event Metrics gathered at irregular time intervals. ### expression A combination of one or more constants, variables, operators, and functions. In the following SQL example, `now() - INTERVAL '7 days'` is an expression that calculates the difference between the `now()` function expression and the duration represented by `INTERVAL '7 days`: ```sql SELECT * FROM home WHERE time >= now() - INTERVAL '7 days' ``` F ---------- ### field A key-value pair in InfluxDB’s data structure that records a data value. Generally, field values change over time. Fields are required in InfluxDB’s data structure. Related entries:[field key](#field-key),[field set](#field-set),[field value](#field-value),[tag](#tag) ### field key The key of the key-value pair. Field keys are strings. Related entries:[field](#field),[field set](#field-set),[field value](#field-value),[tag key](#tag-key) ### field set The collection of field key-value pairs. Related entries:[field](#field),[field key](#field-key),[field value](#field-value),[point](#point) ### field value The value of a key-value pair. Field values are the actual data; they can be strings, floats, integers, unsigned integers or booleans. A field value is always associated with a timestamp. Related entries:[field](#field),[field key](#field-key),[field set](#field-set),[tag set](#tag-set),[tag value](#tag-value),[timestamp](#timestamp) ### file block A file block is a fixed-length chunk of data read into memory when requested by an application. ### float A real number written with a decimal point dividing the integer and fractional parts (`1.0`, `3.14`, `-20.1`). InfluxDB supports 64-bit float values. ### flush interval The global interval for flushing data from each Telegraf output plugin to its destination. This value should not be set lower than the collection interval. Related entries:[collection interval](#collection-interval),[flush jitter](#flush-jitter),[output plugin](#output-plugin) ### flush jitter Flush jitter prevents every Telegraf output plugin from sending writes simultaneously, which can overwhelm some data sinks. Each flush interval, every Telegraf output plugin sleeps for a random time between zero and the flush jitter before emitting metrics. Flush jitter smooths out write spikes when running a large number of Telegraf instances. Related entries:[flush interval](#flush-interval),[output plugin](#output-plugin) ### function A function is an operation that performs a specific task. Functions take input, operate on that input, and then return output. For complete lists of available query language functions, see: * [InfluxQL functions](/influxdb3/enterprise/reference/influxql/functions/) * [SQL functions](/influxdb3/enterprise/reference/sql/functions/). Related entries:[aggregate](#aggregate),[selector](#selector) G ---------- ### gap fill (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the process that recovers a WAL file the replicator expected but that was already deleted (snapshotted and evicted) before it could be sent–typically after a long outage with short WAL retention. The agent detects the missing WAL ID range, records it in a gap ledger, and recovers the data from later-generation files. Related entries:[historic fill (EDR)](#historic-fill-edr),[WAL replication (EDR)](#wal-replication-edr) ### gzip gzip is a type of data compression that compress chunks of data, which is restored by unzipping compressed gzip files. The gzip file extension is `.gz`. H ---------- ### halted state (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the state an agent enters when the downstream persistently refuses data that exists and could be delivered–for example, a schema conflict, an authentication failure, or a malformed batch. Dispatch stops on every tier until the cause is fixed; replication then auto-resumes without a restart. Related entries:[downstream (EDR)](#downstream-edr) ### histogram A visual representation of statistical information that uses rectangles to show the frequency of data items in successive, equal intervals or bins. ### historic fill (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the process that replicates data that existed on the source before EDR was deployed (or before a destination was added), backfilling from snapshot and compacted files below the historic/live boundary. Related entries:[gap fill (EDR)](#gap-fill-edr),[WAL replication (EDR)](#wal-replication-edr) I ---------- ### identifier Identifiers are tokens that refer to specific database objects such as database names, field keys, measurement names, tag keys, etc. Related entries:[database](#database),[field key](#field-key),[measurement](#measurement),[tag key](#tag-key) ### influx `influx` is a command line interface (CLI) that interacts with the InfluxDB v1.x and v2.x server. ### influxctl [`influxctl`](/influxdb3/cloud-dedicated/reference/cli/influxctl/) is a CLI that performs [administrative tasks](/influxdb3/cloud-dedicated/admin/) for an InfluxDB Cloud Dedicated cluster. ### influxd `influxd` is the InfluxDB OSS v1.x and v2.x daemon that runs the InfluxDB server and other required processes. ### InfluxDB An open source time series database (TSDB) developed by InfluxData, optimized for fast, high-availability storage and retrieval of time series data in fields such as operations monitoring, application metrics, Internet of Things sensor data, and real-time analytics. ### influxdb3 `influxdb3` is: * the InfluxDB 3 Core and Enterprise daemon that runs the InfluxDB 3 server * the InfluxDB 3 CLI that interacts with the server for InfluxDB 3 Core and Enterprise ### InfluxQL The SQL-like query language used to query data in InfluxDB. ### input plugin Telegraf input plugins actively gather metrics and deliver them to the core agent, where aggregator, processor, and output plugins can operate on the metrics. To activate an input plugin, enable and configure it in the Telegraf configuration file. Related entries:[aggregator plugin](#aggregator-plugin),[collection interval](#collection-interval),[output plugin](#output-plugin),[processor plugin](#processor-plugin) ### influxdb3 CLI The command line interface for InfluxDB 3. Use the `influxdb3` CLI to administer your instance–for example, to create and manage databases, tables, and tokens–and to write and query data. ### instance An entity comprising data on a server (or virtual server in cloud computing). ### integer A whole number that is positive, negative, or zero (`0`, `-5`, `143`). InfluxDB supports 64-bit integers (minimum: `-9223372036854775808`, maximum: `9223372036854775807`). Related entries:[unsigned integer](#unsigned-integer) ### IOx The IOx storage engine (InfluxDB 3 storage engine) is a real-time, columnar database optimized for time series data built in Rust on top of[Apache Arrow](https://arrow.apache.org/) and[DataFusion](https://arrow.apache.org/datafusion/user-guide/introduction.html). IOx replaces the [TSM (Time Structured Merge tree)](#tsm-time-structured-merge-tree) storage engine. Related entries:[WAL tail](#wal-tail) For InfluxDB 3 Enterprise-specific detail, see[Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). J ---------- ### JWT Typically, JSON web tokens (JWT) are used to authenticate users between an identity provider and a service provider. A server can generate a JWT to assert any business processes. For example, an “admin” token sent to a client can prove the client is logged in as admin. Tokens are signed by one party’s private key (typically, the server). Private keys are used by both parties to verify that a token is legitimate. JWT uses an open standard specified in [RFC 7519](https://tools.ietf.org/html/rfc7519). ### Jaeger Open source tracing used in distributed systems to monitor and troubleshoot transactions. ### JSON JavaScript Object Notation (JSON) is an open-standard file format that uses human-readable text to transmit data objects consisting of attribute–value pairs and array data types. K ---------- ### keyword A keyword is reserved by a program because it has special meaning. Every programming language has a set of keywords (reserved names) that cannot be used as identifiers–for example, you can’t use `SELECT` (an SQL keyword) as a variable name in an SQL query. See keyword lists: * [SQL keywords](/influxdb3/enterprise/reference/sql/#keywords) * [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords) L ---------- ### literal A literal is a value in an expression, a number, character, string, function, record, or array. Literal values are interpreted as defined. ### load balancing Improves workload distribution across multiple computing resources in a network. Load balancing optimizes resource use, maximizes throughput, minimizes response time, and avoids overloading a single resource. Using multiple components with load balancing instead of a single component may increase reliability and availability. If requests to any server in a network increase, requests are forwarded to another server with more capacity. Load balancing can also refer to the communications channels themselves. ### logs Logs record information. Event logs describe system events and activity that help to describe and diagnose problems. Transaction logs describe changes to stored data that help recover data if a database crashes or other errors occur. ### line protocol (LP) The text based format for writing points to InfluxDB. See [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/). M ---------- ### measurement The part of InfluxDB’s data structure that describes the data stored in associated fields. Measurements are strings. Related entries:[field](#field), [series](#series) ### metric Data tracked over time. ### metric buffer The metric buffer caches individual metrics when writes are failing for an Telegraf output plugin. Telegraf will attempt to flush the buffer upon a successful write to the output. The oldest metrics are dropped first when this buffer fills. Related entries:[output plugin](#output-plugin) ### missing values Denoted by a null value. Identifies missing information, which may be useful to include in an error message. N ---------- ### node An independent process or server in a cluster. Related entries:[cluster](#cluster),[server](#server) ### now The local server’s nanosecond timestamp. ### null A data type that represents a missing or unknown value. Denoted by the `null` value. Values of [tags](#tag) and [fields](#field) may be `null`, but timestamp values are never `null`. O ---------- ### operator A symbol that usually represents an action or process. For example: `+`, `-`, `>`. Related entries:[operand](#operand) ### operand The object or value on either side of an [operator](#operator). Related entries:[operator](#operator) ### organization An InfluxDB v2 concept that describes workspace for a group of users. All InfluxDB v2 dashboards, tasks, buckets, members, and so on, belong to an organization. Organizations are not part of InfluxDB 3 Enterprise. ### owner A type of role for a user. Owners have read/write permissions. Users can have owner roles for databases and other resources. Role permissions are separate from API token permissions. For additional information on API tokens, see [token](#token). ### output plugin Telegraf output plugins deliver metrics to their configured destination. To activate an output plugin, enable and configure the plugin in Telegraf’s configuration file. Related entries:[aggregator plugin](#aggregator-plugin),[flush interval](#flush-interval),[input plugin](#input-plugin),[processor plugin](#processor-plugin) P ---------- ### parameter A key-value pair used to pass information to a function that determines how the function operates. Related entries:[argument](#argument) ### pipe Method for passing information from one process to another. For example, an output parameter from one process is input to another process. Information passed through a pipe is retained until the receiving process reads the information. ### plugin A Python file with a specific function signature that corresponds to a[trigger](#trigger) type. Plugins run in the InfluxDB 3 [Processing engine](#processing-engine) to process data, respond to database events, and connect to external systems. Related entries:[Processing engine](#processing-engine),[trigger](#trigger) ### point Single data record identified by its *measurement*, *tag keys*, *tag values*,*field key*, and *timestamp*. In a [series](#series), each point has a unique timestamp. If you write a point to a series with a timestamp that matches an existing point, the field set becomes a union of the old and new field set, where any ties go to the new field set. Related entries:[measurement](#measurement),[tag set](#tag-set),[field set](#field-set),[timestamp](#timestamp) ### primary key With the InfluxDB 3 storage engine, the primary key is the list of columns used to uniquely identify each row in a table. Rows are uniquely identified by their timestamp and tag set. A row’s primary key tag set does not include tags with null values. ### precision The precision configuration setting determines the timestamp precision retained for input data points. All incoming timestamps are truncated to the specified precision. Valid precisions are `ns`, `us` or `µs`, `ms`, and `s`. In Telegraf, truncated timestamps are padded with zeros to create a nanosecond timestamp. Telegraf output plugins emit timestamps in nanoseconds. For example, if the precision is set to `ms`, the nanosecond epoch timestamp `1480000000123456789` is truncated to `1480000000123` in millisecond precision and padded with zeroes to make a new, less precise nanosecond timestamp of `1480000000123000000`. Telegraf output plugins do not alter the timestamp further. The precision setting is ignored for service input plugins. Related entries:[aggregator plugin](#aggregator-plugin),[input plugin](#input-plugin),[output plugin](#output-plugin),[processor plugin](#processor-plugin),[service input plugin](#service-input-plugin) ### predicate expression A predicate expression compares two values and returns `true` or `false` based on the relationship between the two values. A predicate expression is comprised of a left operand, a comparison operator, and a right operand. ### process A set of predetermined rules. A process can refer to instructions being executed by the computer processor or refer to the act of manipulating data. ### Processing engine The Processing engine is a Python virtual machine (VM) embedded within InfluxDB 3 Core and Enterprise for automatically processing data and responding to database events. It executes Python plugins in response to events defined by triggers. The Processing engine runs Python code directly in your database, allowing plugins to react to specific triggers without requiring external services. Related entries:[plugin](#plugin),[trigger](#trigger) ### processor plugin Telegraf processor plugins transform, decorate, and filter metrics collected by input plugins, passing the transformed metrics to the output plugins. Related entries:[aggregator plugin](#aggregator-plugin),[input plugin](#input-plugin),[output plugin](#output-plugin) ### Prometheus format A simple text-based format for exposing metrics and ingesting them into Prometheus. Q ---------- ### query A request for information. An InfluxDB query returns time series data. See [Query data in InfluxDB](/influxdb3/enterprise/query-data/). ### query group An operator-defined, ordered list of query nodes in an InfluxDB 3 Enterprise cluster. Each query group has a name and a replication factor. Query groups are intended to define how query nodes participate in distributed query processing, but aren’t yet operational as of v3.11–the CLI commands store query group definitions in the catalog, but the server doesn’t use them to affect query routing, data placement, or replication. The order of query nodes in a group is significant. When you create or update a group, provide node IDs in the intended order. ### query plan A sequence of steps (*nodes*) that the InfluxDB Querier devises and executes to calculate the result of the query in the least amount of time. A *logical plan* is a high level representation of a query and doesn’t consider cluster configuration or data organization. A *physical plan* represents the query execution plan and data flow through plan nodes that read (*scan*), deduplicate, merge, filter, and sort data. A physical plan is optimized for the cluster configuration and data organization. R ---------- ### REPL A Read-Eval-Print Loop (REPL) is an interactive programming environment where you type a command and immediately see the result. ### regular expressions Regular expressions (regex or regexp) are patterns used to match character combinations in strings. ### rejected points In a batch of data, points that InfluxDB couldn’t write to a database. Field type conflicts are a common cause of rejected points. ### retention period The [duration](#duration) of time that a database retains data. InfluxDB drops points with timestamps older than their database’s retention period relative to [now](#now). The minimum retention period is **one hour**. In InfluxDB Cloud Serverless, *bucket* is synonymous with database. Related entries:[bucket](#bucket)[database](#database) ### retention policy (RP) A retention policy is part of the InfluxDB 1.x data model that describes how long InfluxDB keeps data (duration), how many copies of the data to store when in a in the cluster (replication factor), and the time range covered by shard groups (shard group duration). RPs are unique per database and along with the measurement and tag set define a series. In InfluxDB 3 Enterprise, the equivalent is [retention period](#retention-period), however retention periods are not part of the data model. The retention period describes the data persistence behavior of a database. In InfluxDB Cloud Serverless, *bucket* is synonymous with database. Related entries:[retention period](#retention-period), ### RFC3339 timestamp A timestamp that uses the human-readable DateTime format proposed in[RFC 3339](https://tools.ietf.org/html/rfc3339) (for example: `2020-01-01T00:00:00.00Z`). Related entries:[RFC3339Nano timestamp](#rfc3339nano-timestamp),[timestamp](#timestamp),[unix timestamp](#unix-timestamp) ### RFC3339Nano timestamp A [Golang representation of the RFC 3339 DateTime format](https://go.dev/src/time/format.go)that uses nanosecond resolution–for example:`2006-01-02T15:04:05.999999999Z07:00`. InfluxDB clients can return RFC3339Nano timestamps in log events and CSV-formatted query results. Related entries:[RFC3339 timestamp](#rfc3339-timestamp),[timestamp](#timestamp),[unix timestamp](#unix-timestamp) ### row A row in a [table](#table) represents a specific record or instance of data.[Column](#column) values in a row represent specific attributes or properties of the instance. Each row has a [primary key](#primary-key) that makes the row unique from other rows in the table. Related entries:[column](#column),[primary key](#primary-key),[series](#series),[table](#table) S ---------- ### schema How data is organized in InfluxDB. The fundamentals of the InfluxDB schema are databases, measurements, tag keys, tag values, and field keys. Related entries:[bucket](#bucket),[database](#database),[field key](#field-key),[measurement](#measurement),[series](#series),[tag key](#tag-key),[tag value](#tag-value) ### secret Secrets are key-value pairs that contain information you want to control access to, such as API keys, passwords, or certificates. ### selector A function that returns a single point from the range of specified points. See [SQL selector functions](/influxdb3/enterprise/reference/sql/functions/selector/)for a complete list of available SQL selector functions. Related entries:[aggregate](#aggregate),[function](#function),[transformation](#transformation) ### series In the InfluxDB 3 data structure, a collection of data that share a common*measurement* and *tag set*. Related entries:[field set](#field-set),[measurement](#measurement),[tag set](#tag-set) ### series cardinality The number of unique measurement, tag set, and field key combinations in an InfluxDB 3 Enterprise database. For example, assume that an InfluxDB database has one measurement. The single measurement has two tag keys: `email` and `status`. If there are three different `email` tag values, and each email address is associated with two different `status` tag values, then the series cardinality for the measurement is 6 (3 × 2 = 6): | email |status| |--------------------|------| |lorr@influxdata.com |start | |lorr@influxdata.com |finish| |marv@influxdata.com |start | |marv@influxdata.com |finish| |cliff@influxdata.com|start | |cliff@influxdata.com|finish| In some cases, performing this multiplication may overestimate series cardinality because of the presence of dependent tags. Dependent tags are scoped by another tag and do not increase series cardinality. If we add the tag `firstname` to the preceding example, the series cardinality would not be 18 (3 × 2 × 3 = 18). The series cardinality would remain unchanged at 6, as `firstname` is already scoped by the `email` tag: | email |status|firstname| |--------------------|------|---------| |lorr@influxdata.com |start |lorraine | |lorr@influxdata.com |finish|lorraine | |marv@influxdata.com |start | marvin | |marv@influxdata.com |finish| marvin | |cliff@influxdata.com|start |clifford | |cliff@influxdata.com|finish|clifford | Related entries:[field key](#field-key),[measurement](#measurement),[tag key](#tag-key),[tag set](#tag-set) ### series key A series key identifies a particular series by measurement, tag set, and field key. For example: ```text # measurement, tag set, field key h2o_level, location=santa_monica, h2o_feet ``` Related entries:[series](#series) ### server A computer, virtual or physical, running InfluxDB. ### service input plugin Telegraf input plugins that run in a passive collection mode while the Telegraf agent is running. Service input plugins listen on a socket for known protocol inputs, or apply their own logic to ingested metrics before delivering metrics to the Telegraf agent. Related entries:[aggregator plugin](#aggregator-plugin),[input plugin](#input-plugin),[output plugin](#output-plugin),[processor plugin](#processor-plugin) ### sidecar agent A process that runs alongside a primary application to extend it without modifying it. [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/) runs as a sidecar agent (`influxdb3-edr`) beside an unmodified InfluxDB 3 Enterprise instance, reading its object store directly. Related entries:[downstream (EDR)](#downstream-edr),[upstream (EDR)](#upstream-edr) ### string A data type used to represent text. T ---------- ### TCP Transmission Control Protocol. ### table A collection of related data organized in a structured way with a predefined set of columns and data types. Each row in the table represents a specific record or instance of the data, and each column represents a specific attribute or property of the data. In InfluxDB 3 Enterprise, a table represents a measurement. Related entries:[column](#column),[measurement](#measurement),[primary key](#primary-key),[row](#row) ### tag The key-value pair in InfluxDB’s data structure that records metadata. Tags are an optional part of InfluxDB’s data structure but they are useful for storing commonly queried metadata. Related entries:[field](#field),[tag key](#tag-key),[tag set](#tag-set),[tag value](#tag-value) ### tag key The key of a tag key-value pair. Tag keys are strings and store metadata. Related entries:[field key](#field-key),[tag](#tag),[tag set](#tag-set),[tag value](#tag-value) ### tag set The collection of tag keys and tag values on a point. Related entries:[point](#point),[primary key](#primary-key),[series](#series),[tag](#tag),[tag key](#tag-key),[tag value](#tag-value) ### tag value The value of a tag key-value pair. Tag values are strings and they store metadata. Related entries:[tag](#tag),[tag key](#tag-key),[tag set](#tag-set) ### Telegraf A plugin-driven agent that collects, processes, aggregates, and writes metrics. Related entries:[Telegraf plugins](/telegraf/v1/plugins/),[Use Telegraf to collect data](/influxdb3/enterprise/write-data/use-telegraf/), ### time (data type) A data type that represents a single point in time with nanosecond precision. ### time series data Sequence of data points typically consisting of successive measurements made from the same source over a time interval. Time series data shows how data evolves over time. On a time series data graph, one of the axes is always time. Time series data may be regular or irregular. Regular time series data changes in constant intervals. Irregular time series data changes at non-constant intervals. ### timestamp The date and time associated with a point. Time in InfluxDB is in UTC. To specify time when writing data, see[Elements of line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/#elements-of-line-protocol). Related entries:[point](#point),[unix timestamp](#unix-timestamp),[RFC3339 timestamp](#rfc3339-timestamp) ### token Tokens provide authorization to perform specific actions in InfluxDB. InfluxDB 3 Enterprise supports different types of authorization tokens: * **Admin token:** A token that grants full access to InfluxDB 3 server actions. * **Resource token:** Tokens that grant read and write access to server resources, such as databases and system information. Database tokens allow for reading and writing data in your InfluxDB 3 Enterprise instance. System tokens allow for reading system information and metrics for your server. For more information, see [Manage tokens](/influxdb3/enterprise/admin/tokens/). ### transformation Data transformation refers to the process of converting or modifying input data from one format, value, or structure to another. InfluxQL [transformation functions](/influxdb3/enterprise/reference/influxql/functions/transformations/)modify and return values in each row of queried data, but do not return an aggregated value across those rows. Related entries: [aggregate](#aggregate), [function](#function), [selector](#selector) ### trigger With InfluxDB 3 Core and Enterprise, a Processing engine trigger connects a Processing engine plugin to a specific type of event. A trigger has: * a *trigger specification* that defines the event (data ingestion, a time interval or schedule, or an HTTP request) that runs the plugin. * a plugin file that contains the code to run when the trigger fires. * optional configuration parameters that are passed to the plugin. Related entries:[plugin](#plugin),[Processing engine](#processing-engine) ### TSM (Time Structured Merge tree) The InfluxDB v1 and v2 data storage format that allows greater compaction and higher write and read throughput than B+ or LSM tree implementations. The TSM storage engine has been replaced by the [InfluxDB 3 storage engine (IOx)](#iox). Related entries:[IOx](#iox) U ---------- ### UDP User Datagram Protocol is a packet of information. When a request is made, a UDP packet is sent to the recipient. The sender doesn’t verify the packet is received. The sender continues to send the next packets. This means computers can communicate more quickly. This protocol is used when speed is desirable and error correction is not necessary. ### unix epoch The date and time from which Unix system times are measured. The Unix epoch is `1970-01-01T00:00:00Z`. ### unix timestamp Counts time since **Unix Epoch (1970-01-01T00:00:00Z UTC)** in specified units ([precision](#precision)). Specify timestamp precision when [writing data to InfluxDB](/influxdb3/enterprise/write-data/). InfluxDB supports the following unix timestamp precisions: |Precision|Description | Example | |---------|------------|---------------------| | `ns` |Nanoseconds |`1577836800000000000`| | `us` |Microseconds| `1577836800000000` | | `ms` |Milliseconds| `1577836800000` | | `s` | Seconds | `1577836800` | *The examples above represent **2020-01-01T00:00:00Z UTC**.* Related entries:[timestamp](#timestamp),[RFC3339 timestamp](#rfc3339-timestamp) ### unsigned integer A whole number that is positive or zero (`0`, `143`). Also known as a “uinteger.” InfluxDB supports 64-bit unsigned integers (minimum: `0`, maximum: `18446744073709551615`). Related entries:[integer](#integer) ### upstream (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the source node that sends replicated data to a[downstream](#downstream-edr) node. The upstream initiates all data transfer. A node can be upstream-only (a pure source), or both upstream and downstream at once (a regional relay). Related entries:[downstream (EDR)](#downstream-edr),[sidecar agent](#sidecar-agent) ### user InfluxDB users are granted permission to access InfluxDB. V ---------- ### values per second The preferred measurement of the rate at which data is persisted to InfluxDB. Write speeds are generally quoted in values per second. To calculate the values per second rate, multiply the number of points written per second by the number of values stored per point. For example, if the points have four fields each, and a batch of 5000 points is written 10 times per second, the values per second rate is: **4 field values per point** × **5000 points per batch** × **10 batches per second** = **200,000 values per second** Related entries:[batch](#batch),[field](#field),[point](#point) ### variable A storage location (identified by a memory address) paired with an associated symbolic name (an identifier). A variable contains some known or unknown quantity of information referred to as a value. ### variable assignment A statement that sets or updates the value stored in a variable. W ---------- ### WAL (Write-Ahead Log) The temporary cache for recently written points. To reduce the frequency that permanent storage files are accessed, InfluxDB caches new points in the WAL until their total size or age triggers a flush to more permanent storage. This allows for efficient batching of the writes into the storage engine. Points in the WAL are queryable and persist through a system reboot. On process start, all points in the WAL must be flushed before the system accepts new writes. ### WAL replication (EDR) In [Edge Data Replication (EDR) for InfluxDB 3 Enterprise](/influxdb3/enterprise/edr/), the live replication path that discovers new [WAL](#wal-write-ahead-log)files as they appear on the source’s object store and ships them downstream. WAL replication is EDR’s primary, most precise data source;[historic fill](#historic-fill-edr) and [gap fill](#gap-fill-edr) recover data that outlives WAL retention from compacted files instead. Related entries:[historic fill (EDR)](#historic-fill-edr),[gap fill (EDR)](#gap-fill-edr) ### WAL tail The most recent points in the [WAL](#wal-write-ahead-log) that have not yet been durably persisted beyond the WAL. For details, see[Data durability](/influxdb3/enterprise/reference/internals/durability/#wal-tail). ### windowing Grouping data based on specified time intervals. This is also referred to as “time binning” or “date binning.” [glossary](/influxdb3/enterprise/tags/glossary/) | email | status | | --- | --- | | email | status | | lorr@influxdata.com | start | | lorr@influxdata.com | finish | | marv@influxdata.com | start | | marv@influxdata.com | finish | | cliff@influxdata.com | start | | cliff@influxdata.com | finish | | email | status | firstname | | --- | --- | --- | | email | status | firstname | | lorr@influxdata.com | start | lorraine | | lorr@influxdata.com | finish | lorraine | | marv@influxdata.com | start | marvin | | marv@influxdata.com | finish | marvin | | cliff@influxdata.com | start | clifford | | cliff@influxdata.com | finish | clifford | | Precision | Description | Example | | --- | --- | --- | | Precision | Description | Example | | ns | Nanoseconds | 1577836800000000000 | | us | Microseconds | 1577836800000000 | | ms | Milliseconds | 1577836800000 | | s | Seconds | 1577836800 | --- # InfluxQL reference documentation Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/ InfluxQL (Influx Query Language) is an SQL-like query language used to interact with InfluxDB and work with times series data. > [!Important] > #### InfluxQL feature support > > InfluxQL is being rearchitected to work with the InfluxDB 3 storage engine. > This process is ongoing and some InfluxQL features are still being implemented. > For information about the current implementation status of InfluxQL features, > see [InfluxQL feature support](/influxdb3/enterprise/reference/influxql/feature-support/). ## Notation The syntax is specified using Extended Backus-Naur Form (“EBNF”). EBNF is the same notation used in the [Go programming language specification](https://golang.org/ref/spec). ```go Production = production_name "=" [ Expression ] "." . Expression = Alternative { "|" Alternative } . Alternative = Term { Term } . Term = production_name | token [ "…" token ] | Group | Option | Repetition . Group = "(" Expression ")" . Option = "[" Expression "]" . Repetition = "{" Expression "}" . ``` Notation operators in order of increasing precedence: ``` | alternation () grouping [] option (0 or 1 times) {} repetition (0 to n times) ``` ## Query representation ### Characters InfluxQL is Unicode text encoded in [UTF-8](http://en.wikipedia.org/wiki/UTF-8). ``` newline = /* the Unicode code point U+000A */ . unicode_char = /* an arbitrary Unicode code point except newline */ . ``` ### Letters and digits Letters are the set of ASCII letters and the underscore (`_`, U+005F). Only decimal digits are supported. ``` letter = ascii_letter | "_" . ascii_letter = "A" … "Z" | "a" … "z" . digit = "0" … "9" . ``` ### Identifiers Identifiers are tokens that refer to[database](/influxdb3/enterprise/reference/glossary/#database) names,[retention policy](/influxdb3/enterprise/reference/glossary/#retention-policy-rp) names,[measurement](/influxdb3/enterprise/reference/glossary/#measurement) names,[tag keys](/influxdb3/enterprise/reference/glossary/#tag-key), and[field keys](/influxdb3/enterprise/reference/glossary/#field-key). The rules are as follows: * Double-quoted identifiers can contain any Unicode character except for a new line. * Double-quoted identifiers can also contain escaped `"` characters (that is, `\"`) * Double-quoted identifiers can include InfluxQL [keywords](#keywords). * Unquoted identifiers must start with an upper or lowercase ASCII character or “\_”. * Unquoted identifiers may contain only ASCII letters, decimal digits, and “\_”. ``` identifier = unquoted_identifier | quoted_identifier . unquoted_identifier = ( letter ) { letter | digit } . quoted_identifier = `"` unicode_char { unicode_char } `"` . ``` #### Examples ``` cpu _cpu_stats "1h" "anything really" "1_Crazy-1337.identifier>NAME👍" ``` ### Keywords ``` ALL ALTER ANY AS ASC BEGIN BY CREATE CONTINUOUS DATABASE DATABASES DEFAULT DELETE DESC DESTINATIONS DIAGNOSTICS DISTINCT DROP DURATION END EVERY EXPLAIN FIELD FOR FROM GRANT GRANTS GROUP GROUPS IN INF INSERT INTO KEY KEYS KILL LIMIT SHOW MEASUREMENT MEASUREMENTS NAME OFFSET ON ORDER PASSWORD POLICY POLICIES PRIVILEGES QUERIES QUERY READ REPLICATION RESAMPLE RETENTION REVOKE SELECT SERIES SET SHARD SHARDS SLIMIT SOFFSET STATS SUBSCRIPTION SUBSCRIPTIONS TAG TO USER USERS VALUES WHERE WITH WRITE ``` If you use an InfluxQL keyword as an[identifier](/influxdb3/enterprise/reference/glossary/#identifier), double-quote the identifier in every query. The `time` keyword is a special case.`time` can be a database name,[measurement](/influxdb3/enterprise/reference/glossary/#measurement) name,[retention policy](/influxdb3/enterprise/reference/glossary/#retention-policy-rp) name, and[user](/influxdb3/enterprise/reference/glossary/#user) name. In those cases, you don’t need to double-quote `time` in queries. `time` can’t be a [field key](/influxdb3/enterprise/reference/glossary/#field-key) or[tag key](/influxdb3/enterprise/reference/glossary/#tag-key); InfluxDB rejects writes with `time` as a field key or tag key and returns an error. ### Literals #### Integers InfluxQL supports decimal integer literals. Hexadecimal and octal literals aren’t currently supported. ``` int_lit = ( "1" … "9" ) { digit } . ``` #### Floats InfluxQL supports floating-point literals. Exponents aren’t currently supported. ``` float_lit = int_lit "." int_lit . ``` #### Strings String literals must be surrounded by single quotes. Strings may contain `'` characters as long as they are escaped (that is, , `\'`) ``` string_lit = `'` { unicode_char } `'` . ``` #### Durations Duration literals specify a length of time. An integer literal followed immediately (with no spaces) by one of the duration units listed below is interpreted as a duration literal. Durations can be specified with mixed units. ##### Duration units |Units | Meaning | |------|---------------------------------------| | ns | nanoseconds (1 billionth of a second) | |u or µ|microseconds (1 millionth of a second) | | ms |milliseconds (1 thousandth of a second)| | s | second | | m | minute | | h | hour | | d | day | | w | week | ``` duration_lit = int_lit duration_unit . duration_unit = "ns" | "u" | "µ" | "ms" | "s" | "m" | "h" | "d" | "w" . ``` #### Dates & Times Unlike other notations used in InfluxQL, the date and time literal format isn’t specified by EBNF. InfluxQL date and time is specified using Go’s time parsing format and[reference date](https://pkg.go.dev/time#pkg-constants) written in the format required by InfluxQL. The reference date time is: InfluxQL reference date time: January 2nd, 2006 at 3:04:05 PM ``` time_lit = "2006-01-02 15:04:05.999999" | "2006-01-02" . ``` #### Booleans ``` bool_lit = TRUE | FALSE . ``` #### Regular Expressions ``` regex_lit = "/" { unicode_char } "/" . ``` **Comparators:**`=~` matches against`!~` doesn’t match against > [!Note] > InfluxQL supports using regular expressions when specifying: > > * [field keys](/influxdb3/enterprise/reference/glossary/#field-key) and [tag keys](/influxdb3/enterprise/reference/glossary/\> #tag-key) in the [`SELECT` clause](/influxdb3/enterprise/reference/influxql/select/) > * [measurements](/influxdb3/enterprise/reference/glossary/#measurement) in the [`FROM` clause](/influxdb3/enterprise/reference/\> influxql/select/#from-clause) > * [tag values](/influxdb3/enterprise/reference/glossary/#tag-value) and > string [field values](/influxdb3/enterprise/reference/glossary/#field-value)in the [`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/) > * [tag keys](/influxdb3/enterprise/reference/glossary/#tag-key) in the[`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/) > > Currently, InfluxQL doesn’t support using regular expressions to match non-string > field values in the `WHERE` clause, [databases](/influxdb3/enterprise/reference/glossary/#database), > and [retention polices](/influxdb3/enterprise/reference/glossary/#retention-policy-rp). ## Queries A query is composed of one or more statements separated by a semicolon (`;`). ``` query = statement { ";" statement } . statement = explain_stmt | explain_analyze_stmt | select_stmt | show_field_keys_stmt | show_measurements_stmt | show_retention_policies_stmt | show_tag_keys_stmt | show_tag_values_with_key = stmt . ``` ## Statements ### EXPLAIN Parses and plans the query, and then prints a summary of estimated costs. Many SQL engines use the `EXPLAIN` statement to show join order, join algorithms, and predicate and expression pushdown. Since InfluxQL doesn’t support joins, the cost of an InfluxQL query is typically a function of the total series accessed, the number of iterator accesses to a TSM file, and the number of TSM blocks that need to be scanned. A query plan generated by `EXPLAIN` contains the following elements: * expression * auxiliary fields * number of shards * number of series * cached values * number of files * number of blocks * size of blocks ``` explain_stmt = "EXPLAIN" select_stmt . ``` #### Example ```sql > explain select sum(pointReq) from "_internal"."monitor"."write" group by hostname; > QUERY PLAN ------ EXPRESSION: sum(pointReq::integer) NUMBER OF SHARDS: 2 NUMBER OF SERIES: 2 CACHED VALUES: 110 NUMBER OF FILES: 1 NUMBER OF BLOCKS: 1 SIZE OF BLOCKS: 931 ``` ### EXPLAIN ANALYZE Executes the specified `SELECT` statement and returns data about the query performance and storage during runtime, visualized as a tree. Use this statement to analyze query performance and storage, including [execution time](#execution_time) and [planning time](#planning_time), and the [iterator type](#iterator-type) and [cursor type](#cursor-type). For example, if you execute the following statement: ```sql explain analyze select mean(usage_steal) from cpu where time >= '2018-02-22T00:00:00Z' and time < '2018-02-22T12:00:00Z' ``` The output is similar to the following: ``` ## EXPLAIN ANALYZE . └── select ├── execution_time: 2.25823ms ├── planning_time: 18.381616ms ├── total_time: 20.639846ms └── field_iterators ├── labels │ └── statement: SELECT mean(usage_steal::float) FROM telegraf."default".cpu └── expression ├── labels │ └── expr: mean(usage_steal::float) └── create_iterator ├── labels │ ├── measurement: cpu │ └── shard_id: 608 ├── cursors_ref: 779 ├── cursors_aux: 0 ├── cursors_cond: 0 ├── float_blocks_decoded: 431 ├── float_blocks_size_bytes: 1003552 ├── integer_blocks_decoded: 0 ├── integer_blocks_size_bytes: 0 ├── unsigned_blocks_decoded: 0 ├── unsigned_blocks_size_bytes: 0 ├── string_blocks_decoded: 0 ├── string_blocks_size_bytes: 0 ├── boolean_blocks_decoded: 0 ├── boolean_blocks_size_bytes: 0 └── planning_time: 14.805277ms``` ``` > [!Note] > `EXPLAIN ANALYZE` ignores query output, so the cost of serialization to JSON or > CSV isn’t accounted for. ##### execution\_time Shows the amount of time the query took to execute, including reading the time series data, performing operations as data flows through iterators, and draining processed data from iterators. Execution time doesn’t include the time taken to serialize the output into JSON or other formats. ##### planning\_time Shows the amount of time the query took to plan. Planning a query in InfluxDB requires a number of steps. Depending on the complexity of the query, planning can require more work and consume more CPU and memory resources than executing the query. For example, the number of series keys required to execute a query affects how quickly the query is planned and how much memory the planning requires. First, InfluxDB determines the effective time range of the query and selects the shards to access. Next, for each shard and each measurement, InfluxDB performs the following steps: 1. Select matching series keys from the index, filtered by tag predicates in the `WHERE` clause. 2. Group filtered series keys into tag sets based on the `GROUP BY` dimensions. 3. Enumerate each tag set and create a cursor and iterator for each series key. 4. Merge iterators and return the merged result to the query executor. ##### iterator type `EXPLAIN ANALYZE` supports the following iterator types: * `create_iterator` node represents work done by the local influxd instance──a complex composition of nested iterators combined and merged to produce the final query output. * (InfluxDB Enterprise only) `remote_iterator` node represents work done on remote machines. For more information about iterators, see [Understanding iterators](/influxdb3/enterprise/reference/influxql/internals/#understanding-iterators). ##### cursor type `EXPLAIN ANALYZE` distinguishes 3 cursor types. While the cursor types have the same data structures and equal CPU and I/O costs, each cursor type is constructed for a different reason and separated in the final output. Consider the following cursor types when tuning a statement: * cursor\_ref: Reference cursor created for `SELECT` projections that include a function, such as `last()` or `mean()`. * cursor\_aux: Auxiliary cursor created for simple expression projections (not selectors or an aggregation). For example, `SELECT foo FROM m` or `SELECT foo+bar FROM m`, where `foo` and `bar` are fields. * cursor\_cond: Condition cursor created for fields referenced in a `WHERE` clause. For more information about cursors, see [Understanding cursors](/influxdb3/enterprise/reference/influxql/internals/#cursors). ##### block types `EXPLAIN ANALYZE` separates storage block types, and reports the total number of blocks decoded and their size (in bytes) on disk. The following block types are supported: | Type | Description | |----------|-------------------------------------| | `float` |64-bit IEEE-754 floating-point number| |`integer` | 64-bit signed integer | |`unsigned`| 64-bit unsigned integer | |`boolean` | 1-bit, LSB encoded | | `string` | UTF-8 string | ### SELECT ``` select_stmt = "SELECT" fields from_clause [ where_clause ] [ group_by_clause ] [ order_by_clause ] [ limit_clause ] [ offset_clause ] [ slimit_clause ] [ soffset_clause ] [ timezone_clause ] . ``` #### Example Select from measurements grouped by the day with a timezone ```sql SELECT mean("value") FROM "cpu" GROUP BY region, time(1d) fill(0) tz('America/Chicago') ``` ### SHOW FIELD KEYS ``` show_field_keys_stmt = "SHOW FIELD KEYS" [on_clause] [ from_clause ] . ``` #### Examples ```sql -- show field keys and field value data types from all measurements SHOW FIELD KEYS -- show field keys and field value data types from specified measurement SHOW FIELD KEYS FROM "cpu" ``` ### SHOW MEASUREMENTS ``` show_measurements_stmt = "SHOW MEASUREMENTS" [on_clause] [ with_measurement_clause ] [ where_clause ] [ limit_clause ] [ offset_clause ] . ``` #### Examples ```sql -- show all measurements SHOW MEASUREMENTS -- show measurements where region tag = 'uswest' AND host tag = 'serverA' SHOW MEASUREMENTS WHERE "region" = 'uswest' AND "host" = 'serverA' -- show measurements that start with 'h2o' SHOW MEASUREMENTS WITH MEASUREMENT =~ /h2o.*/ ``` ### SHOW TAG KEYS ``` show_tag_keys_stmt = "SHOW TAG KEYS" [on_clause] [ from_clause ] [ where_clause ] [ limit_clause ] [ offset_clause ] . ``` #### Examples ```sql -- show all tag keys SHOW TAG KEYS -- show all tag keys from the cpu measurement SHOW TAG KEYS FROM "cpu" -- show all tag keys from the cpu measurement where the region key = 'uswest' SHOW TAG KEYS FROM "cpu" WHERE "region" = 'uswest' -- show all tag keys where the host key = 'serverA' SHOW TAG KEYS WHERE "host" = 'serverA' ``` ### SHOW TAG VALUES ``` show_tag_values_stmt = "SHOW TAG VALUES" [on_clause] [ from_clause ] with_tag_clause [ where_clause ] [ limit_clause ] [ offset_clause ] . ``` #### Examples ```sql -- show all tag values across all measurements for the region tag SHOW TAG VALUES WITH KEY = "region" -- show tag values from the cpu measurement for the region tag SHOW TAG VALUES FROM "cpu" WITH KEY = "region" -- show tag values across all measurements for all tag keys that do not include the letter c SHOW TAG VALUES WITH KEY !~ /.*c.*/ -- show tag values from the cpu measurement for region & host tag keys where service = 'redis' SHOW TAG VALUES FROM "cpu" WITH KEY IN ("region", "host") WHERE "service" = 'redis' ``` ## Clauses ``` from_clause = "FROM" measurements . group_by_clause = "GROUP BY" dimensions fill(fill_option). limit_clause = "LIMIT" int_lit . offset_clause = "OFFSET" int_lit . slimit_clause = "SLIMIT" int_lit . soffset_clause = "SOFFSET" int_lit . timezone_clause = tz(string_lit) . on_clause = "ON" db_name . order_by_clause = "ORDER BY" sort_fields . where_clause = "WHERE" expr . with_measurement_clause = "WITH MEASUREMENT" ( "=" measurement | "=~" regex_lit ) . with_tag_clause = "WITH KEY" ( "=" tag_key | "!=" tag_key | "=~" regex_lit | "IN (" tag_keys ")" ) . ``` ## Expressions ``` binary_op = "+" | "-" | "*" | "/" | "%" | "&" | "|" | "^" | "AND" | "OR" | "=" | "!=" | "<>" | "<" | "<=" | ">" | ">=" . expr = unary_expr { binary_op unary_expr } . unary_expr = "(" expr ")" | var_ref | time_lit | string_lit | int_lit | float_lit | bool_lit | duration_lit | regex_lit . ``` ## Default time range The default time range is the Unix epoch (`1970-01-01T00:00:00Z`) to *now*. ## Comments Use comments with InfluxQL statements to describe your queries. * A single line comment begins with two hyphens (`--`) and ends where InfluxDB detects a line break. This comment type cannot span several lines. * A multi-line comment begins with `/*` and ends with `*/`. This comment type can span several lines. Multi-line comments do not support nested multi-line comments. ## Other ``` alias = "AS" identifier . back_ref = ( policy_name ".:MEASUREMENT" ) | ( db_name "." [ policy_name ] ".:MEASUREMENT" ) . db_name = identifier . dimension = expr . dimensions = dimension { "," dimension } . field_key = identifier . field = expr [ alias ] . fields = field { "," field } . fill_option = "null" | "none" | "previous" | int_lit | float_lit | "linear" . host = string_lit . measurement = measurement_name | ( policy_name "." measurement_name ) | ( db_name "." [ policy_name ] "." measurement_name ) . measurements = measurement { "," measurement } . measurement_name = identifier | regex_lit . policy_name = identifier . retention_policy = identifier . retention_policy_name = "NAME" identifier . series_id = int_lit . sort_field = field_key [ ASC | DESC ] . sort_fields = sort_field { "," sort_field } . tag_key = identifier . tag_keys = tag_key { "," tag_key } . var_ref = measurement . ``` | Units | Meaning | | --- | --- | | Units | Meaning | | ns | nanoseconds (1 billionth of a second) | | u or µ | microseconds (1 millionth of a second) | | ms | milliseconds (1 thousandth of a second) | | s | second | | m | minute | | h | hour | | d | day | | w | week | | Type | Description | | --- | --- | | Type | Description | | float | 64-bit IEEE-754 floating-point number | | integer | 64-bit signed integer | | unsigned | 64-bit unsigned integer | | boolean | 1-bit, LSB encoded | | string | UTF-8 string | --- # InfluxQL feature support Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/feature-support/ InfluxQL is being rearchitected to work with the InfluxDB 3 storage engine. This process is ongoing and some InfluxQL features are still being implemented. This page provides information about the current implementation status of InfluxQL features. ## In-progress features ### SLIMIT clause InfluxQL in InfluxDB 3 Enterprise does not currently support the `SLIMIT` clause, which limits the number of [series](/influxdb3/enterprise/reference/glossary/#series)returned in query results. ### SOFFSET clause InfluxQL in InfluxDB 3 Enterprise does not currently support the `SOFFSET` clause, which specifies the number of [series](/influxdb3/enterprise/reference/glossary/#series)to skip before returning results. ### Metaqueries InfluxQL metaqueries return information about the schema of time series data stored in InfluxDB. The following table provides information about what metaqueries are available in InfluxDB 3 Enterprise: | Metaquery |Supported| |---------------------------|---------| | SHOW DATABASES | | |**SHOW RETENTION POLICIES**| **✓** | | **SHOW MEASUREMENTS** | **✓** | | SHOW SERIES | | | SHOW SERIES CARDINALITY | | | **SHOW TAG KEYS** | **✓** | | SHOW TAG KEY CARDINALITY | | | **SHOW TAG VALUES** | **✓** | |SHOW TAG VALUES CARDINALITY| | | **SHOW FIELD KEYS** | **✓** | |SHOW FIELD KEYS CARDINALITY| | > [!Note] > #### Cardinality metaqueries > > With the InfluxDB 3 storage engine, series cardinality is no longer a limiting > factor for database performance. > Cardinality-related metaqueries will likely not be supported with the InfluxDB 3 > storage engine. ## Function support ### Aggregate functions | Function |Supported| |-------------------------------------------------------------------------------------|---------| | [COUNT()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#count) | **✓** | |[DISTINCT()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#distinct)| **✓** | |[INTEGRAL()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#integral)| **✓** | | [MEAN()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#mean) | **✓** | | [MEDIAN()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#median) | **✓** | | [MODE()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#mode) | **✓** | | [SPREAD()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#spread) | **✓** | | [STDDEV()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#stddev) | **✓** | | [SUM()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#sum) | **✓** | ### Selector functions | Function |Supported| |----------------------------------------------------------------------------------------|---------| | [BOTTOM()](/influxdb3/enterprise/reference/influxql/functions/selectors/#bottom) | **✓** | | [FIRST()](/influxdb3/enterprise/reference/influxql/functions/selectors/#first) | **✓** | | [LAST()](/influxdb3/enterprise/reference/influxql/functions/selectors/#last) | **✓** | | [MAX()](/influxdb3/enterprise/reference/influxql/functions/selectors/#max) | **✓** | | [MIN()](/influxdb3/enterprise/reference/influxql/functions/selectors/#min) | **✓** | |[PERCENTILE()](/influxdb3/enterprise/reference/influxql/functions/selectors/#percentile)| **✓** | | SAMPLE() | | | [TOP()](/influxdb3/enterprise/reference/influxql/functions/selectors/#top) | **✓** | ### Transformations | Function |Supported| |--------------------------------------------------------------------------------------------------------------------------|---------| | [ABS()](/influxdb3/enterprise/reference/influxql/functions/transformations/#abs) | **✓** | | [ACOS()](/influxdb3/enterprise/reference/influxql/functions/transformations/#acos) | **✓** | | [ASIN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#asin) | **✓** | | [ATAN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#atan) | **✓** | | [ATAN2()](/influxdb3/enterprise/reference/influxql/functions/transformations/#atan2) | **✓** | | [CEIL()](/influxdb3/enterprise/reference/influxql/functions/transformations/#ceil) | **✓** | | [COS()](/influxdb3/enterprise/reference/influxql/functions/transformations/#cos) | **✓** | | [CUMULATIVE\_SUM()](/influxdb3/enterprise/reference/influxql/functions/transformations/#cumulative_sum) | **✓** | | [DERIVATIVE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#derivative) | **✓** | | [DIFFERENCE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#difference) | **✓** | | [ELAPSED()](/influxdb3/enterprise/reference/influxql/functions/transformations/#elapsed) | **✓** | | [EXP()](/influxdb3/enterprise/reference/influxql/functions/transformations/#exp) | **✓** | | [FLOOR()](/influxdb3/enterprise/reference/influxql/functions/transformations/#floor) | **✓** | | [LN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#ln) | **✓** | | [LOG()](/influxdb3/enterprise/reference/influxql/functions/transformations/#log) | **✓** | | [LOG2()](/influxdb3/enterprise/reference/influxql/functions/transformations/#log2) | **✓** | | [LOG10()](/influxdb3/enterprise/reference/influxql/functions/transformations/#log10) | **✓** | | [MOVING\_AVERAGE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#moving_average) | **✓** | |[NON\_NEGATIVE\_DERIVATIVE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#non_negative_derivative)| **✓** | |[NON\_NEGATIVE\_DIFFERENCE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#non_negative_difference)| **✓** | | [POW()](/influxdb3/enterprise/reference/influxql/functions/transformations/#pow) | **✓** | | [ROUND()](/influxdb3/enterprise/reference/influxql/functions/transformations/#round) | **✓** | | [SIN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#sin) | **✓** | | [SQRT()](/influxdb3/enterprise/reference/influxql/functions/transformations/#sqrt) | **✓** | | [TAN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#tan) | **✓** | ### Technical and predictive analysis | Function |Supported| |--------------------------------------|---------| | CHANDE\_MOMENTUM\_OSCILLATOR() | | |DOUBLE\_EXPONENTIAL\_MOVING\_AVERAGE()| | | EXPONENTIAL\_MOVING\_AVERAGE() | | | HOLT\_WINTERS() | | | HOLT\_WINTERS\_WITH\_FIT() | | | KAUFMANS\_EFFICIENCY\_RATIO() | | |KAUFMANS\_ADAPTIVE\_MOVING\_AVERAGE() | | | RELATIVE\_STRENGTH\_INDEX() | | |TRIPLE\_EXPONENTIAL\_MOVING\_AVERAGE()| | | TRIPLE\_EXPONENTIAL\_DERIVATIVE() | | ### Date and time functions | Function |Supported| |----------------------------------------------------------------------------|---------| | [now()](/influxdb3/enterprise/reference/influxql/functions/date-time/#now) | **✓** | |[time()](/influxdb3/enterprise/reference/influxql/functions/date-time/#time)| **✓** | | [tz()](/influxdb3/enterprise/reference/influxql/functions/date-time/#tz) | **✓** | ### Miscellaneous functions | Function |Supported| |-----------------------------------------------------------------------|---------| |[fill()](/influxdb3/enterprise/reference/influxql/functions/misc/#fill)| **✓** | | Metaquery | Supported | | --- | --- | | Metaquery | Supported | | SHOW DATABASES | | | SHOW RETENTION POLICIES | ✓ | | SHOW MEASUREMENTS | ✓ | | SHOW SERIES | | | SHOW SERIES CARDINALITY | | | SHOW TAG KEYS | ✓ | | SHOW TAG KEY CARDINALITY | | | SHOW TAG VALUES | ✓ | | SHOW TAG VALUES CARDINALITY | | | SHOW FIELD KEYS | ✓ | | SHOW FIELD KEYS CARDINALITY | | | Function | Supported | | --- | --- | | Function | Supported | | COUNT() | ✓ | | DISTINCT() | ✓ | | INTEGRAL() | ✓ | | MEAN() | ✓ | | MEDIAN() | ✓ | | MODE() | ✓ | | SPREAD() | ✓ | | STDDEV() | ✓ | | SUM() | ✓ | | Function | Supported | | --- | --- | | Function | Supported | | BOTTOM() | ✓ | | FIRST() | ✓ | | LAST() | ✓ | | MAX() | ✓ | | MIN() | ✓ | | PERCENTILE() | ✓ | | SAMPLE() | | | TOP() | ✓ | | Function | Supported | | --- | --- | | Function | Supported | | ABS() | ✓ | | ACOS() | ✓ | | ASIN() | ✓ | | ATAN() | ✓ | | ATAN2() | ✓ | | CEIL() | ✓ | | COS() | ✓ | | CUMULATIVE_SUM() | ✓ | | DERIVATIVE() | ✓ | | DIFFERENCE() | ✓ | | ELAPSED() | ✓ | | EXP() | ✓ | | FLOOR() | ✓ | | LN() | ✓ | | LOG() | ✓ | | LOG2() | ✓ | | LOG10() | ✓ | | MOVING_AVERAGE() | ✓ | | NON_NEGATIVE_DERIVATIVE() | ✓ | | NON_NEGATIVE_DIFFERENCE() | ✓ | | POW() | ✓ | | ROUND() | ✓ | | SIN() | ✓ | | SQRT() | ✓ | | TAN() | ✓ | | Function | Supported | | --- | --- | | Function | Supported | | CHANDE_MOMENTUM_OSCILLATOR() | | | DOUBLE_EXPONENTIAL_MOVING_AVERAGE() | | | EXPONENTIAL_MOVING_AVERAGE() | | | HOLT_WINTERS() | | | HOLT_WINTERS_WITH_FIT() | | | KAUFMANS_EFFICIENCY_RATIO() | | | KAUFMANS_ADAPTIVE_MOVING_AVERAGE() | | | RELATIVE_STRENGTH_INDEX() | | | TRIPLE_EXPONENTIAL_MOVING_AVERAGE() | | | TRIPLE_EXPONENTIAL_DERIVATIVE() | | | Function | Supported | | --- | --- | | Function | Supported | | now() | ✓ | | time() | ✓ | | tz() | ✓ | | Function | Supported | | --- | --- | | Function | Supported | | fill() | ✓ | --- # View InfluxQL functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/functions/ Use InfluxQL functions to aggregate, select, transform, analyze, and predict data. > [!Note] > #### Missing InfluxQL functions > > Some InfluxQL functions are in the process of being rearchitected to work with > the InfluxDB 3 storage engine. If a function you need is not here, check the[InfluxQL feature support page](/influxdb3/enterprise/reference/influxql/feature-support/#function-support)for more information. ## InfluxQL functions (by type) * [Aggregates](/influxdb3/enterprise/reference/influxql/functions/aggregates/) * [COUNT()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#count) * [DISTINCT()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#distinct) * [MEAN()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#mean) * [MEDIAN()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#median) * [MODE()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#mode) * [SPREAD()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#spread) * [STDDEV()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#stddev) * [SUM()](/influxdb3/enterprise/reference/influxql/functions/aggregates/#sum) * [Selectors](/influxdb3/enterprise/reference/influxql/functions/selectors/) * [BOTTOM()](/influxdb3/enterprise/reference/influxql/functions/selectors/#bottom) * [FIRST()](/influxdb3/enterprise/reference/influxql/functions/selectors/#first) * [LAST()](/influxdb3/enterprise/reference/influxql/functions/selectors/#last) * [MAX()](/influxdb3/enterprise/reference/influxql/functions/selectors/#max) * [MIN()](/influxdb3/enterprise/reference/influxql/functions/selectors/#min) * [PERCENTILE()](/influxdb3/enterprise/reference/influxql/functions/selectors/#percentile) * [TOP()](/influxdb3/enterprise/reference/influxql/functions/selectors/#top) * [Transformations](/influxdb3/enterprise/reference/influxql/functions/transformations/) * [ABS()](/influxdb3/enterprise/reference/influxql/functions/transformations/#abs) * [ACOS()](/influxdb3/enterprise/reference/influxql/functions/transformations/#acos) * [ASIN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#asin) * [ATAN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#atan) * [ATAN2()](/influxdb3/enterprise/reference/influxql/functions/transformations/#atan2) * [CEIL()](/influxdb3/enterprise/reference/influxql/functions/transformations/#ceil) * [COS()](/influxdb3/enterprise/reference/influxql/functions/transformations/#cos) * [CUMULATIVE\_SUM()](/influxdb3/enterprise/reference/influxql/functions/transformations/#cumulative_sum) * [DERIVATIVE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#derivative) * [DIFFERENCE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#difference) * [EXP()](/influxdb3/enterprise/reference/influxql/functions/transformations/#exp) * [FLOOR()](/influxdb3/enterprise/reference/influxql/functions/transformations/#floor) * [LN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#ln) * [LOG()](/influxdb3/enterprise/reference/influxql/functions/transformations/#log) * [LOG2()](/influxdb3/enterprise/reference/influxql/functions/transformations/#log2) * [LOG10()](/influxdb3/enterprise/reference/influxql/functions/transformations/#log10) * [MOVING\_AVERAGE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#moving_average) * [NON\_NEGATIVE\_DERIVATIVE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#non_negative_derivative) * [NON\_NEGATIVE\_DIFFERENCE()](/influxdb3/enterprise/reference/influxql/functions/transformations/#non_negative_difference) * [POW()](/influxdb3/enterprise/reference/influxql/functions/transformations/#pow) * [ROUND()](/influxdb3/enterprise/reference/influxql/functions/transformations/#round) * [SIN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#sin) * [SQRT()](/influxdb3/enterprise/reference/influxql/functions/transformations/#sqrt) * [TAN()](/influxdb3/enterprise/reference/influxql/functions/transformations/#tan) * [Date and time](/influxdb3/enterprise/reference/influxql/functions/date-time/) * [now()](/influxdb3/enterprise/reference/influxql/functions/date-time/#now) * [time()](/influxdb3/enterprise/reference/influxql/functions/date-time/#time) * [Miscellaneous](/influxdb3/enterprise/reference/influxql/functions/misc/) * [fill()](/influxdb3/enterprise/reference/influxql/functions/misc/#fill) --- # InfluxQL aggregate functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/functions/aggregates/ Use aggregate functions to assess, aggregate, and return values in your data. Aggregate functions return one row containing the aggregate values from each InfluxQL group. *Examples use the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ## COUNT() Returns the number of non-null [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql COUNT(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports all field types. #### Examples [](#count-the-number-of-non-null-values-in-a-field) Count the number of non-null values in a field ```sql SELECT COUNT(temp) FROM home ``` name: home | time |count| |--------------------|-----| |1970-01-01T00:00:00Z| 26 | [](#count-the-number-of-non-null-values-in-each-field) Count the number of non-null values in each field ```sql SELECT COUNT(*) FROM home ``` name: home | time |count\_co|count\_hum|count\_temp| |--------------------|---------|----------|-----------| |1970-01-01T00:00:00Z| 26 | 26 | 26 | [](#count-the-number-of-non-null-values-in-fields-where-the-field-key-matches-a-regular-expression) Count the number of non-null values in fields where the field key matches a regular expression ```sql SELECT COUNT(/^[th]/) FROM home ``` name: home | time |count\_hum|count\_temp| |--------------------|----------|-----------| |1970-01-01T00:00:00Z| 26 | 26 | [](#count-distinct-values-for-a-field) Count distinct values for a field InfluxQL supports nesting [`DISTINCT()`](#distinct) in `COUNT()`. ```sql SELECT COUNT(DISTINCT(co)) FROM home ``` name: home | time |count| |--------------------|-----| |1970-01-01T00:00:00Z| 12 | [](#count-the-number-of-non-null-field-values-within-time-windows-grouped-by-time) Count the number of non-null field values within time windows (grouped by time) ```sql SELECT COUNT(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |count| |--------------------|-----| |2022-01-01T06:00:00Z| 4 | |2022-01-01T12:00:00Z| 6 | |2022-01-01T18:00:00Z| 3 | ## DISTINCT() Returns the list of unique [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql DISTINCT(field_key) ``` #### Arguments * **field\_key**: Field key to return distinct values from. Supports all field types. #### Notable behaviors * InfluxQL supports nesting `DISTINCT()` with [`COUNT()`](#count-distinct-values-for-a-field). #### Examples [](#list-the-distinct-field-values) List the distinct field values ```sql SELECT DISTINCT(co) FROM home ``` name: home | time |distinct| |--------------------|--------| |1970-01-01T00:00:00Z| 0 | |1970-01-01T00:00:00Z| 1 | |1970-01-01T00:00:00Z| 3 | |1970-01-01T00:00:00Z| 4 | |1970-01-01T00:00:00Z| 7 | |1970-01-01T00:00:00Z| 5 | |1970-01-01T00:00:00Z| 9 | |1970-01-01T00:00:00Z| 18 | |1970-01-01T00:00:00Z| 14 | |1970-01-01T00:00:00Z| 22 | |1970-01-01T00:00:00Z| 17 | |1970-01-01T00:00:00Z| 26 | ## INTEGRAL() Returns the area under the curve for queried [field values](/influxdb3/enterprise/reference/glossary/#field-value)and converts those results into the summed area per **unit** of time. > [!Important] > * `INTEGRAL()` does not support [`fill()`](/influxdb3/enterprise/reference/influxql/group-by/#group-by-time-and-fill-gaps). > * `INTEGRAL()` supports int64 and float64 field value [data types](/influxdb3/enterprise/reference/glossary/#data-type). ```sql INTEGRAL(field_expression[, unit]) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). * **unit**: Unit of time to use when calculating the integral. Default is `1s` (one second). #### Examples [](#calculate-the-integral-for-a-field) Calculate the integral for a field ```sql SELECT INTEGRAL(co) FROM home WHERE room = 'Kitchen' ``` name: home | time |integral| |--------------------|--------| |1970-01-01T00:00:00Z| 266400 | [](#calculate-the-integral-for-a-field-and-specify-the-unit-option) Calculate the integral for a field and specify the unit option ```sql SELECT INTEGRAL(co, 1h) FROM home WHERE room = 'Kitchen' ``` name: home | time |integral| |--------------------|--------| |1970-01-01T00:00:00Z| 74 | [](#calculate-the-integral-for-_each_-field-and-specify-the-unit-option) Calculate the integral for *each* field and specify the unit option Return the area under the curve (in minutes) for the field values associated with each field key that stores numeric values in the `h2o_feet` measurement. The `h2o_feet` measurement has on numeric field: `water_level`. ```sql SELECT INTEGRAL(*, 1h) FROM home WHERE room = 'Kitchen' ``` name: home | time |integral\_co|integral\_hum|integral\_temp| |--------------------|------------|-------------|--------------| |1970-01-01T00:00:00Z| 74 | 435 | 272.25 | [](#calculate-the-integral-for-the-field-keys-that-matches-a-regular-expression) Calculate the integral for the field keys that matches a regular expression ```sql SELECT INTEGRAL(/^[th]/, 1h) FROM home WHERE room = 'Kitchen' ``` name: home | time |integral\_hum|integral\_temp| |--------------------|-------------|--------------| |1970-01-01T00:00:00Z| 435 | 272.25 | [](#calculate-the-integral-for-a-field-grouping-by-time) Calculate the integral for a field grouping by time ```sql SELECT INTEGRAL(co, 1h) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |integral| |--------------------|--------| |2022-01-01T06:00:00Z| 0 | |2022-01-01T12:00:00Z| 30 | |2022-01-01T18:00:00Z| 44 | ## MEAN() Returns the arithmetic mean (average) of [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql MEAN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Examples [](#calculate-the-mean-value-of-a-field) Calculate the mean value of a field ```sql SELECT MEAN(temp) FROM home ``` name: home | time | mean | |--------------------|------------------| |1970-01-01T00:00:00Z|22.396153846153844| [](#calculate-the-mean-value-of-each-field) Calculate the mean value of each field ```sql SELECT MEAN(*) FROM home ``` name: home | time | mean\_co |mean\_hum| mean\_temp | |--------------------|-----------------|---------|------------------| |1970-01-01T00:00:00Z|5.269230769230769| 36.15 |22.396153846153844| [](#calculate-the-mean-value-of-fields-where-the-field-key-matches-a-regular-expression) Calculate the mean value of fields where the field key matches a regular expression ```sql SELECT MEAN(/^[th]/) FROM home ``` name: home | time |mean\_hum| mean\_temp | |--------------------|---------|------------------| |1970-01-01T00:00:00Z| 36.15 |22.396153846153844| [](#calculate-the-mean-value-of-a-field-within-time-windows-grouped-by-time) Calculate the mean value of a field within time windows (grouped by time) ```sql SELECT MEAN(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time | mean | |--------------------|------------------| |2022-01-01T06:00:00Z| 22.275 | |2022-01-01T12:00:00Z|22.649999999999995| |2022-01-01T18:00:00Z|23.033333333333335| ## MEDIAN() Returns the middle value from a sorted list of [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql MEDIAN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Notable behaviors * `MEDIAN()` is nearly equivalent to[`PERCENTILE(field_key, 50)`](/influxdb3/enterprise/reference/influxql/functions/selectors/#percentile), except `MEDIAN()` returns the average of the two middle field values if the field contains an even number of values. #### Examples [](#calculate-the-median-value-of-a-field) Calculate the median value of a field ```sql SELECT MEDIAN(temp) FROM home ``` name: home | time |median| |--------------------|------| |1970-01-01T00:00:00Z|22.45 | [](#calculate-the-median-value-of-each-field) Calculate the median value of each field ```sql SELECT MEDIAN(*) FROM home ``` name: home | time |median\_co|median\_hum|median\_temp| |--------------------|----------|-----------|------------| |1970-01-01T00:00:00Z| 1 | 36.05 | 22.45 | [](#calculate-the-median-value-of-fields-where-the-field-key-matches-a-regular-expression) Calculate the median value of fields where the field key matches a regular expression ```sql SELECT MEDIAN(/^[th]/) FROM home ``` name: home | time |median\_hum|median\_temp| |--------------------|-----------|------------| |1970-01-01T00:00:00Z| 36.05 | 22.45 | [](#calculate-the-median-value-of-a-field-within-time-windows-grouped-by-time) Calculate the median value of a field within time windows (grouped by time) ```sql SELECT MEDIAN(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time | median | |--------------------|------------------| |2022-01-01T06:00:00Z|22.549999999999997| |2022-01-01T12:00:00Z| 22.7 | |2022-01-01T18:00:00Z| 23.1 | ## MODE() Returns the most frequent value in a list of[field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql MODE(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports all field types. #### Notable behaviors * `MODE()` returns the field value with the earliest[timestamp](/influxdb3/enterprise/reference/glossary/#timestamp)if there’s a tie between two or more values for the maximum number of occurrences. #### Examples [](#calculate-the-mode-value-of-a-field) Calculate the mode value of a field ```sql SELECT MODE(co) FROM home ``` name: home | time |mode| |--------------------|----| |1970-01-01T00:00:00Z| 0 | [](#calculate-the-mode-value-of-each-field) Calculate the mode value of each field ```sql SELECT MODE(*) FROM home ``` name: home | time |mode\_co|mode\_hum|mode\_temp| |--------------------|--------|---------|----------| |1970-01-01T00:00:00Z| 0 | 36 | 22.7 | [](#calculate-the-mode-of-field-keys-that-match-a-regular-expression) Calculate the mode of field keys that match a regular expression ```sql SELECT MODE(/^[th]/) FROM home ``` name: home | time |mode\_hum|mode\_temp| |--------------------|---------|----------| |1970-01-01T00:00:00Z| 36 | 22.7 | [](#calculate-the-mode-a-field-within-time-windows-grouped-by-time) Calculate the mode a field within time windows (grouped by time) ```sql SELECT MODE(co) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |mode| |--------------------|----| |2022-01-01T06:00:00Z| 0 | |2022-01-01T12:00:00Z| 1 | |2022-01-01T18:00:00Z| 18 | ## SPREAD() Returns the difference between the minimum and maximum[field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql SPREAD(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Examples [](#calculate-the-spread-of-a-field) Calculate the spread of a field ```sql SELECT SPREAD(temp) FROM home ``` name: home | time | spread | |--------------------|------------------| |1970-01-01T00:00:00Z|2.3000000000000007| [](#calculate-the-spread-of-each-field) Calculate the spread of each field ```sql SELECT SPREAD(*) FROM home ``` name: home | time |spread\_co|spread\_hum| spread\_temp | |--------------------|----------|-----------|------------------| |1970-01-01T00:00:00Z| 26 | 1 |2.3000000000000007| [](#calculate-the-spread-of-field-keys-that-match-a-regular-expression) Calculate the spread of field keys that match a regular expression ```sql SELECT SPREAD(/^[th]/) FROM home ``` name: home | time |spread\_hum| spread\_temp | |--------------------|-----------|------------------| |1970-01-01T00:00:00Z| 1 |2.3000000000000007| [](#calculate-the-spread-of-a-field-within-time-windows-grouped-by-time) Calculate the spread of a field within time windows (grouped by time) ```sql SELECT SPREAD(co) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |spread| |--------------------|------| |2022-01-01T06:00:00Z| 0 | |2022-01-01T12:00:00Z| 9 | |2022-01-01T18:00:00Z| 8 | ## STDDEV() Returns the standard deviation of [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql STDDEV(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Examples [](#calculate-the-standard-deviation-of-a-field) Calculate the standard deviation of a field ```sql SELECT STDDEV(temp) FROM home ``` name: home | time | stddev | |--------------------|------------------| |1970-01-01T00:00:00Z|0.5553238833191091| [](#calculate-the-standard-deviation-of-each-field) Calculate the standard deviation of each field ```sql SELECT STDDEV(*) FROM home ``` name: home | time | stddev\_co | stddev\_hum | stddev\_temp | |--------------------|-----------------|-------------------|------------------| |1970-01-01T00:00:00Z|7.774613519951676|0.25495097567963926|0.5553238833191091| [](#calculate-the-standard-deviation-of-fields-where-the-field-key-matches-a-regular-expression) Calculate the standard deviation of fields where the field key matches a regular expression ```sql SELECT STDDEV(/^[th]/) FROM home ``` name: home | time | stddev\_hum | stddev\_temp | |--------------------|-------------------|------------------| |1970-01-01T00:00:00Z|0.25495097567963926|0.5553238833191091| [](#calculate-the-standard-deviation-of-a-field-within-time-windows-grouped-by-time) Calculate the standard deviation of a field within time windows (grouped by time) ```sql SELECT STDDEV(co) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time | stddev | |--------------------|------------------| |2022-01-01T06:00:00Z| 0 | |2022-01-01T12:00:00Z|3.6742346141747673| |2022-01-01T18:00:00Z| 4 | ## SUM() Returns the sum of [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql SUM(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Examples [](#calculate-the-sum-of-values-in-a-field) Calculate the sum of values in a field ```sql SELECT SUM(co) FROM home ``` name: home | time |sum| |--------------------|---| |1970-01-01T00:00:00Z|137| [](#calculate-the-sum-of-values-in-each-field) Calculate the sum of values in each field ```sql SELECT SUM(*) FROM home ``` name: home | time |sum\_co|sum\_hum|sum\_temp| |--------------------|-------|--------|---------| |1970-01-01T00:00:00Z| 137 | 939.9 | 582.3 | [](#calculate-the-sum-of-values-for-fields-where-the-field-key-matches-a-regular-expression) Calculate the sum of values for fields where the field key matches a regular expression ```sql SELECT SUM(/^[th]/) FROM home ``` name: home | time |sum\_hum|sum\_temp| |--------------------|--------|---------| |1970-01-01T00:00:00Z| 939.9 | 582.3 | [](#calculate-the-sum-of-values-in-a-field-within-time-windows-grouped-by-time) Calculate the sum of values in a field within time windows (grouped by time) ```sql SELECT SUM(co) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |sum| |--------------------|---| |2022-01-01T06:00:00Z| 0 | |2022-01-01T12:00:00Z|21 | |2022-01-01T18:00:00Z|66 | #### Related * [Aggregate data with InfluxQL](/influxdb3/enterprise/query-data/influxql/aggregate-select/) | time | count | | --- | --- | | time | count | | 1970-01-01T00:00:00Z | 26 | | time | count_co | count_hum | count_temp | | --- | --- | --- | --- | | time | count_co | count_hum | count_temp | | 1970-01-01T00:00:00Z | 26 | 26 | 26 | | time | count_hum | count_temp | | --- | --- | --- | | time | count_hum | count_temp | | 1970-01-01T00:00:00Z | 26 | 26 | | time | count | | --- | --- | | time | count | | 1970-01-01T00:00:00Z | 12 | | time | count | | --- | --- | | time | count | | 2022-01-01T06:00:00Z | 4 | | 2022-01-01T12:00:00Z | 6 | | 2022-01-01T18:00:00Z | 3 | | time | distinct | | --- | --- | | time | distinct | | 1970-01-01T00:00:00Z | 0 | | 1970-01-01T00:00:00Z | 1 | | 1970-01-01T00:00:00Z | 3 | | 1970-01-01T00:00:00Z | 4 | | 1970-01-01T00:00:00Z | 7 | | 1970-01-01T00:00:00Z | 5 | | 1970-01-01T00:00:00Z | 9 | | 1970-01-01T00:00:00Z | 18 | | 1970-01-01T00:00:00Z | 14 | | 1970-01-01T00:00:00Z | 22 | | 1970-01-01T00:00:00Z | 17 | | 1970-01-01T00:00:00Z | 26 | | time | integral | | --- | --- | | time | integral | | 1970-01-01T00:00:00Z | 266400 | | time | integral | | --- | --- | | time | integral | | 1970-01-01T00:00:00Z | 74 | | time | integral_co | integral_hum | integral_temp | | --- | --- | --- | --- | | time | integral_co | integral_hum | integral_temp | | 1970-01-01T00:00:00Z | 74 | 435 | 272.25 | | time | integral_hum | integral_temp | | --- | --- | --- | | time | integral_hum | integral_temp | | 1970-01-01T00:00:00Z | 435 | 272.25 | | time | integral | | --- | --- | | time | integral | | 2022-01-01T06:00:00Z | 0 | | 2022-01-01T12:00:00Z | 30 | | 2022-01-01T18:00:00Z | 44 | | time | mean | | --- | --- | | time | mean | | 1970-01-01T00:00:00Z | 22.396153846153844 | | time | mean_co | mean_hum | mean_temp | | --- | --- | --- | --- | | time | mean_co | mean_hum | mean_temp | | 1970-01-01T00:00:00Z | 5.269230769230769 | 36.15 | 22.396153846153844 | | time | mean_hum | mean_temp | | --- | --- | --- | | time | mean_hum | mean_temp | | 1970-01-01T00:00:00Z | 36.15 | 22.396153846153844 | | time | mean | | --- | --- | | time | mean | | 2022-01-01T06:00:00Z | 22.275 | | 2022-01-01T12:00:00Z | 22.649999999999995 | | 2022-01-01T18:00:00Z | 23.033333333333335 | | time | median | | --- | --- | | time | median | | 1970-01-01T00:00:00Z | 22.45 | | time | median_co | median_hum | median_temp | | --- | --- | --- | --- | | time | median_co | median_hum | median_temp | | 1970-01-01T00:00:00Z | 1 | 36.05 | 22.45 | | time | median_hum | median_temp | | --- | --- | --- | | time | median_hum | median_temp | | 1970-01-01T00:00:00Z | 36.05 | 22.45 | | time | median | | --- | --- | | time | median | | 2022-01-01T06:00:00Z | 22.549999999999997 | | 2022-01-01T12:00:00Z | 22.7 | | 2022-01-01T18:00:00Z | 23.1 | | time | mode | | --- | --- | | time | mode | | 1970-01-01T00:00:00Z | 0 | | time | mode_co | mode_hum | mode_temp | | --- | --- | --- | --- | | time | mode_co | mode_hum | mode_temp | | 1970-01-01T00:00:00Z | 0 | 36 | 22.7 | | time | mode_hum | mode_temp | | --- | --- | --- | | time | mode_hum | mode_temp | | 1970-01-01T00:00:00Z | 36 | 22.7 | | time | mode | | --- | --- | | time | mode | | 2022-01-01T06:00:00Z | 0 | | 2022-01-01T12:00:00Z | 1 | | 2022-01-01T18:00:00Z | 18 | | time | spread | | --- | --- | | time | spread | | 1970-01-01T00:00:00Z | 2.3000000000000007 | | time | spread_co | spread_hum | spread_temp | | --- | --- | --- | --- | | time | spread_co | spread_hum | spread_temp | | 1970-01-01T00:00:00Z | 26 | 1 | 2.3000000000000007 | | time | spread_hum | spread_temp | | --- | --- | --- | | time | spread_hum | spread_temp | | 1970-01-01T00:00:00Z | 1 | 2.3000000000000007 | | time | spread | | --- | --- | | time | spread | | 2022-01-01T06:00:00Z | 0 | | 2022-01-01T12:00:00Z | 9 | | 2022-01-01T18:00:00Z | 8 | | time | stddev | | --- | --- | | time | stddev | | 1970-01-01T00:00:00Z | 0.5553238833191091 | | time | stddev_co | stddev_hum | stddev_temp | | --- | --- | --- | --- | | time | stddev_co | stddev_hum | stddev_temp | | 1970-01-01T00:00:00Z | 7.774613519951676 | 0.25495097567963926 | 0.5553238833191091 | | time | stddev_hum | stddev_temp | | --- | --- | --- | | time | stddev_hum | stddev_temp | | 1970-01-01T00:00:00Z | 0.25495097567963926 | 0.5553238833191091 | | time | stddev | | --- | --- | | time | stddev | | 2022-01-01T06:00:00Z | 0 | | 2022-01-01T12:00:00Z | 3.6742346141747673 | | 2022-01-01T18:00:00Z | 4 | | time | sum | | --- | --- | | time | sum | | 1970-01-01T00:00:00Z | 137 | | time | sum_co | sum_hum | sum_temp | | --- | --- | --- | --- | | time | sum_co | sum_hum | sum_temp | | 1970-01-01T00:00:00Z | 137 | 939.9 | 582.3 | | time | sum_hum | sum_temp | | --- | --- | --- | | time | sum_hum | sum_temp | | 1970-01-01T00:00:00Z | 939.9 | 582.3 | | time | sum | | --- | --- | | time | sum | | 2022-01-01T06:00:00Z | 0 | | 2022-01-01T12:00:00Z | 21 | | 2022-01-01T18:00:00Z | 66 | --- # InfluxQL date and time functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/functions/date-time/ Use InfluxQL date and time functions to perform time-related operations. ## now() Returns the current system time (UTC).*Supported only in the [`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/).* ```sql now() ``` ## time() Used in the [`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/)to group data into time-based intervals, also known as “windows”, using the specified interval. Timestamps in the `time` column are updated to the start boundary of the window they’re in and grouped by `time`. Windows use preset round-number boundaries based on the specified interval that are independent of time conditions in the[`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/). This operation can be used to do the following: * Downsample data by aggregating multiple points in each window into a single point per window. * Normalize irregular time series data to occur at regular intervals. *Supported only in the [`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/).* ```sql time(interval[, offset]) ``` #### Arguments * **interval**: Duration literal that specifies the window interval. * **offset**: Duration literal that shifts preset time boundaries forward or backward. Can be positive or negative. *Default is `0s`.* ##### Examples [](#downsample-data-into-time-based-intervals) Downsample data into time-based intervals The following example uses the[Bitcoin price sample dataset](/influxdb3/enterprise/reference/sample-data/#bitcoin-price-data). ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'GBP' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-15T00:00:00Z' GROUP BY time(2d) ``` name: bitcoin | time | mean | |--------------------|------------------| |2023-05-01T00:00:00Z|23680.120447159094| |2023-05-03T00:00:00Z|24048.71484033149 | |2023-05-05T00:00:00Z| 24461.9194901099 | |2023-05-07T00:00:00Z|23796.43801933702 | |2023-05-09T00:00:00Z|23118.709889285707| |2023-05-11T00:00:00Z|22465.008364444446| |2023-05-13T00:00:00Z|22499.464763186803| ## tz() Applies a timezone offset to timestamps in query results. Offsets include any seasonal offset such as Daylight Savings Time (DST) or British Summer Time (BST).*Supported only in the [time zone clause](/influxdb3/enterprise/reference/influxql/time-and-timezone/#time-zone-clause).* ```sql tz(time_zone) ``` #### Arguments * **time\_zone**: Timezone string literal to adjust times to. Uses timezone names defined in the[Internet Assigned Numbers Authority time zone database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List). #### Examples [](#return-the-utc-offset-for-chicagos-time-zone) Return the UTC offset for Chicago’s time zone The following example uses the[Home sensor sample dataset](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' tz('America/Chicago') ``` name: home | time |co |hum | room |temp| |-------------------------|---|----|-------|----| |2022-01-01T02:00:00-06:00| 0 |35.9|Kitchen| 21 | |2022-01-01T03:00:00-06:00| 0 |36.2|Kitchen| 23 | |2022-01-01T04:00:00-06:00| 0 |36.1|Kitchen|22.7| |2022-01-01T05:00:00-06:00| 0 | 36 |Kitchen|22.4| |2022-01-01T06:00:00-06:00| 0 | 36 |Kitchen|22.5| | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 23680.120447159094 | | 2023-05-03T00:00:00Z | 24048.71484033149 | | 2023-05-05T00:00:00Z | 24461.9194901099 | | 2023-05-07T00:00:00Z | 23796.43801933702 | | 2023-05-09T00:00:00Z | 23118.709889285707 | | 2023-05-11T00:00:00Z | 22465.008364444446 | | 2023-05-13T00:00:00Z | 22499.464763186803 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T02:00:00-06:00 | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T03:00:00-06:00 | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T04:00:00-06:00 | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T05:00:00-06:00 | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T06:00:00-06:00 | 0 | 36 | Kitchen | 22.5 | --- # InfluxQL miscellaneous functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/functions/misc/ Use InfluxQL miscellaneous functions to perform different operations in InfluxQL queries. ## fill() Fills *null* field values returned from empty time windows in `GROUP BY time()`queries with a specified fill value. *Supported only in the [`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/).* ```sql fill(behavior) ``` #### Arguments * **behavior**: Defines the behavior of the fill operation. If no `FILL` clause is included, the default behavior is `fill(null)`. The following options are available: * **numeric literal**: Replaces null values with the specified numeric literal. * **linear**: Uses linear interpolation between existing values to replace null values. * **none**: Removes rows with null field values. * **null**: Keeps null values and associated timestamps. * **previous**: Replaces null values with the most recent non-null value. #### Examples The following example uses the[Bitcoin price sample dataset](/influxdb3/enterprise/reference/sample-data/#bitcoin-price-data). [fill(numeric\_literal)](#fillnumeric_literal)[fill(linear)](#filllinear)[fill(none)](#fillnone)[fill(null)](#fillnull)[fill(previous)](#fillprevious) ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(0) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z| 0 | |2023-05-01T01:30:00Z|29263.2886| ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(linear) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z|29285.3651| |2023-05-01T01:30:00Z|29263.2886| ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(none) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:30:00Z|29263.2886| ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(null) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z| | |2023-05-01T01:30:00Z|29263.2886| ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(previous) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z|29307.4416| |2023-05-01T01:30:00Z|29263.2886| | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | 0 | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | 29285.3651 | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | 29307.4416 | | 2023-05-01T01:30:00Z | 29263.2886 | --- # InfluxQL selector functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/functions/selectors/ Use selector functions to assess, select, and return values in your data. Selector functions return one or more rows with the selected values from each InfluxQL group. *Examples use the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* > [!Important] > #### Missing InfluxQL functions > > Some InfluxQL functions are in the process of being rearchitected to work with > the InfluxDB 3 storage engine. If a function you need is not here, check the[InfluxQL feature support page](/influxdb3/enterprise/reference/influxql/feature-support/#function-support)for more information. ## BOTTOM() Returns the smallest `N` [field values](/influxdb3/enterprise/reference/glossary/#field-value).`BOTTOM()` supports int64 and float64 field value [data types](/influxdb3/enterprise/reference/glossary/#field-value). ```sql BOTTOM(field_expression[, tag_expression_1[, ..., tag_expression_n]], N) ``` > [!Note] > **Note:** `BOTTOM()` returns the field value with the earliest timestamp if > there’s a tie between two or more values for the smallest value. #### Arguments * **field\_expression**: Expression to identify the field to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key)or constant. * **tag\_expression**: Expression to identify a tag key to segment by. Can be a [tag key](/influxdb3/enterprise/reference/glossary/#tag-key)or constant. Comma-delimit multiple tags. * **N**: Number of results to return from each InfluxQL group or specified tag segment. #### Notable behaviors * `BOTTOM()` [maintains original timestamps when grouping by time](#timestamps-when-grouping-by-time). * `BOTTOM()` [may return fewer points than expected](#selector-functions-may-return-fewer-points-than-expected). #### Examples [](#select-the-bottom-three-values-of-a-field) Select the bottom three values of a field ```sql SELECT BOTTOM(temp, 3) FROM home ``` name: home | time |bottom| |--------------------|------| |2022-01-01T08:00:00Z| 21 | |2022-01-01T08:00:00Z| 21.1 | |2022-01-01T09:00:00Z| 21.4 | [](#select-the-bottom-field-value-for-two-unique-tag-values) Select the bottom field value for two unique tag values ```sql SELECT BOTTOM(temp, room, 2) FROM home ``` name: home | time |bottom| room | |--------------------|------|-----------| |2022-01-01T08:00:00Z| 21 | Kitchen | |2022-01-01T08:00:00Z| 21.1 |Living Room| [](#select-the-bottom-three-field-values-and-the-tag-value-associated-with-each) Select the bottom three field values and the tag value associated with each ```sql SELECT BOTTOM(temp, 3), room FROM home ``` name: home | time |bottom| room | |--------------------|------|-----------| |2022-01-01T08:00:00Z| 21 | Kitchen | |2022-01-01T08:00:00Z| 21.1 |Living Room| |2022-01-01T09:00:00Z| 21.4 |Living Room| [](#select-the-bottom-field-values-for-unique-tag-values-and-within-time-windows-grouped-by-time) Select the bottom field values for unique tag values and within time windows (grouped by time) ```sql SELECT BOTTOM(temp, room, 2) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' GROUP BY time(2h) ``` name: home | time |bottom| room | |--------------------|------|-----------| |2022-01-01T08:00:00Z| 21 | Kitchen | |2022-01-01T08:00:00Z| 21.1 |Living Room| |2022-01-01T10:00:00Z| 21.8 |Living Room| |2022-01-01T11:00:00Z| 22.4 | Kitchen | |2022-01-01T12:00:00Z| 22.2 |Living Room| |2022-01-01T12:00:00Z| 22.5 | Kitchen | Notice that when grouping by time, `BOTTOM()`[maintains the point’s original timestamp](#timestamps-when-grouping-by-time). ## FIRST() Returns the [field value](/influxdb3/enterprise/reference/glossary/#field-value) with the oldest timestamp. ```sql FIRST(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports all field [data types](/influxdb/v2/query-data/influxql/explore-data/select/#data-types). #### Notable behaviors * `FIRST()` [overrides original timestamps when grouping by time](#timestamps-when-grouping-by-time) #### Examples [](#select-the-first-value-for-a-field) Select the first value for a field ```sql SELECT FIRST(temp) FROM home ``` name: home | time |first| |--------------------|-----| |2022-01-01T08:00:00Z|21.1 | [](#select-the-first-value-from-each-field) Select the first value from each field ```sql SELECT FIRST(*) FROM home ``` name: home | time |first\_co|first\_hum|first\_temp| |--------------------|---------|----------|-----------| |1970-01-01T00:00:00Z| 0 | 35.9 | 21.1 | [](#select-the-first-value-from-field-keys-that-match-a-regular-expression) Select the first value from field keys that match a regular expression ```sql SELECT FIRST(/^[th]/) FROM home ``` name: home | time |first\_hum|first\_temp| |--------------------|----------|-----------| |1970-01-01T00:00:00Z| 35.9 | 21.1 | [](#select-the-first-value-from-a-field-within-time-windows-grouped-by-time) Select the first value from a field within time windows (grouped by time) ``` SELECT FIRST(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |first| |--------------------|-----| |2022-01-01T06:00:00Z| 21 | |2022-01-01T12:00:00Z|22.5 | |2022-01-01T18:00:00Z|23.3 | Notice that when grouping by time, `FIRST()`[overrides the point’s original timestamp](#timestamps-when-grouping-by-time). ## LAST() Returns the [field value](/influxdb3/enterprise/reference/glossary/#field-value) with the most recent timestamp. ```sql LAST(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports all field [data types](/influxdb/v2/query-data/influxql/explore-data/select/#data-types). #### Notable behaviors * `LAST()` [overrides original timestamps when grouping by time](#timestamps-when-grouping-by-time) #### Examples [](#select-the-last-value-for-a-field) Select the last value for a field ```sql SELECT LAST(temp) FROM home ``` name: home | time |last| |--------------------|----| |2022-01-01T20:00:00Z|22.7| [](#select-the-last-value-from-each-field) Select the last value from each field ```sql SELECT LAST(*) FROM home ``` name: home | time |last\_co|last\_hum|last\_temp| |--------------------|--------|---------|----------| |1970-01-01T00:00:00Z| 26 | 36.5 | 22.7 | [](#select-the-last-value-from-field-keys-that-match-a-regular-expression) Select the last value from field keys that match a regular expression ```sql SELECT LAST(/^[th]/) FROM home ``` name: home | time |last\_hum|last\_temp| |--------------------|---------|----------| |1970-01-01T00:00:00Z| 36.5 | 22.7 | [](#select-the-last-value-from-a-field-within-time-windows-grouped-by-time) Select the last value from a field within time windows (grouped by time) ``` SELECT LAST(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |last| |--------------------|----| |2022-01-01T06:00:00Z|22.4| |2022-01-01T12:00:00Z|22.7| |2022-01-01T18:00:00Z|22.7| Notice that when grouping by time, `LAST()`[overrides the point’s original timestamp](#timestamps-when-grouping-by-time). ## MAX() Returns the greatest [field value](/influxdb3/enterprise/reference/glossary/#field-value). ```sql MAX(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Notable behaviors * `MAX()` [overrides original timestamps when grouping by time](#timestamps-when-grouping-by-time). #### Examples [](#select-the-maximum-value-from-a-field) Select the maximum value from a field ```sql SELECT MAX(co) FROM home ``` name: home | time |max| |--------------------|---| |2022-01-01T20:00:00Z|26 | [](#select-the-maximum-value-from-each-field) Select the maximum value from each field ```sql SELECT MAX(*) FROM home ``` name: home | time |max\_co|max\_hum|max\_temp| |--------------------|-------|--------|---------| |1970-01-01T00:00:00Z| 26 | 36.9 | 23.3 | [](#select-the-maximum-value-from-field-keys-that-match-a-regular-expression) Select the maximum value from field keys that match a regular expression ```sql SELECT MAX(/^[th]/) FROM home ``` name: home | time |max\_hum|max\_temp| |--------------------|--------|---------| |1970-01-01T00:00:00Z| 36.9 | 23.3 | [](#select-the-maximum-value-from-a-field-within-time-windows-grouped-by-time) Select the maximum value from a field within time windows (grouped by time) ```sql SELECT MAX(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |max | |--------------------|----| |2022-01-01T06:00:00Z| 23 | |2022-01-01T12:00:00Z|22.8| |2022-01-01T18:00:00Z|23.3| Notice that when grouping by time, `MAX()`[overrides the point’s original timestamp](#timestamps-when-grouping-by-time). ## MIN() Returns the lowest [field value](/influxdb3/enterprise/reference/glossary/#field-value). ```sql MIN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. #### Notable behaviors * `MIN()` [overrides original timestamps when grouping by time](#timestamps-when-grouping-by-time). #### Examples [](#select-the-minimum-value-from-a-field) Select the minimum value from a field ```sql SELECT MIN(temp) FROM home ``` name: home | time |min| |--------------------|---| |2022-01-01T08:00:00Z|21 | [](#select-the-minimum-value-from-each-field) Select the minimum value from each field ```sql SELECT MIN(*) FROM home ``` name: home | time |min\_co|min\_hum|min\_temp| |--------------------|-------|--------|---------| |1970-01-01T00:00:00Z| 0 | 35.9 | 21 | [](#select-the-minimum-value-from-field-keys-that-match-a-regular-expression) Select the minimum value from field keys that match a regular expression ```sql SELECT MIN(/^[th]/) FROM home ``` name: home | time |min\_hum|min\_temp| |--------------------|--------|---------| |1970-01-01T00:00:00Z| 35.9 | 21 | [](#select-the-minimum-value-from-a-field-within-time-windows-grouped-by-time) Select the minimum value from a field within time windows (grouped by time) ```sql SELECT MIN(temp) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |min | |--------------------|----| |2022-01-01T06:00:00Z| 21 | |2022-01-01T12:00:00Z|22.4| |2022-01-01T18:00:00Z|22.7| Notice that when grouping by time, `MIN()`[overrides the point’s original timestamp](#timestamps-when-grouping-by-time). ## PERCENTILE() Returns the `N`th percentile [field value](/influxdb3/enterprise/reference/glossary/#field-value). ```sql PERCENTILE(field_expression, N) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports **numeric fields**. * **N**: Percentile to return. Must be an integer or float value **greater than 0 and less than or equal to 100**. #### Notable behaviors * `PERCENTILE()` [overrides original timestamps when grouping by time](#timestamps-when-grouping-by-time). * `PERCENTILE(example_field, 100)` is equivalent to [`MAX(example_field)`](#max). * `PERCENTILE(example_field, 50)` is nearly equivalent to[`MEDIAN(example_field)`](/influxdb3/enterprise/reference/influxql/functions/aggregates/#median), except `MEDIAN()` returns the average of the two middle values if the field contains an even number of values. * `PERCENTILE(example_field, 0)` returns *null*. #### Examples [](#select-the-50th-percentile-value-from-a-field) Select the 50th percentile value from a field ```sql SELECT PERCENTILE(temp, 50) FROM home ``` name: home | time |percentile| |--------------------|----------| |2022-01-01T11:00:00Z| 22.4 | [](#select-the-50th-percentile-value-from-each-field) Select the 50th percentile value from each field ```sql SELECT PERCENTILE(*, 50) FROM home ``` name: home | time |percentile\_co|percentile\_hum|percentile\_temp| |--------------------|--------------|---------------|----------------| |1970-01-01T00:00:00Z| 1 | 36 | 22.4 | [](#select-the-50th-percentile-value-from-field-keys-that-match-a-regular-expression) Select the 50th percentile value from field keys that match a regular expression ```sql SELECT PERCENTILE(/^[th]/, 50) FROM home ``` name: home | time |percentile\_hum|percentile\_temp| |--------------------|---------------|----------------| |1970-01-01T00:00:00Z| 36 | 22.4 | [](#select-the-50th-percentile-value-from-a-field-within-time-windows-grouped-by-time) Select the 50th percentile value from a field within time windows (grouped by time) ```sql SELECT PERCENTILE(temp, 50) FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY time(6h) ``` name: home | time |percentile| |--------------------|----------| |2022-01-01T06:00:00Z| 22.4 | |2022-01-01T12:00:00Z| 22.7 | |2022-01-01T18:00:00Z| 23.1 | Notice that when grouping by time, `PERCENTILE()`[overrides the point’s original timestamp](#timestamps-when-grouping-by-time). ## TOP() Returns the greatest `N` [field values](/influxdb3/enterprise/reference/glossary/#field-value).`TOP()` supports int64 and float64 field value [data types](/influxdb3/enterprise/reference/glossary/#field-value). ```sql TOP(field_expression[, tag_expression_1[, ..., tag_expression_n]], N) ``` > [!Note] > **Note:** `TOP()` returns the field value with the earliest timestamp if there’s > a tie between two or more values for the greatest value. #### Arguments * **field\_expression**: Expression to identify the field to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key)or constant. * **tag\_expression**: Expression to identify a tag key to segment by. Can be a [tag key](/influxdb3/enterprise/reference/glossary/#tag-key)or constant. Comma-delimit multiple tags. * **N**: Number of results to return from each InfluxQL group or specified tag segment. #### Notable behaviors * `TOP()` [maintains original timestamps when grouping by time](#timestamps-when-grouping-by-time). * `TOP()` [may return fewer points than expected](#selector-functions-may-return-fewer-points-than-expected). #### Examples [](#select-the-top-three-values-of-a-field) Select the top three values of a field ```sql SELECT TOP(temp, 3) FROM home ``` name: home | time |top | |--------------------|----| |2022-01-01T09:00:00Z| 23 | |2022-01-01T18:00:00Z|23.3| |2022-01-01T19:00:00Z|23.1| [](#select-the-top-field-value-for-two-unique-tag-values) Select the top field value for two unique tag values ```sql SELECT TOP(temp, room, 2) FROM home ``` name: home | time |top | room | |--------------------|----|-----------| |2022-01-01T18:00:00Z|23.3| Kitchen | |2022-01-01T18:00:00Z|22.8|Living Room| [](#select-the-top-three-field-values-and-the-tag-value-associated-with-each) Select the top three field values and the tag value associated with each ```sql SELECT TOP(temp, 3), room FROM home ``` name: home | time |top | room | |--------------------|----|-------| |2022-01-01T09:00:00Z| 23 |Kitchen| |2022-01-01T18:00:00Z|23.3|Kitchen| |2022-01-01T19:00:00Z|23.1|Kitchen| [](#select-the-top-field-values-for-unique-tag-values-and-within-time-windows-grouped-by-time) Select the top field values for unique tag values and within time windows (grouped by time) ```sql SELECT TOP(temp, room, 2) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' GROUP BY time(2h) ``` name: home | time |top | room | |--------------------|----|-----------| |2022-01-01T09:00:00Z| 23 | Kitchen | |2022-01-01T09:00:00Z|21.4|Living Room| |2022-01-01T10:00:00Z|22.7| Kitchen | |2022-01-01T11:00:00Z|22.2|Living Room| |2022-01-01T12:00:00Z|22.5| Kitchen | |2022-01-01T12:00:00Z|22.2|Living Room| Notice that when grouping by time, `TOP()`[maintains the point’s original timestamp](#timestamps-when-grouping-by-time). ## Notable behaviors of selector functions ### Timestamps when grouping by time When using selector functions with a `GROUP BY time()` clause, most selector functions return the timestamp of the starting boundary for each time interval. However functions with an `N` argument that specifies the number of results to return per group maintain the original timestamp of each returned point. ###### Return the start time of each time interval * [FIRST()](#first) * [LAST()](#last) * [MAX()](#max) * [MIN()](#min) * [PERCENTILE()](#percentile) ###### Maintain the original timestamp * [BOTTOM()](#bottom) * [TOP()](#top) ### Selector functions may return fewer points than expected Queries that use the following selector functions with an `N` argument may return fewer points than expected. * [BOTTOM()](#bottom) * [TOP()](#top) If the InfluxQL group or specified tag key contains `X` points or unique tag values and `X` is less than `N`, the function returns `X` results instead of `N`for each group or tag value. [](#view-example-for-fnfield_key-n) View example for `FN(field_key, N)` The example below selects the bottom 5 temperatures from the Kitchen between2022-01-01T08:00:00Zand 2022-01-01T10:00:00Z. There are only 3 points recorded for the Kitchen during the queried time range, so the query returns 3 points instead of 5. ```sql SELECT BOTTOM(temp, 5) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' AND room = 'Kitchen' ``` | time |bottom| |--------------------|------| |2022-01-01T08:00:00Z| 21 | |2022-01-01T09:00:00Z| 23 | |2022-01-01T10:00:00Z| 22.7 | [](#view-example-for-fnfield_key-tag_key-n) View example for `FN(field_key, tag_key, N)` The example below selects the top temperature from 3 unique values of the `room` tag. However, the `room` tag only has 2 unique values, so results only contain 2 values. ```sql SELECT TOP(temp, room, 3) FROM home ``` | time |top | room | |--------------------|----|-----------| |2022-01-01T18:00:00Z|23.3| Kitchen | |2022-01-01T18:00:00Z|22.8|Living Room| #### Related * [Aggregate data with InfluxQL](/influxdb3/enterprise/query-data/influxql/aggregate-select/) | time | bottom | | --- | --- | | time | bottom | | 2022-01-01T08:00:00Z | 21 | | 2022-01-01T08:00:00Z | 21.1 | | 2022-01-01T09:00:00Z | 21.4 | | time | bottom | room | | --- | --- | --- | | time | bottom | room | | 2022-01-01T08:00:00Z | 21 | Kitchen | | 2022-01-01T08:00:00Z | 21.1 | Living Room | | time | bottom | room | | --- | --- | --- | | time | bottom | room | | 2022-01-01T08:00:00Z | 21 | Kitchen | | 2022-01-01T08:00:00Z | 21.1 | Living Room | | 2022-01-01T09:00:00Z | 21.4 | Living Room | | time | bottom | room | | --- | --- | --- | | time | bottom | room | | 2022-01-01T08:00:00Z | 21 | Kitchen | | 2022-01-01T08:00:00Z | 21.1 | Living Room | | 2022-01-01T10:00:00Z | 21.8 | Living Room | | 2022-01-01T11:00:00Z | 22.4 | Kitchen | | 2022-01-01T12:00:00Z | 22.2 | Living Room | | 2022-01-01T12:00:00Z | 22.5 | Kitchen | | time | first | | --- | --- | | time | first | | 2022-01-01T08:00:00Z | 21.1 | | time | first_co | first_hum | first_temp | | --- | --- | --- | --- | | time | first_co | first_hum | first_temp | | 1970-01-01T00:00:00Z | 0 | 35.9 | 21.1 | | time | first_hum | first_temp | | --- | --- | --- | | time | first_hum | first_temp | | 1970-01-01T00:00:00Z | 35.9 | 21.1 | | time | first | | --- | --- | | time | first | | 2022-01-01T06:00:00Z | 21 | | 2022-01-01T12:00:00Z | 22.5 | | 2022-01-01T18:00:00Z | 23.3 | | time | last | | --- | --- | | time | last | | 2022-01-01T20:00:00Z | 22.7 | | time | last_co | last_hum | last_temp | | --- | --- | --- | --- | | time | last_co | last_hum | last_temp | | 1970-01-01T00:00:00Z | 26 | 36.5 | 22.7 | | time | last_hum | last_temp | | --- | --- | --- | | time | last_hum | last_temp | | 1970-01-01T00:00:00Z | 36.5 | 22.7 | | time | last | | --- | --- | | time | last | | 2022-01-01T06:00:00Z | 22.4 | | 2022-01-01T12:00:00Z | 22.7 | | 2022-01-01T18:00:00Z | 22.7 | | time | max | | --- | --- | | time | max | | 2022-01-01T20:00:00Z | 26 | | time | max_co | max_hum | max_temp | | --- | --- | --- | --- | | time | max_co | max_hum | max_temp | | 1970-01-01T00:00:00Z | 26 | 36.9 | 23.3 | | time | max_hum | max_temp | | --- | --- | --- | | time | max_hum | max_temp | | 1970-01-01T00:00:00Z | 36.9 | 23.3 | | time | max | | --- | --- | | time | max | | 2022-01-01T06:00:00Z | 23 | | 2022-01-01T12:00:00Z | 22.8 | | 2022-01-01T18:00:00Z | 23.3 | | time | min | | --- | --- | | time | min | | 2022-01-01T08:00:00Z | 21 | | time | min_co | min_hum | min_temp | | --- | --- | --- | --- | | time | min_co | min_hum | min_temp | | 1970-01-01T00:00:00Z | 0 | 35.9 | 21 | | time | min_hum | min_temp | | --- | --- | --- | | time | min_hum | min_temp | | 1970-01-01T00:00:00Z | 35.9 | 21 | | time | min | | --- | --- | | time | min | | 2022-01-01T06:00:00Z | 21 | | 2022-01-01T12:00:00Z | 22.4 | | 2022-01-01T18:00:00Z | 22.7 | | time | percentile | | --- | --- | | time | percentile | | 2022-01-01T11:00:00Z | 22.4 | | time | percentile_co | percentile_hum | percentile_temp | | --- | --- | --- | --- | | time | percentile_co | percentile_hum | percentile_temp | | 1970-01-01T00:00:00Z | 1 | 36 | 22.4 | | time | percentile_hum | percentile_temp | | --- | --- | --- | | time | percentile_hum | percentile_temp | | 1970-01-01T00:00:00Z | 36 | 22.4 | | time | percentile | | --- | --- | | time | percentile | | 2022-01-01T06:00:00Z | 22.4 | | 2022-01-01T12:00:00Z | 22.7 | | 2022-01-01T18:00:00Z | 23.1 | | time | top | | --- | --- | | time | top | | 2022-01-01T09:00:00Z | 23 | | 2022-01-01T18:00:00Z | 23.3 | | 2022-01-01T19:00:00Z | 23.1 | | time | top | room | | --- | --- | --- | | time | top | room | | 2022-01-01T18:00:00Z | 23.3 | Kitchen | | 2022-01-01T18:00:00Z | 22.8 | Living Room | | time | top | room | | --- | --- | --- | | time | top | room | | 2022-01-01T09:00:00Z | 23 | Kitchen | | 2022-01-01T18:00:00Z | 23.3 | Kitchen | | 2022-01-01T19:00:00Z | 23.1 | Kitchen | | time | top | room | | --- | --- | --- | | time | top | room | | 2022-01-01T09:00:00Z | 23 | Kitchen | | 2022-01-01T09:00:00Z | 21.4 | Living Room | | 2022-01-01T10:00:00Z | 22.7 | Kitchen | | 2022-01-01T11:00:00Z | 22.2 | Living Room | | 2022-01-01T12:00:00Z | 22.5 | Kitchen | | 2022-01-01T12:00:00Z | 22.2 | Living Room | | time | bottom | | --- | --- | | time | bottom | | 2022-01-01T08:00:00Z | 21 | | 2022-01-01T09:00:00Z | 23 | | 2022-01-01T10:00:00Z | 22.7 | | time | top | room | | --- | --- | --- | | time | top | room | | 2022-01-01T18:00:00Z | 23.3 | Kitchen | | 2022-01-01T18:00:00Z | 22.8 | Living Room | --- # InfluxQL transformation functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/functions/transformations/ InfluxQL transformation functions modify and return values in each row of queried data. > [!Important] > #### Missing InfluxQL functions > > Some InfluxQL functions are in the process of being rearchitected to work with > the InfluxDB 3 storage engine. If a function you need is not here, check the[InfluxQL feature support page](/influxdb3/enterprise/reference/influxql/feature-support/#function-support)for more information. ## Notable behaviors of transformation functions #### Must use aggregate or selector functions when grouping by time Most transformation functions support `GROUP BY` clauses that group by tags, but do not directly support `GROUP BY` clauses that group by time. To use transformation functions with with a `GROUP BY time()` clause, apply an [aggregate](/influxdb3/enterprise/reference/influxql/functions/aggregates/)or [selector](/influxdb3/enterprise/reference/influxql/functions/selectors/)function to the **field\_expression** argument. The transformation operates on the result of the aggregate or selector operation. ## ABS() Returns the absolute value of the field value. ```sql ABS(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-abs-to-a-field) Apply `ABS()` to a field ```sql SELECT a, ABS(a) FROM numbers LIMIT 6 ``` name: numbers | time | a | abs | |--------------------|-------------------|-----------------| |2023-01-01T00:00:00Z| 0.33909108671076 |0.33909108671076 | |2023-01-01T00:01:00Z|\-0.774984088561186|0.774984088561186| |2023-01-01T00:02:00Z|\-0.921037167720451|0.921037167720451| |2023-01-01T00:03:00Z|\-0.73880754843378 |0.73880754843378 | |2023-01-01T00:04:00Z|\-0.905980032168252|0.905980032168252| |2023-01-01T00:05:00Z|\-0.891164752631417|0.891164752631417| [](#apply-abs-to-each-field) Apply `ABS()` to each field ```sql SELECT ABS(*) FROM numbers LIMIT 6 ``` name: numbers | time | abs\_a | abs\_b | |--------------------|-----------------|------------------| |2023-01-01T00:00:00Z|0.33909108671076 |0.163643058925645 | |2023-01-01T00:01:00Z|0.774984088561186|0.137034364053949 | |2023-01-01T00:02:00Z|0.921037167720451|0.482943221384294 | |2023-01-01T00:03:00Z|0.73880754843378 |0.0729732928756677| |2023-01-01T00:04:00Z|0.905980032168252| 1.77857552719844 | |2023-01-01T00:05:00Z|0.891164752631417|0.741147445214238 | [](#apply-abs-to-time-windows-grouped-by-time) Apply `ABS()` to time windows (grouped by time) ```sql SELECT ABS(MEAN(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | abs | |--------------------|--------------------| |2023-01-01T00:00:00Z| 0.4345725888930678 | |2023-01-01T00:10:00Z|0.12861008519618367 | |2023-01-01T00:20:00Z|0.030168160597251192| |2023-01-01T00:30:00Z|0.02928699660831855 | |2023-01-01T00:40:00Z|0.02211434600834538 | |2023-01-01T00:50:00Z|0.15530468657783394 | ## ACOS() Returns the arccosine (in radians) of the field value. Field values must be between -1 and 1. ```sql ACOS(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-acos-to-a-field) Apply `ACOS()` to a field ```sql SELECT a, ACOS(a) FROM numbers LIMIT 6 ``` name: numbers | time | a | acos | |--------------------|-------------------|------------------| |2023-01-01T00:00:00Z| 0.33909108671076 |1.2248457522250173| |2023-01-01T00:01:00Z|\-0.774984088561186| 2.4574862443115 | |2023-01-01T00:02:00Z|\-0.921037167720451|2.741531473732281 | |2023-01-01T00:03:00Z|\-0.73880754843378 |2.4020955294179256| |2023-01-01T00:04:00Z|\-0.905980032168252|2.7044854502651114| |2023-01-01T00:05:00Z|\-0.891164752631417| 2.6707024029338 | [](#apply-acos-to-each-field) Apply `ACOS()` to each field ```sql SELECT ACOS(*) FROM numbers LIMIT 6 ``` name: numbers | time | acos\_a | acos\_b | |--------------------|------------------|------------------| |2023-01-01T00:00:00Z|1.2248457522250173|1.7351786975993897| |2023-01-01T00:01:00Z| 2.4574862443115 |1.433329416131427 | |2023-01-01T00:02:00Z|2.741531473732281 |2.074809114132046 | |2023-01-01T00:03:00Z|2.4020955294179256|1.6438345403920092| |2023-01-01T00:04:00Z|2.7044854502651114| | |2023-01-01T00:05:00Z| 2.6707024029338 |0.7360183965088304| [](#apply-acos-to-time-windows-grouped-by-time) Apply `ACOS()` to time windows (grouped by time) ```sql SELECT ACOS(MEAN(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | acos | |--------------------|------------------| |2023-01-01T00:00:00Z|2.0203599837582877| |2023-01-01T00:10:00Z|1.441829029328407 | |2023-01-01T00:20:00Z|1.5406235882252437| |2023-01-01T00:30:00Z|1.5415051418561052| |2023-01-01T00:40:00Z|1.5486801779072885| |2023-01-01T00:50:00Z| 1.41486045205998 | ## ASIN() Returns the arcsine (in radians) of the field value. Field values must be between -1 and 1. ```sql ASIN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-asin-to-a-field) Apply `ASIN()` to a field ```sql SELECT a, ASIN(a) FROM numbers LIMIT 6 ``` name: numbers | time | a | asin | |--------------------|-------------------|--------------------| |2023-01-01T00:00:00Z| 0.33909108671076 |0.34595057456987915 | |2023-01-01T00:01:00Z|\-0.774984088561186|\-0.8866899175166036| |2023-01-01T00:02:00Z|\-0.921037167720451|\-1.1707351469373848| |2023-01-01T00:03:00Z|\-0.73880754843378 |\-0.8312992026230288| |2023-01-01T00:04:00Z|\-0.905980032168252|\-1.133689123470215 | |2023-01-01T00:05:00Z|\-0.891164752631417|\-1.0999060761389035| [](#apply-asin-to-each-field) Apply `ASIN()` to each field ```sql SELECT ASIN(*) FROM numbers LIMIT 6 ``` name: numbers | time | asin\_a | asin\_b | |--------------------|--------------------|---------------------| |2023-01-01T00:00:00Z|0.34595057456987915 |\-0.1643823708044932 | |2023-01-01T00:01:00Z|\-0.8866899175166036| 0.1374669106634696 | |2023-01-01T00:02:00Z|\-1.1707351469373848|\-0.5040127873371497 | |2023-01-01T00:03:00Z|\-0.8312992026230288|\-0.07303821359711259| |2023-01-01T00:04:00Z|\-1.133689123470215 | | |2023-01-01T00:05:00Z|\-1.0999060761389035| 0.8347779302860662 | [](#apply-asin-to-time-windows-grouped-by-time) Apply `ASIN()` to time windows (grouped by time) ```sql SELECT ASIN(MEAN(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | asin | |--------------------|---------------------| |2023-01-01T00:00:00Z|\-0.44956365696339134| |2023-01-01T00:10:00Z| 0.1289672974664895 | |2023-01-01T00:20:00Z|0.030172738569652847 | |2023-01-01T00:30:00Z|0.029291184938791334 | |2023-01-01T00:40:00Z|0.022116148887608062 | |2023-01-01T00:50:00Z| 0.15593587473491674 | ## ATAN() Returns the arctangent (in radians) of the field value. ```sql ATAN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-atan-to-a-field) Apply `ATAN()` to a field ```sql SELECT a, ATAN(a) FROM numbers LIMIT 6 ``` name: numbers | time | a | atan | |--------------------|-------------------|--------------------| |2023-01-01T00:00:00Z| 0.33909108671076 |0.32692355076199897 | |2023-01-01T00:01:00Z|\-0.774984088561186|\-0.659300127490126 | |2023-01-01T00:02:00Z|\-0.921037167720451|\-0.7443170183837121| |2023-01-01T00:03:00Z|\-0.73880754843378 |\-0.6362993731936669| |2023-01-01T00:04:00Z|\-0.905980032168252|\-0.7361091800814261| |2023-01-01T00:05:00Z|\-0.891164752631417|\-0.727912249468035 | [](#apply-atan-to-each-field) Apply `ATAN()` to each field ```sql SELECT ATAN(*) FROM numbers LIMIT 6 ``` name: numbers | time | atan\_a | atan\_b | |--------------------|--------------------|---------------------| |2023-01-01T00:00:00Z|0.32692355076199897 |\-0.1622053541422186 | |2023-01-01T00:01:00Z|\-0.659300127490126 | 0.13618613793696105 | |2023-01-01T00:02:00Z|\-0.7443170183837121|\-0.4499093121666581 | |2023-01-01T00:03:00Z|\-0.6362993731936669|\-0.07284417510130452| |2023-01-01T00:04:00Z|\-0.7361091800814261| 1.0585985450688151 | |2023-01-01T00:05:00Z|\-0.727912249468035 | 0.6378113578294793 | [](#apply-atan-to-time-windows-grouped-by-time) Apply `ATAN()` to time windows (grouped by time) ```sql SELECT ATAN(MEAN(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | atan | |--------------------|--------------------| |2023-01-01T00:00:00Z|\-0.4099506966510045| |2023-01-01T00:10:00Z| 0.1279079463727065 | |2023-01-01T00:20:00Z|0.030159013397288013| |2023-01-01T00:30:00Z|0.02927862748761639 | |2023-01-01T00:40:00Z|0.022110742100818606| |2023-01-01T00:50:00Z|0.15407382461141705 | ## ATAN2() Returns the the arctangent of `y/x` in radians. ```sql ATAN2(expression_y, expression_x) ``` #### Arguments * **expression\_y**: Expression to identify the `y` numeric value or one or more fields to operate on. Can be a number literal, [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. * **expression\_x**: Expression to identify the `x` numeric value or one or more fields to operate on. Can be a number literal, [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-atan2-to-a-field-divided-by-another-field) Apply `ATAN2()` to a field divided by another field ```sql SELECT ATAN2(a, b) FROM numbers LIMIT 6 ``` name: numbers | time | atan2 | |--------------------|---------------------| |2023-01-01T00:00:00Z| 2.0204217911794937 | |2023-01-01T00:01:00Z| \-1.395783190047229 | |2023-01-01T00:02:00Z| \-2.053731408859953 | |2023-01-01T00:03:00Z| \-1.669248713922653 | |2023-01-01T00:04:00Z|\-0.47112754043763505| |2023-01-01T00:05:00Z|\-0.8770454978291377 | [](#apply-atan2-to-each-field-divided-by-a-numeric-value) Apply `ATAN2()` to each field divided by a numeric value ```sql SELECT ATAN2(*, 2) FROM numbers LIMIT 6 ``` name: numbers | time | atan2\_a | atan2\_b | |--------------------|---------------------|----------------------| |2023-01-01T00:00:00Z| 0.16794843225523703 | \-0.0816396675119722 | |2023-01-01T00:01:00Z|\-0.36967737169970566| 0.06841026268126137 | |2023-01-01T00:02:00Z|\-0.4315666721698651 | \-0.2369359777533473 | |2023-01-01T00:03:00Z|\-0.35385538623378937|\-0.036470468100670846| |2023-01-01T00:04:00Z|\-0.4253376417906667 | 0.7268651162204586 | |2023-01-01T00:05:00Z|\-0.41917415992493756| 0.35488446257957357 | [](#apply-atan2-to-time-windows-grouped-by-time) Apply `ATAN2()` to time windows (grouped by time) ```sql SELECT ATAN2(MEAN(a), MEAN(b)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | atan2 | |--------------------|-------------------| |2023-01-01T00:00:00Z|\-1.278967897411707| |2023-01-01T00:10:00Z|2.3520553840586773 | |2023-01-01T00:20:00Z| 2.226497789888965 | |2023-01-01T00:30:00Z|3.0977773783018656 | |2023-01-01T00:40:00Z|2.9285769547942677 | |2023-01-01T00:50:00Z|0.9505419744107901 | ## CEIL() Returns the subsequent value rounded up to the nearest integer. ```sql CEIL(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-ceil-to-a-field) Apply `CEIL()` to a field ```sql SELECT b, CEIL(b) FROM numbers LIMIT 6 ``` name: numbers | time | b |ceil| |--------------------|--------------------|----| |2023-01-01T00:00:00Z|\-0.163643058925645 |\-0 | |2023-01-01T00:01:00Z| 0.137034364053949 | 1 | |2023-01-01T00:02:00Z|\-0.482943221384294 |\-0 | |2023-01-01T00:03:00Z|\-0.0729732928756677|\-0 | |2023-01-01T00:04:00Z| 1.77857552719844 | 2 | |2023-01-01T00:05:00Z| 0.741147445214238 | 1 | [](#apply-ceil-to-each-field) Apply `CEIL()` to each field ```sql SELECT CEIL(*) FROM numbers LIMIT 6 ``` name: numbers | time |ceil\_a|ceil\_b| |--------------------|-------|-------| |2023-01-01T00:00:00Z| 1 | \-0 | |2023-01-01T00:01:00Z| \-0 | 1 | |2023-01-01T00:02:00Z| \-0 | \-0 | |2023-01-01T00:03:00Z| \-0 | \-0 | |2023-01-01T00:04:00Z| \-0 | 2 | |2023-01-01T00:05:00Z| \-0 | 1 | [](#apply-ceil-to-time-windows-grouped-by-time) Apply `CEIL()` to time windows (grouped by time) ```sql SELECT CEIL(MEAN(b)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time |ceil| |--------------------|----| |2023-01-01T00:00:00Z| 1 | |2023-01-01T00:10:00Z|\-0 | |2023-01-01T00:20:00Z|\-0 | |2023-01-01T00:30:00Z|\-0 | |2023-01-01T00:40:00Z|\-0 | |2023-01-01T00:50:00Z| 1 | ## COS() Returns the cosine of the field value. ```sql COS(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-cos-to-a-field) Apply `COS()` to a field ```sql SELECT b, COS(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | cos | |--------------------|--------------------|---------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 | 0.9866403278718959 | |2023-01-01T00:01:00Z| 0.137034364053949 | 0.9906254752128878 | |2023-01-01T00:02:00Z|\-0.482943221384294 | 0.8856319645801471 | |2023-01-01T00:03:00Z|\-0.0729732928756677| 0.9973386305831397 | |2023-01-01T00:04:00Z| 1.77857552719844 |\-0.20628737691395405| |2023-01-01T00:05:00Z| 0.741147445214238 | 0.7376943643170851 | [](#apply-cos-to-each-field) Apply `COS()` to each field ```sql SELECT COS(*) FROM numbers LIMIT 6 ``` name: numbers | time | cos\_a | cos\_b | |--------------------|------------------|---------------------| |2023-01-01T00:00:00Z|0.9430573869206459| 0.9866403278718959 | |2023-01-01T00:01:00Z|0.7144321674550146| 0.9906254752128878 | |2023-01-01T00:02:00Z|0.6049946586273094| 0.8856319645801471 | |2023-01-01T00:03:00Z|0.7392720891861374| 0.9973386305831397 | |2023-01-01T00:04:00Z|0.616914561474936 |\-0.20628737691395405| |2023-01-01T00:05:00Z|0.6285065034701617| 0.7376943643170851 | [](#apply-cos-to-time-windows-grouped-by-time) Apply `COS()` to time windows (grouped by time) ```sql SELECT COS(MEAN(b)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | cos | |--------------------|------------------| |2023-01-01T00:00:00Z|0.9914907269510592| |2023-01-01T00:10:00Z|0.9918765457796455| |2023-01-01T00:20:00Z|0.9997307399250498| |2023-01-01T00:30:00Z|0.7850670342365872| |2023-01-01T00:40:00Z|0.9947779847618986| |2023-01-01T00:50:00Z|0.9938532355205111| ## CUMULATIVE\_SUM() Returns the running total of subsequent [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql CUMULATIVE_SUM(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-cumulative_sum-to-a-field) Apply `CUMULATIVE_SUM()` to a field ```sql SELECT CUMULATIVE_SUM(b) FROM numbers LIMIT 6 ``` name: numbers | time | cumulative\_sum | |--------------------|---------------------| |2023-01-01T00:00:00Z| \-0.163643058925645 | |2023-01-01T00:01:00Z|\-0.02660869487169601| |2023-01-01T00:02:00Z|\-0.5095519162559901 | |2023-01-01T00:03:00Z|\-0.5825252091316577 | |2023-01-01T00:04:00Z| 1.1960503180667823 | |2023-01-01T00:05:00Z| 1.9371977632810204 | [](#apply-cumulative_sum-to-each-field) Apply `CUMULATIVE_SUM()` to each field ```sql SELECT CUMULATIVE_SUM(*) FROM numbers LIMIT 6 ``` name: numbers | time | cumulative\_sum\_a | cumulative\_sum\_b | |--------------------|---------------------|---------------------| |2023-01-01T00:00:00Z| 0.33909108671076 | \-0.163643058925645 | |2023-01-01T00:01:00Z|\-0.43589300185042595|\-0.02660869487169601| |2023-01-01T00:02:00Z|\-1.3569301695708769 |\-0.5095519162559901 | |2023-01-01T00:03:00Z| \-2.095737718004657 |\-0.5825252091316577 | |2023-01-01T00:04:00Z| \-3.001717750172909 | 1.1960503180667823 | |2023-01-01T00:05:00Z| \-3.892882502804326 | 1.9371977632810204 | [](#apply-cumulative_sum-to-field-keys-that-match-a-regular-expression) Apply `CUMULATIVE_SUM()` to field keys that match a regular expression ```sql SELECT CUMULATIVE_SUM(/[ab]/) FROM numbers LIMIT 6 ``` name: numbers | time | cumulative\_sum\_a | cumulative\_sum\_b | |--------------------|---------------------|---------------------| |2023-01-01T00:00:00Z| 0.33909108671076 | \-0.163643058925645 | |2023-01-01T00:01:00Z|\-0.43589300185042595|\-0.02660869487169601| |2023-01-01T00:02:00Z|\-1.3569301695708769 |\-0.5095519162559901 | |2023-01-01T00:03:00Z| \-2.095737718004657 |\-0.5825252091316577 | |2023-01-01T00:04:00Z| \-3.001717750172909 | 1.1960503180667823 | |2023-01-01T00:05:00Z| \-3.892882502804326 | 1.9371977632810204 | [](#apply-cumulative_sum-to-time-windows-grouped-by-time) Apply `CUMULATIVE_SUM()` to time windows (grouped by time) ```sql SELECT CUMULATIVE_SUM(SUM(b)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | cumulative\_sum | |--------------------|---------------------| |2023-01-01T00:00:00Z| 1.3054783385851743 | |2023-01-01T00:10:00Z|0.029980276948385454 | |2023-01-01T00:20:00Z|\-0.20208529969578404| |2023-01-01T00:30:00Z| \-6.882005145666267 | |2023-01-01T00:40:00Z| \-7.904410787756402 | |2023-01-01T00:50:00Z| \-6.795080184131271 | ## DERIVATIVE() Returns the rate of change between subsequent [field values](/influxdb3/enterprise/reference/glossary/#field-value)per `unit`. ```sql SELECT DERIVATIVE(field_expression[, unit]) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports numeric field types. * **unit**: Unit of time to use to calculate the rate of change. Supports [duration literals](/influxdb3/enterprise/reference/influxql/#durations).*Default is `1s` (per second)*. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Related functions #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-derivative-to-a-field-to-calculate-the-per-second-change) Apply `DERIVATIVE()` to a field to calculate the per second change ```sql SELECT DERIVATIVE(b) FROM numbers LIMIT 6 ``` name: numbers | time | derivative | |--------------------|----------------------| |2023-01-01T00:01:00Z| 0.005011290382993233 | |2023-01-01T00:02:00Z|\-0.01033295975730405 | |2023-01-01T00:03:00Z| 0.006832832141810439 | |2023-01-01T00:04:00Z| 0.03085914700123513 | |2023-01-01T00:05:00Z|\-0.017290468033070033| |2023-01-01T00:06:00Z|\-0.007557890705063634| [](#apply-derivative-to-a-field-to-calculate-the-per-5-minute-change) Apply `DERIVATIVE()` to a field to calculate the per 5 minute change ```sql SELECT DERIVATIVE(b, 5m) FROM numbers LIMIT 6 ``` name: numbers | time | derivative | |--------------------|--------------------| |2023-01-01T00:01:00Z| 1.5033871148979698 | |2023-01-01T00:02:00Z|\-3.0998879271912148| |2023-01-01T00:03:00Z| 2.0498496425431316 | |2023-01-01T00:04:00Z| 9.257744100370537 | |2023-01-01T00:05:00Z|\-5.187140409921009 | |2023-01-01T00:06:00Z| \-2.26736721151909 | [](#apply-derivative-to-each-field) Apply `DERIVATIVE()` to each field ```sql SELECT DERIVATIVE(*) FROM numbers LIMIT 6 ``` name: numbers | time | derivative\_a | derivative\_b | |--------------------|-----------------------|----------------------| |2023-01-01T00:01:00Z|\-0.018567919587865765 | 0.005011290382993233 | |2023-01-01T00:02:00Z|\-0.0024342179859877505|\-0.01033295975730405 | |2023-01-01T00:03:00Z| 0.0030371603214445152 | 0.006832832141810439 | |2023-01-01T00:04:00Z|\-0.0027862080622411984| 0.03085914700123513 | |2023-01-01T00:05:00Z|0.00024692132561391543 |\-0.017290468033070033| |2023-01-01T00:06:00Z| 0.016704951104985283 |\-0.007557890705063634| [](#apply-derivative-to-field-keys-that-match-a-regular-expression) Apply `DERIVATIVE()` to field keys that match a regular expression ```sql SELECT DERIVATIVE(/[ab]/) FROM numbers LIMIT 6 ``` name: numbers | time | derivative\_a | derivative\_b | |--------------------|-----------------------|----------------------| |2023-01-01T00:01:00Z|\-0.018567919587865765 | 0.005011290382993233 | |2023-01-01T00:02:00Z|\-0.0024342179859877505|\-0.01033295975730405 | |2023-01-01T00:03:00Z| 0.0030371603214445152 | 0.006832832141810439 | |2023-01-01T00:04:00Z|\-0.0027862080622411984| 0.03085914700123513 | |2023-01-01T00:05:00Z|0.00024692132561391543 |\-0.017290468033070033| |2023-01-01T00:06:00Z| 0.016704951104985283 |\-0.007557890705063634| [](#apply-derivative-to-time-windows-grouped-by-time) Apply `DERIVATIVE()` to time windows (grouped by time) ```sql SELECT DERIVATIVE(MEAN(b), 1m) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | derivative | |--------------------|----------------------| |2023-01-01T00:10:00Z|\-0.025809764002219633| |2023-01-01T00:20:00Z| 0.010434324849926194 | |2023-01-01T00:30:00Z|\-0.06447854269326314 | |2023-01-01T00:40:00Z| 0.05657514203880348 | |2023-01-01T00:50:00Z| 0.021317362457152655 | ## DIFFERENCE() Returns the result of subtraction between subsequent [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql DIFFERENCE(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Related functions #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-difference-to-a-field) Apply `DIFFERENCE()` to a field ```sql SELECT DIFFERENCE(b) FROM numbers LIMIT 6 ``` name: numbers | time | difference | |--------------------|---------------------| |2023-01-01T00:01:00Z| 0.300677422979594 | |2023-01-01T00:02:00Z| \-0.619977585438243 | |2023-01-01T00:03:00Z| 0.40996992850862635 | |2023-01-01T00:04:00Z| 1.8515488200741077 | |2023-01-01T00:05:00Z|\-1.0374280819842019 | |2023-01-01T00:06:00Z|\-0.45347344230381803| [](#apply-difference-to-each-field) Apply `DIFFERENCE()` to each field ```sql SELECT DIFFERENCE(*) FROM numbers LIMIT 6 ``` name: numbers | time | difference\_a | difference\_b | |--------------------|---------------------|---------------------| |2023-01-01T00:01:00Z| \-1.114075175271946 | 0.300677422979594 | |2023-01-01T00:02:00Z|\-0.14605307915926502| \-0.619977585438243 | |2023-01-01T00:03:00Z| 0.18222961928667092 | 0.40996992850862635 | |2023-01-01T00:04:00Z|\-0.1671724837344719 | 1.8515488200741077 | |2023-01-01T00:05:00Z|0.014815279536834924 |\-1.0374280819842019 | |2023-01-01T00:06:00Z| 1.002297066299117 |\-0.45347344230381803| [](#apply-difference-to-field-keys-that-match-a-regular-expression) Apply `DIFFERENCE()` to field keys that match a regular expression ```sql SELECT DIFFERENCE(/[ab]/) FROM numbers LIMIT 6 ``` name: numbers | time | difference\_a | difference\_b | |--------------------|---------------------|---------------------| |2023-01-01T00:01:00Z| \-1.114075175271946 | 0.300677422979594 | |2023-01-01T00:02:00Z|\-0.14605307915926502| \-0.619977585438243 | |2023-01-01T00:03:00Z| 0.18222961928667092 | 0.40996992850862635 | |2023-01-01T00:04:00Z|\-0.1671724837344719 | 1.8515488200741077 | |2023-01-01T00:05:00Z|0.014815279536834924 |\-1.0374280819842019 | |2023-01-01T00:06:00Z| 1.002297066299117 |\-0.45347344230381803| [](#apply-difference-to-time-windows-grouped-by-time) Apply `DIFFERENCE()` to time windows (grouped by time) ```sql SELECT DIFFERENCE(MEAN(b)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | difference | |--------------------|--------------------| |2023-01-01T00:10:00Z|\-0.2580976400221963| |2023-01-01T00:20:00Z|0.10434324849926194 | |2023-01-01T00:30:00Z|\-0.6447854269326314| |2023-01-01T00:40:00Z| 0.5657514203880348 | |2023-01-01T00:50:00Z|0.21317362457152655 | ## ELAPSED() Returns the difference between subsequent [field value’s](/influxdb3/enterprise/reference/glossary/#field-value)timestamps in a specified `unit` of time. ```sql ELAPSED(field_expression[, unit ]) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports all field types. * **unit**: Unit of time to return the elapsed time in. Supports [duration literals](/influxdb3/enterprise/reference/influxql/#durations).*Default is `1ns` (nanoseconds)*. #### Notable behaviors * If the `unit` is greater than the elapsed time between points, `ELAPSED()`returns `0`. * `ELAPSED()` supports the `GROUP BY time()` clause but the query results aren’t very useful. An `ELAPSED()` query with a nested function and a `GROUP BY time()` clause returns the interval specified in the `GROUP BY time()` clause. #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-elapsed-to-a-field-and-return-elapsed-time-in-nanoseconds) Apply `ELAPSED()` to a field and return elapsed time in nanoseconds ```sql SELECT ELAPSED(b) FROM numbers LIMIT 6 ``` name: numbers | time | elapsed | |--------------------|-----------| |2023-01-01T00:01:00Z|60000000000| |2023-01-01T00:02:00Z|60000000000| |2023-01-01T00:03:00Z|60000000000| |2023-01-01T00:04:00Z|60000000000| |2023-01-01T00:05:00Z|60000000000| |2023-01-01T00:06:00Z|60000000000| [](#apply-elapsed-to-a-field-and-return-elapsed-time-in-seconds) Apply `ELAPSED()` to a field and return elapsed time in seconds ```sql SELECT ELAPSED(b, 1s) FROM numbers LIMIT 6 ``` name: numbers | time |elapsed| |--------------------|-------| |2023-01-01T00:01:00Z| 60 | |2023-01-01T00:02:00Z| 60 | |2023-01-01T00:03:00Z| 60 | |2023-01-01T00:04:00Z| 60 | |2023-01-01T00:05:00Z| 60 | |2023-01-01T00:06:00Z| 60 | [](#apply-elapsed-to-each-field) Apply `ELAPSED()` to each field ```sql SELECT ELAPSED(*) FROM numbers LIMIT 6 ``` name: numbers | time |elapsed\_a |elapsed\_b | |--------------------|-----------|-----------| |2023-01-01T00:01:00Z|60000000000|60000000000| |2023-01-01T00:02:00Z|60000000000|60000000000| |2023-01-01T00:03:00Z|60000000000|60000000000| |2023-01-01T00:04:00Z|60000000000|60000000000| |2023-01-01T00:05:00Z|60000000000|60000000000| |2023-01-01T00:06:00Z|60000000000|60000000000| [](#apply-elapsed-to-field-keys-that-match-a-regular-expression) Apply `ELAPSED()` to field keys that match a regular expression ```sql SELECT ELAPSED(/[ab]/, 1s) FROM numbers LIMIT 6 ``` name: numbers | time |elapsed\_a|elapsed\_b| |--------------------|----------|----------| |2023-01-01T00:01:00Z| 60 | 60 | |2023-01-01T00:02:00Z| 60 | 60 | |2023-01-01T00:03:00Z| 60 | 60 | |2023-01-01T00:04:00Z| 60 | 60 | |2023-01-01T00:05:00Z| 60 | 60 | |2023-01-01T00:06:00Z| 60 | 60 | ## EXP() Returns the exponential of the field value. ```sql EXP(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-exp-to-a-field) Apply `EXP()` to a field ```sql SELECT a, EXP(a) FROM numbers LIMIT 6 ``` name: numbers | time | a | exp | |--------------------|-------------------|-------------------| |2023-01-01T00:00:00Z| 0.33909108671076 |1.4036711951820788 | |2023-01-01T00:01:00Z|\-0.774984088561186| 0.460711111517308 | |2023-01-01T00:02:00Z|\-0.921037167720451|0.39810592427186076| |2023-01-01T00:03:00Z|\-0.73880754843378 |0.4776831901055915 | |2023-01-01T00:04:00Z|\-0.905980032168252|0.40414561525252984| |2023-01-01T00:05:00Z|\-0.891164752631417|0.4101777188333968 | [](#apply-exp-to-each-field) Apply `EXP()` to each field ```sql SELECT EXP(*) FROM numbers LIMIT 6 ``` name: numbers | time | exp\_a | exp\_b | |--------------------|-------------------|------------------| |2023-01-01T00:00:00Z|1.4036711951820788 |0.8490450268435884| |2023-01-01T00:01:00Z| 0.460711111517308 | 1.14686755886191 | |2023-01-01T00:02:00Z|0.39810592427186076|0.6169648527893578| |2023-01-01T00:03:00Z|0.4776831901055915 |0.929625657322271 | |2023-01-01T00:04:00Z|0.40414561525252984|5.921415512753404 | |2023-01-01T00:05:00Z|0.4101777188333968 | 2.09834186598405 | [](#apply-exp-to-time-windows-grouped-by-time) Apply `EXP()` to time windows (grouped by time) ```sql SELECT EXP(MEAN(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | exp | |--------------------|------------------| |2023-01-01T00:00:00Z|0.6475413743155294| |2023-01-01T00:10:00Z|1.137246608416461 | |2023-01-01T00:20:00Z|1.030627830373793 | |2023-01-01T00:30:00Z|1.029720078241656 | |2023-01-01T00:40:00Z|1.0223606806499268| |2023-01-01T00:50:00Z|1.1680137850180072| ## FLOOR() Returns the subsequent value rounded down to the nearest integer. ```sql FLOOR(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-floor-to-a-field) Apply `FLOOR()` to a field ```sql SELECT b, FLOOR(b) FROM numbers LIMIT 6 ``` name: numbers | time | b |floor| |--------------------|--------------------|-----| |2023-01-01T00:00:00Z|\-0.163643058925645 | \-1 | |2023-01-01T00:01:00Z| 0.137034364053949 | 0 | |2023-01-01T00:02:00Z|\-0.482943221384294 | \-1 | |2023-01-01T00:03:00Z|\-0.0729732928756677| \-1 | |2023-01-01T00:04:00Z| 1.77857552719844 | 1 | |2023-01-01T00:05:00Z| 0.741147445214238 | 0 | [](#apply-floor-to-each-field) Apply `FLOOR()` to each field ```sql SELECT FLOOR(*) FROM numbers LIMIT 6 ``` name: numbers | time |floor\_a|floor\_b| |--------------------|--------|--------| |2023-01-01T00:00:00Z| 0 | \-1 | |2023-01-01T00:01:00Z| \-1 | 0 | |2023-01-01T00:02:00Z| \-1 | \-1 | |2023-01-01T00:03:00Z| \-1 | \-1 | |2023-01-01T00:04:00Z| \-1 | 1 | |2023-01-01T00:05:00Z| \-1 | 0 | [](#apply-floor-to-time-windows-grouped-by-time) Apply `FLOOR()` to time windows (grouped by time) ```sql SELECT FLOOR(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time |floor| |--------------------|-----| |2023-01-01T00:00:00Z| \-5 | |2023-01-01T00:10:00Z| 1 | |2023-01-01T00:20:00Z| 0 | |2023-01-01T00:30:00Z| 0 | |2023-01-01T00:40:00Z| 0 | |2023-01-01T00:50:00Z| 1 | ## LN() Returns the natural logarithm of the field value. Field values must be greater than or equal to 0. ```sql LN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-ln-to-a-field) Apply `LN()` to a field ```sql SELECT b, LN(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | ln | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 | | |2023-01-01T00:01:00Z| 0.137034364053949 | \-1.98752355209665 | |2023-01-01T00:02:00Z|\-0.482943221384294 | | |2023-01-01T00:03:00Z|\-0.0729732928756677| | |2023-01-01T00:04:00Z| 1.77857552719844 | 0.5758127783016702 | |2023-01-01T00:05:00Z| 0.741147445214238 |\-0.2995556920844895| [](#apply-ln-to-each-field) Apply `LN()` to each field ```sql SELECT LN(*) FROM numbers LIMIT 6 ``` name: numbers | time | ln\_a | ln\_b | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z|\-1.0814865153308908| | |2023-01-01T00:01:00Z| | \-1.98752355209665 | |2023-01-01T00:02:00Z| | | |2023-01-01T00:03:00Z| | | |2023-01-01T00:04:00Z| | 0.5758127783016702 | |2023-01-01T00:05:00Z| |\-0.2995556920844895| [](#apply-ln-to-time-windows-grouped-by-time) Apply `LN()` to time windows (grouped by time) ```sql SELECT LN(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | ln | |--------------------|--------------------| |2023-01-01T00:00:00Z| | |2023-01-01T00:10:00Z|0.25161504572793725 | |2023-01-01T00:20:00Z|\-1.1983831026157092| |2023-01-01T00:30:00Z|\-1.2280265702380913| |2023-01-01T00:40:00Z|\-1.5089436474159283| |2023-01-01T00:50:00Z| 0.4402187212890264 | ## LOG() Returns the logarithm of the field value with base `b`. Field values must be greater than or equal to 0. ```sql LOG(field_expression, b) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. * **b**: Logarithm base to use in the operation. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-log-to-a-field-with-a-base-of-3) Apply `LOG()` to a field with a base of 3 ```sql SELECT b, LOG(b, 3) FROM numbers LIMIT 6 ``` name: numbers | time | b | log | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 | | |2023-01-01T00:01:00Z| 0.137034364053949 |\-1.8091219009630797| |2023-01-01T00:02:00Z|\-0.482943221384294 | | |2023-01-01T00:03:00Z|\-0.0729732928756677| | |2023-01-01T00:04:00Z| 1.77857552719844 | 0.5241273780031629 | |2023-01-01T00:05:00Z| 0.741147445214238 |\-0.2726673414946528| [](#apply-log-to-each-field-with-a-base-of-5) Apply `LOG()` to each field with a base of 5 ```sql SELECT LOG(*, 5) FROM numbers LIMIT 6 ``` name: numbers | time | log\_a | log\_b | |--------------------|--------------------|---------------------| |2023-01-01T00:00:00Z|\-0.6719653532302217| | |2023-01-01T00:01:00Z| |\-1.2349178161776593 | |2023-01-01T00:02:00Z| | | |2023-01-01T00:03:00Z| | | |2023-01-01T00:04:00Z| | 0.3577725949246566 | |2023-01-01T00:05:00Z| |\-0.18612441633827553| [](#apply-log-to-time-windows-grouped-by-time) Apply `LOG()` to time windows (grouped by time) ```sql SELECT LOG(SUM(a), 10) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | log | |--------------------|--------------------| |2023-01-01T00:00:00Z| | |2023-01-01T00:10:00Z|0.10927502592347751 | |2023-01-01T00:20:00Z|\-0.5204511686721008| |2023-01-01T00:30:00Z|\-0.5333251630849791| |2023-01-01T00:40:00Z|\-0.6553258995757036| |2023-01-01T00:50:00Z| 0.1911845614863297 | ## LOG2() Returns the logarithm of the field value to the base 2. Field values must be greater than or equal to 0. ```sql LOG2(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-log2-to-a-field) Apply `LOG2()` to a field ```sql SELECT b, LOG2(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | log2 | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 | | |2023-01-01T00:01:00Z| 0.137034364053949 |\-2.8673903722598544| |2023-01-01T00:02:00Z|\-0.482943221384294 | | |2023-01-01T00:03:00Z|\-0.0729732928756677| | |2023-01-01T00:04:00Z| 1.77857552719844 | 0.8307222397363156 | |2023-01-01T00:05:00Z| 0.741147445214238 |\-0.4321675114403543| [](#apply-log2-to-each-field) Apply `LOG2()` to each field ```sql SELECT LOG2(*) FROM numbers LIMIT 6 ``` name: numbers | time | log2\_a | log2\_b | |--------------------|-------------------|--------------------| |2023-01-01T00:00:00Z|\-1.560255232456162| | |2023-01-01T00:01:00Z| |\-2.8673903722598544| |2023-01-01T00:02:00Z| | | |2023-01-01T00:03:00Z| | | |2023-01-01T00:04:00Z| | 0.8307222397363156 | |2023-01-01T00:05:00Z| |\-0.4321675114403543| [](#apply-log2-to-time-windows-grouped-by-time) Apply `LOG2()` to time windows (grouped by time) ```sql SELECT LOG2(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | log2 | |--------------------|--------------------| |2023-01-01T00:00:00Z| | |2023-01-01T00:10:00Z|0.36300377868474476 | |2023-01-01T00:20:00Z|\-1.7289013592288134| |2023-01-01T00:30:00Z|\-1.7716678429623767| |2023-01-01T00:40:00Z|\-2.1769455171078644| |2023-01-01T00:50:00Z| 0.6351013661101591 | ## LOG10() Returns the logarithm of the field value to the base 10. Field values must be greater than or equal to 0. ```sql LOG10(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-log10-to-a-field) Apply `LOG10()` to a field ```sql SELECT b, LOG10(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | log10 | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 | | |2023-01-01T00:01:00Z| 0.137034364053949 |\-0.8631705113283253| |2023-01-01T00:02:00Z|\-0.482943221384294 | | |2023-01-01T00:03:00Z|\-0.0729732928756677| | |2023-01-01T00:04:00Z| 1.77857552719844 |0.25007231222579585 | |2023-01-01T00:05:00Z| 0.741147445214238 |\-0.1300953840950034| [](#apply-log10-to-each-field) Apply `LOG10()` to each field ```sql SELECT LOG10(*) FROM numbers LIMIT 6 ``` name: numbers | time | log10\_a | log10\_b | |--------------------|---------------------|--------------------| |2023-01-01T00:00:00Z|\-0.46968362586098245| | |2023-01-01T00:01:00Z| |\-0.8631705113283253| |2023-01-01T00:02:00Z| | | |2023-01-01T00:03:00Z| | | |2023-01-01T00:04:00Z| |0.25007231222579585 | |2023-01-01T00:05:00Z| |\-0.1300953840950034| [](#apply-log10-to-time-windows-grouped-by-time) Apply `LOG10()` to time windows (grouped by time) ```sql SELECT LOG10(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | log10 | |--------------------|--------------------| |2023-01-01T00:00:00Z| | |2023-01-01T00:10:00Z|0.10927502592347751 | |2023-01-01T00:20:00Z|\-0.520451168672101 | |2023-01-01T00:30:00Z|\-0.5333251630849791| |2023-01-01T00:40:00Z|\-0.6553258995757036| |2023-01-01T00:50:00Z|0.19118456148632973 | ## MOVING\_AVERAGE() Returns the rolling average across a window of subsequent [field values](/influxdb3/enterprise/reference/glossary/#field-value). ```sql MOVING_AVERAGE(field_expression, N) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports all field types. * **N**: Number of field values to use when calculating the moving average. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-moving_average-to-a-field) Apply `MOVING_AVERAGE()` to a field ```sql SELECT MOVING_AVERAGE(a, 3) FROM numbers LIMIT 6 ``` name: numbers | time | moving\_average | |--------------------|--------------------| |2023-01-01T00:02:00Z|\-0.4523100565236256| |2023-01-01T00:03:00Z|\-0.8116096015718056| |2023-01-01T00:04:00Z|\-0.8552749161074944| |2023-01-01T00:05:00Z|\-0.8453174444111498| |2023-01-01T00:06:00Z|\-0.5620041570439896| |2023-01-01T00:07:00Z|\-0.3569778402485757| [](#apply-moving_average-to-each-field) Apply `MOVING_AVERAGE()` to each field ```sql SELECT MOVING_AVERAGE(*, 3) FROM numbers LIMIT 6 ``` name: numbers | time | moving\_average\_a | moving\_average\_b | |--------------------|--------------------|---------------------| |2023-01-01T00:02:00Z|\-0.4523100565236256|\-0.16985063875199669| |2023-01-01T00:03:00Z|\-0.8116096015718056|\-0.13962738340200423| |2023-01-01T00:04:00Z|\-0.8552749161074944| 0.40755300431282615 | |2023-01-01T00:05:00Z|\-0.8453174444111498| 0.815583226512337 | |2023-01-01T00:06:00Z|\-0.5620041570439896| 0.9357989917743662 | |2023-01-01T00:07:00Z|\-0.3569778402485757| 0.15985821845558748 | [](#apply-moving_average-to-field-keys-that-match-a-regular-expression) Apply `MOVING_AVERAGE()` to field keys that match a regular expression ```sql SELECT MOVING_AVERAGE(/[ab]/, 3) FROM numbers LIMIT 6 ``` name: numbers | time | moving\_average\_a | moving\_average\_b | |--------------------|--------------------|---------------------| |2023-01-01T00:02:00Z|\-0.4523100565236256|\-0.16985063875199669| |2023-01-01T00:03:00Z|\-0.8116096015718056|\-0.13962738340200423| |2023-01-01T00:04:00Z|\-0.8552749161074944| 0.40755300431282615 | |2023-01-01T00:05:00Z|\-0.8453174444111498| 0.815583226512337 | |2023-01-01T00:06:00Z|\-0.5620041570439896| 0.9357989917743662 | |2023-01-01T00:07:00Z|\-0.3569778402485757| 0.15985821845558748 | [](#apply-moving_average-to-time-windows-grouped-by-time) Apply `MOVING_AVERAGE()` to time windows (grouped by time) ```sql SELECT MOVING_AVERAGE(SUM(a), 3) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | moving\_average | |--------------------|--------------------| |2023-01-01T00:20:00Z|\-0.9193144769987766| |2023-01-01T00:30:00Z| 0.626884141339178 | |2023-01-01T00:40:00Z|0.27189834404638374 | |2023-01-01T00:50:00Z| 0.6890200973149928 | ## NON\_NEGATIVE\_DERIVATIVE() Returns only non-negative rate of change between subsequent[field values](/influxdb3/enterprise/reference/glossary/#field-value). Negative rates of change return *null*. ```sql NON_NEGATIVE_DERIVATIVE(field_expression[, unit]) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports numeric field types. * **unit**: Unit of time to use to calculate the rate of change. Supports [duration literals](/influxdb3/enterprise/reference/influxql/#durations).*Default is `1s` (per second)*. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Related functions #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-non_negative_derivative-to-a-field-to-calculate-the-per-second-change) Apply `NON_NEGATIVE_DERIVATIVE()` to a field to calculate the per second change ```sql SELECT NON_NEGATIVE_DERIVATIVE(b) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_derivative| |--------------------|-------------------------| |2023-01-01T00:01:00Z| 0.005011290382993233 | |2023-01-01T00:03:00Z| 0.006832832141810439 | |2023-01-01T00:04:00Z| 0.03085914700123513 | |2023-01-01T00:08:00Z| 0.0227877053636946 | |2023-01-01T00:10:00Z| 0.001676063810538834 | |2023-01-01T00:11:00Z| 0.014999637478226817 | [](#apply-non_negative_derivative-to-a-field-to-calculate-the-per-5-minute-change) Apply `NON_NEGATIVE_DERIVATIVE()` to a field to calculate the per 5 minute change ```sql SELECT NON_NEGATIVE_DERIVATIVE(b, 5m) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_derivative| |--------------------|-------------------------| |2023-01-01T00:01:00Z| 1.5033871148979698 | |2023-01-01T00:03:00Z| 2.0498496425431316 | |2023-01-01T00:04:00Z| 9.257744100370537 | |2023-01-01T00:08:00Z| 6.836311609108379 | |2023-01-01T00:10:00Z| 0.5028191431616502 | |2023-01-01T00:11:00Z| 4.499891243468045 | [](#apply-non_negative_derivative-to-each-field) Apply `NON_NEGATIVE_DERIVATIVE()` to each field ```sql SELECT NON_NEGATIVE_DERIVATIVE(*) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_derivative\_a|non\_negative\_derivative\_b| |--------------------|----------------------------|----------------------------| |2023-01-01T00:01:00Z| | 0.005011290382993233 | |2023-01-01T00:03:00Z| 0.0030371603214445152 | 0.006832832141810439 | |2023-01-01T00:04:00Z| | 0.03085914700123513 | |2023-01-01T00:05:00Z| 0.00024692132561391543 | | |2023-01-01T00:06:00Z| 0.016704951104985283 | | |2023-01-01T00:08:00Z| | 0.0227877053636946 | |2023-01-01T00:09:00Z| 0.018437240876186967 | | |2023-01-01T00:10:00Z| | 0.001676063810538834 | |2023-01-01T00:11:00Z| | 0.014999637478226817 | |2023-01-01T00:13:00Z| 0.006694752202850366 | | |2023-01-01T00:14:00Z| 0.011836797386191167 | | [](#apply-non_negative_derivative-to-field-keys-that-match-a-regular-expression) Apply `NON_NEGATIVE_DERIVATIVE()` to field keys that match a regular expression ```sql SELECT NON_NEGATIVE_DERIVATIVE(/[ab]/) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_derivative\_a|non\_negative\_derivative\_b| |--------------------|----------------------------|----------------------------| |2023-01-01T00:01:00Z| | 0.005011290382993233 | |2023-01-01T00:03:00Z| 0.0030371603214445152 | 0.006832832141810439 | |2023-01-01T00:04:00Z| | 0.03085914700123513 | |2023-01-01T00:05:00Z| 0.00024692132561391543 | | |2023-01-01T00:06:00Z| 0.016704951104985283 | | |2023-01-01T00:08:00Z| | 0.0227877053636946 | |2023-01-01T00:09:00Z| 0.018437240876186967 | | |2023-01-01T00:10:00Z| | 0.001676063810538834 | |2023-01-01T00:11:00Z| | 0.014999637478226817 | |2023-01-01T00:13:00Z| 0.006694752202850366 | | |2023-01-01T00:14:00Z| 0.011836797386191167 | | [](#apply-non_negative_derivative-to-time-windows-grouped-by-time) Apply `NON_NEGATIVE_DERIVATIVE()` to time windows (grouped by time) ```sql SELECT NON_NEGATIVE_DERIVATIVE(MEAN(b), 1m) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time |non\_negative\_derivative| |--------------------|-------------------------| |2023-01-01T00:20:00Z| 0.010434324849926194 | |2023-01-01T00:40:00Z| 0.05657514203880348 | |2023-01-01T00:50:00Z| 0.021317362457152655 | ## NON\_NEGATIVE\_DIFFERENCE() Returns only non-negative result of subtraction between subsequent[field values](/influxdb3/enterprise/reference/glossary/#field-value). Negative differences return *null*. ```sql NON_NEGATIVE_DIFFERENCE(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, regular expression, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Related functions #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-non_negative_difference-to-a-field) Apply `NON_NEGATIVE_DIFFERENCE()` to a field ```sql SELECT NON_NEGATIVE_DIFFERENCE(b) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_difference| |--------------------|-------------------------| |2023-01-01T00:01:00Z| 0.300677422979594 | |2023-01-01T00:03:00Z| 0.40996992850862635 | |2023-01-01T00:04:00Z| 1.8515488200741077 | |2023-01-01T00:08:00Z| 1.367262321821676 | |2023-01-01T00:10:00Z| 0.10056382863233004 | |2023-01-01T00:11:00Z| 0.899978248693609 | [](#apply-non_negative_difference-to-each-field) Apply `NON_NEGATIVE_DIFFERENCE()` to each field ```sql SELECT NON_NEGATIVE_DIFFERENCE(*) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_difference\_a|non\_negative\_difference\_b| |--------------------|----------------------------|----------------------------| |2023-01-01T00:01:00Z| | 0.300677422979594 | |2023-01-01T00:03:00Z| 0.18222961928667092 | 0.40996992850862635 | |2023-01-01T00:04:00Z| | 1.8515488200741077 | |2023-01-01T00:05:00Z| 0.014815279536834924 | | |2023-01-01T00:06:00Z| 1.002297066299117 | | |2023-01-01T00:08:00Z| | 1.367262321821676 | |2023-01-01T00:09:00Z| 1.106234452571218 | | |2023-01-01T00:10:00Z| | 0.10056382863233004 | |2023-01-01T00:11:00Z| | 0.899978248693609 | |2023-01-01T00:13:00Z| 0.401685132171022 | | |2023-01-01T00:14:00Z| 0.71020784317147 | | [](#apply-non_negative_difference-to-field-keys-that-match-a-regular-expression) Apply `NON_NEGATIVE_DIFFERENCE()` to field keys that match a regular expression ```sql SELECT NON_NEGATIVE_DIFFERENCE(/[ab]/) FROM numbers LIMIT 6 ``` name: numbers | time |non\_negative\_difference\_a|non\_negative\_difference\_b| |--------------------|----------------------------|----------------------------| |2023-01-01T00:01:00Z| | 0.300677422979594 | |2023-01-01T00:03:00Z| 0.18222961928667092 | 0.40996992850862635 | |2023-01-01T00:04:00Z| | 1.8515488200741077 | |2023-01-01T00:05:00Z| 0.014815279536834924 | | |2023-01-01T00:06:00Z| 1.002297066299117 | | |2023-01-01T00:08:00Z| | 1.367262321821676 | |2023-01-01T00:09:00Z| 1.106234452571218 | | |2023-01-01T00:10:00Z| | 0.10056382863233004 | |2023-01-01T00:11:00Z| | 0.899978248693609 | |2023-01-01T00:13:00Z| 0.401685132171022 | | |2023-01-01T00:14:00Z| 0.71020784317147 | | [](#apply-non_negative_difference-to-time-windows-grouped-by-time) Apply `NON_NEGATIVE_DIFFERENCE()` to time windows (grouped by time) ```sql SELECT NON_NEGATIVE_DIFFERENCE(MEAN(b)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time |non\_negative\_difference| |--------------------|-------------------------| |2023-01-01T00:20:00Z| 0.10434324849926194 | |2023-01-01T00:40:00Z| 0.5657514203880348 | |2023-01-01T00:50:00Z| 0.21317362457152655 | ## POW() Returns the field value to the power of `x`. ```sql POW(field_expression, x) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. * **x**: Power to raise to. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-pow-to-a-field-with-a-power-of-3) Apply `POW()` to a field with a power of 3 ```sql SELECT b, POW(b, 3) FROM numbers LIMIT 6 ``` name: numbers | time | b | pow | |--------------------|--------------------|-----------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 |\-0.004382205777325515 | |2023-01-01T00:01:00Z| 0.137034364053949 | 0.002573288422171338 | |2023-01-01T00:02:00Z|\-0.482943221384294 | \-0.1126388541916811 | |2023-01-01T00:03:00Z|\-0.0729732928756677|\-0.0003885901893904874| |2023-01-01T00:04:00Z| 1.77857552719844 | 5.626222933751733 | |2023-01-01T00:05:00Z| 0.741147445214238 | 0.4071119474284653 | [](#apply-pow-to-each-field-with-a-power-of-5) Apply `POW()` to each field with a power of 5 ```sql SELECT POW(*, 5) FROM numbers LIMIT 6 ``` name: numbers | time | pow\_a | pow\_b | |--------------------|---------------------|-------------------------| |2023-01-01T00:00:00Z|0.004483135555212479 |\-0.00011735131084020357 | |2023-01-01T00:01:00Z|\-0.2795528536239978 | 0.000048322282876973225 | |2023-01-01T00:02:00Z|\-0.6628050073932118 | \-0.026271227986693114 | |2023-01-01T00:03:00Z|\-0.22011853819169455|\-0.000002069282189962477| |2023-01-01T00:04:00Z|\-0.6103699296012646 | 17.797604890097084 | |2023-01-01T00:05:00Z|\-0.5620694808926487 | 0.22362640363833164 | [](#apply-pow-to-time-windows-grouped-by-time) Apply `POW()` to time windows (grouped by time) ```sql SELECT POW(SUM(a), 10) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | pow | |--------------------|------------------------| |2023-01-01T00:00:00Z| 2402278.159218532 | |2023-01-01T00:10:00Z| 12.380844221267186 | |2023-01-01T00:20:00Z|0.000006244365466732681 | |2023-01-01T00:30:00Z|0.0000046424621235691315| |2023-01-01T00:40:00Z| 2.7973126174031977e-7 | |2023-01-01T00:50:00Z| 81.6292140233699 | ## ROUND() Returns a field value rounded to the nearest integer. ```sql ROUND(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-round-to-a-field) Apply `ROUND()` to a field ```sql SELECT b, ROUND(b) FROM numbers LIMIT 6 ``` name: numbers | time | b |round| |--------------------|--------------------|-----| |2023-01-01T00:00:00Z|\-0.163643058925645 | \-0 | |2023-01-01T00:01:00Z| 0.137034364053949 | 0 | |2023-01-01T00:02:00Z|\-0.482943221384294 | \-0 | |2023-01-01T00:03:00Z|\-0.0729732928756677| \-0 | |2023-01-01T00:04:00Z| 1.77857552719844 | 2 | |2023-01-01T00:05:00Z| 0.741147445214238 | 1 | [](#apply-round-to-each-field) Apply `ROUND()` to each field ```sql SELECT ROUND(*) FROM numbers LIMIT 6 ``` name: numbers | time |round\_a|round\_b| |--------------------|--------|--------| |2023-01-01T00:00:00Z| 0 | \-0 | |2023-01-01T00:01:00Z| \-1 | 0 | |2023-01-01T00:02:00Z| \-1 | \-0 | |2023-01-01T00:03:00Z| \-1 | \-0 | |2023-01-01T00:04:00Z| \-1 | 2 | |2023-01-01T00:05:00Z| \-1 | 1 | [](#apply-round-to-time-windows-grouped-by-time) Apply `ROUND()` to time windows (grouped by time) ```sql SELECT ROUND(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time |round| |--------------------|-----| |2023-01-01T00:00:00Z| \-4 | |2023-01-01T00:10:00Z| 1 | |2023-01-01T00:20:00Z| 0 | |2023-01-01T00:30:00Z| 0 | |2023-01-01T00:40:00Z| 0 | |2023-01-01T00:50:00Z| 2 | ## SIN() Returns the sine of a field value. ```sql SIN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-sin-to-a-field) Apply `SIN()` to a field ```sql SELECT b, SIN(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | sin | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 |\-0.1629136686003898| |2023-01-01T00:01:00Z| 0.137034364053949 |0.13660588515594851 | |2023-01-01T00:02:00Z|\-0.482943221384294 |\-0.4643877941052164| |2023-01-01T00:03:00Z|\-0.0729732928756677|\-0.0729085450859347| |2023-01-01T00:04:00Z| 1.77857552719844 | 0.9784914502058565 | |2023-01-01T00:05:00Z| 0.741147445214238 | 0.6751348197618099 | [](#apply-sin-to-each-field) Apply `SIN()` to each field ```sql SELECT SIN(*) FROM numbers LIMIT 6 ``` name: numbers | time | sin\_a | sin\_b | |--------------------|--------------------|--------------------| |2023-01-01T00:00:00Z| 0.3326300722640741 |\-0.1629136686003898| |2023-01-01T00:01:00Z|\-0.6997047077914582|0.13660588515594851 | |2023-01-01T00:02:00Z|\-0.7962295291135749|\-0.4643877941052164| |2023-01-01T00:03:00Z|\-0.673406844448706 |\-0.0729085450859347| |2023-01-01T00:04:00Z|\-0.7870301289278495| 0.9784914502058565 | |2023-01-01T00:05:00Z|\-0.7778043295686337| 0.6751348197618099 | [](#apply-sin-to-time-windows-grouped-by-time) Apply `SIN()` to time windows (grouped by time) ```sql SELECT SIN(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | sin | |--------------------|-------------------| |2023-01-01T00:00:00Z| 0.933528830283535 | |2023-01-01T00:10:00Z|0.9597472276784815 | |2023-01-01T00:20:00Z|0.29712628761434723| |2023-01-01T00:30:00Z|0.2887011711003489 | |2023-01-01T00:40:00Z|0.21934537994884437| |2023-01-01T00:50:00Z|0.9998424824522808 | ## SQRT() Returns the square root of a field value. Field values must be greater than or equal to 0. Negative field values return null. ```sql SQRT(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-sqrt-to-a-field) Apply `SQRT()` to a field ```sql SELECT b, SQRT(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | sqrt | |--------------------|--------------------|------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 | | |2023-01-01T00:01:00Z| 0.137034364053949 |0.370181528515334 | |2023-01-01T00:02:00Z|\-0.482943221384294 | | |2023-01-01T00:03:00Z|\-0.0729732928756677| | |2023-01-01T00:04:00Z| 1.77857552719844 |1.3336324558132349| |2023-01-01T00:05:00Z| 0.741147445214238 |0.860899207349059 | [](#apply-sqrt-to-each-field) Apply `SQRT()` to each field ```sql SELECT SQRT(*) FROM numbers LIMIT 6 ``` name: numbers | time | sqrt\_a | sqrt\_b | |--------------------|------------------|------------------| |2023-01-01T00:00:00Z|0.5823152811928947| | |2023-01-01T00:01:00Z| |0.370181528515334 | |2023-01-01T00:02:00Z| | | |2023-01-01T00:03:00Z| | | |2023-01-01T00:04:00Z| |1.3336324558132349| |2023-01-01T00:05:00Z| |0.860899207349059 | [](#apply-sqrt-to-time-windows-grouped-by-time) Apply `SQRT()` to time windows (grouped by time) ```sql SELECT SQRT(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | sqrt | |--------------------|------------------| |2023-01-01T00:00:00Z| | |2023-01-01T00:10:00Z|1.134063865909604 | |2023-01-01T00:20:00Z|0.5492555015405052| |2023-01-01T00:30:00Z|0.5411746169982342| |2023-01-01T00:40:00Z|0.4702589287652642| |2023-01-01T00:50:00Z|1.2462130097934059| ## TAN() Returns the tangent of the field value. ```sql TAN(field_expression) ``` #### Arguments * **field\_expression**: Expression to identify one or more fields to operate on. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, or wildcard (`*`). Supports numeric field types. #### Notable behaviors * [Must use aggregate or selector functions when grouping by time](#must-use-aggregate-or-selector-functions-when-grouping-by-time). #### Examples The following examples use the[Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data). [](#apply-tan-to-a-field) Apply `TAN()` to a field ```sql SELECT b, TAN(b) FROM numbers LIMIT 6 ``` name: numbers | time | b | tan | |--------------------|--------------------|---------------------| |2023-01-01T00:00:00Z|\-0.163643058925645 |\-0.16511961248511045| |2023-01-01T00:01:00Z| 0.137034364053949 | 0.13789861917955581 | |2023-01-01T00:02:00Z|\-0.482943221384294 |\-0.5243575352718546 | |2023-01-01T00:03:00Z|\-0.0729732928756677|\-0.07310309943905952| |2023-01-01T00:04:00Z| 1.77857552719844 | \-4.743341375725582 | |2023-01-01T00:05:00Z| 0.741147445214238 | 0.9151958486043346 | [](#apply-tan-to-each-field) Apply `TAN()` to each field ```sql SELECT TAN(*) FROM numbers LIMIT 6 ``` name: numbers | time | tan\_a | tan\_b | |--------------------|--------------------|---------------------| |2023-01-01T00:00:00Z| 0.3527145610408791 |\-0.16511961248511045| |2023-01-01T00:01:00Z|\-0.9793857830953787| 0.13789861917955581 | |2023-01-01T00:02:00Z|\-1.3160934857179802|\-0.5243575352718546 | |2023-01-01T00:03:00Z|\-0.9109052733075013|\-0.07310309943905952| |2023-01-01T00:04:00Z|\-1.2757522322802637| \-4.743341375725582 | |2023-01-01T00:05:00Z|\-1.2375438046768912| 0.9151958486043346 | [](#apply-tan-to-time-windows-grouped-by-time) Apply `TAN()` to time windows (grouped by time) ```sql SELECT TAN(SUM(a)) FROM numbers WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-01-01T01:00:00Z' GROUP BY time(10m) ``` name: numbers | time | tan | |--------------------|-------------------| |2023-01-01T00:00:00Z|\-2.603968631156288| |2023-01-01T00:10:00Z|3.4171098358131733 | |2023-01-01T00:20:00Z|0.31117972731464494| |2023-01-01T00:30:00Z|0.30154101138968664| |2023-01-01T00:40:00Z|0.22482036866737865| |2023-01-01T00:50:00Z| 56.3338223288096 | | time | a | abs | | --- | --- | --- | | time | a | abs | | 2023-01-01T00:00:00Z | 0.33909108671076 | 0.33909108671076 | | 2023-01-01T00:01:00Z | -0.774984088561186 | 0.774984088561186 | | 2023-01-01T00:02:00Z | -0.921037167720451 | 0.921037167720451 | | 2023-01-01T00:03:00Z | -0.73880754843378 | 0.73880754843378 | | 2023-01-01T00:04:00Z | -0.905980032168252 | 0.905980032168252 | | 2023-01-01T00:05:00Z | -0.891164752631417 | 0.891164752631417 | | time | abs_a | abs_b | | --- | --- | --- | | time | abs_a | abs_b | | 2023-01-01T00:00:00Z | 0.33909108671076 | 0.163643058925645 | | 2023-01-01T00:01:00Z | 0.774984088561186 | 0.137034364053949 | | 2023-01-01T00:02:00Z | 0.921037167720451 | 0.482943221384294 | | 2023-01-01T00:03:00Z | 0.73880754843378 | 0.0729732928756677 | | 2023-01-01T00:04:00Z | 0.905980032168252 | 1.77857552719844 | | 2023-01-01T00:05:00Z | 0.891164752631417 | 0.741147445214238 | | time | abs | | --- | --- | | time | abs | | 2023-01-01T00:00:00Z | 0.4345725888930678 | | 2023-01-01T00:10:00Z | 0.12861008519618367 | | 2023-01-01T00:20:00Z | 0.030168160597251192 | | 2023-01-01T00:30:00Z | 0.02928699660831855 | | 2023-01-01T00:40:00Z | 0.02211434600834538 | | 2023-01-01T00:50:00Z | 0.15530468657783394 | | time | a | acos | | --- | --- | --- | | time | a | acos | | 2023-01-01T00:00:00Z | 0.33909108671076 | 1.2248457522250173 | | 2023-01-01T00:01:00Z | -0.774984088561186 | 2.4574862443115 | | 2023-01-01T00:02:00Z | -0.921037167720451 | 2.741531473732281 | | 2023-01-01T00:03:00Z | -0.73880754843378 | 2.4020955294179256 | | 2023-01-01T00:04:00Z | -0.905980032168252 | 2.7044854502651114 | | 2023-01-01T00:05:00Z | -0.891164752631417 | 2.6707024029338 | | time | acos_a | acos_b | | --- | --- | --- | | time | acos_a | acos_b | | 2023-01-01T00:00:00Z | 1.2248457522250173 | 1.7351786975993897 | | 2023-01-01T00:01:00Z | 2.4574862443115 | 1.433329416131427 | | 2023-01-01T00:02:00Z | 2.741531473732281 | 2.074809114132046 | | 2023-01-01T00:03:00Z | 2.4020955294179256 | 1.6438345403920092 | | 2023-01-01T00:04:00Z | 2.7044854502651114 | | | 2023-01-01T00:05:00Z | 2.6707024029338 | 0.7360183965088304 | | time | acos | | --- | --- | | time | acos | | 2023-01-01T00:00:00Z | 2.0203599837582877 | | 2023-01-01T00:10:00Z | 1.441829029328407 | | 2023-01-01T00:20:00Z | 1.5406235882252437 | | 2023-01-01T00:30:00Z | 1.5415051418561052 | | 2023-01-01T00:40:00Z | 1.5486801779072885 | | 2023-01-01T00:50:00Z | 1.41486045205998 | | time | a | asin | | --- | --- | --- | | time | a | asin | | 2023-01-01T00:00:00Z | 0.33909108671076 | 0.34595057456987915 | | 2023-01-01T00:01:00Z | -0.774984088561186 | -0.8866899175166036 | | 2023-01-01T00:02:00Z | -0.921037167720451 | -1.1707351469373848 | | 2023-01-01T00:03:00Z | -0.73880754843378 | -0.8312992026230288 | | 2023-01-01T00:04:00Z | -0.905980032168252 | -1.133689123470215 | | 2023-01-01T00:05:00Z | -0.891164752631417 | -1.0999060761389035 | | time | asin_a | asin_b | | --- | --- | --- | | time | asin_a | asin_b | | 2023-01-01T00:00:00Z | 0.34595057456987915 | -0.1643823708044932 | | 2023-01-01T00:01:00Z | -0.8866899175166036 | 0.1374669106634696 | | 2023-01-01T00:02:00Z | -1.1707351469373848 | -0.5040127873371497 | | 2023-01-01T00:03:00Z | -0.8312992026230288 | -0.07303821359711259 | | 2023-01-01T00:04:00Z | -1.133689123470215 | | | 2023-01-01T00:05:00Z | -1.0999060761389035 | 0.8347779302860662 | | time | asin | | --- | --- | | time | asin | | 2023-01-01T00:00:00Z | -0.44956365696339134 | | 2023-01-01T00:10:00Z | 0.1289672974664895 | | 2023-01-01T00:20:00Z | 0.030172738569652847 | | 2023-01-01T00:30:00Z | 0.029291184938791334 | | 2023-01-01T00:40:00Z | 0.022116148887608062 | | 2023-01-01T00:50:00Z | 0.15593587473491674 | | time | a | atan | | --- | --- | --- | | time | a | atan | | 2023-01-01T00:00:00Z | 0.33909108671076 | 0.32692355076199897 | | 2023-01-01T00:01:00Z | -0.774984088561186 | -0.659300127490126 | | 2023-01-01T00:02:00Z | -0.921037167720451 | -0.7443170183837121 | | 2023-01-01T00:03:00Z | -0.73880754843378 | -0.6362993731936669 | | 2023-01-01T00:04:00Z | -0.905980032168252 | -0.7361091800814261 | | 2023-01-01T00:05:00Z | -0.891164752631417 | -0.727912249468035 | | time | atan_a | atan_b | | --- | --- | --- | | time | atan_a | atan_b | | 2023-01-01T00:00:00Z | 0.32692355076199897 | -0.1622053541422186 | | 2023-01-01T00:01:00Z | -0.659300127490126 | 0.13618613793696105 | | 2023-01-01T00:02:00Z | -0.7443170183837121 | -0.4499093121666581 | | 2023-01-01T00:03:00Z | -0.6362993731936669 | -0.07284417510130452 | | 2023-01-01T00:04:00Z | -0.7361091800814261 | 1.0585985450688151 | | 2023-01-01T00:05:00Z | -0.727912249468035 | 0.6378113578294793 | | time | atan | | --- | --- | | time | atan | | 2023-01-01T00:00:00Z | -0.4099506966510045 | | 2023-01-01T00:10:00Z | 0.1279079463727065 | | 2023-01-01T00:20:00Z | 0.030159013397288013 | | 2023-01-01T00:30:00Z | 0.02927862748761639 | | 2023-01-01T00:40:00Z | 0.022110742100818606 | | 2023-01-01T00:50:00Z | 0.15407382461141705 | | time | atan2 | | --- | --- | | time | atan2 | | 2023-01-01T00:00:00Z | 2.0204217911794937 | | 2023-01-01T00:01:00Z | -1.395783190047229 | | 2023-01-01T00:02:00Z | -2.053731408859953 | | 2023-01-01T00:03:00Z | -1.669248713922653 | | 2023-01-01T00:04:00Z | -0.47112754043763505 | | 2023-01-01T00:05:00Z | -0.8770454978291377 | | time | atan2_a | atan2_b | | --- | --- | --- | | time | atan2_a | atan2_b | | 2023-01-01T00:00:00Z | 0.16794843225523703 | -0.0816396675119722 | | 2023-01-01T00:01:00Z | -0.36967737169970566 | 0.06841026268126137 | | 2023-01-01T00:02:00Z | -0.4315666721698651 | -0.2369359777533473 | | 2023-01-01T00:03:00Z | -0.35385538623378937 | -0.036470468100670846 | | 2023-01-01T00:04:00Z | -0.4253376417906667 | 0.7268651162204586 | | 2023-01-01T00:05:00Z | -0.41917415992493756 | 0.35488446257957357 | | time | atan2 | | --- | --- | | time | atan2 | | 2023-01-01T00:00:00Z | -1.278967897411707 | | 2023-01-01T00:10:00Z | 2.3520553840586773 | | 2023-01-01T00:20:00Z | 2.226497789888965 | | 2023-01-01T00:30:00Z | 3.0977773783018656 | | 2023-01-01T00:40:00Z | 2.9285769547942677 | | 2023-01-01T00:50:00Z | 0.9505419744107901 | | time | b | ceil | | --- | --- | --- | | time | b | ceil | | 2023-01-01T00:00:00Z | -0.163643058925645 | -0 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 1 | | 2023-01-01T00:02:00Z | -0.482943221384294 | -0 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | -0 | | 2023-01-01T00:04:00Z | 1.77857552719844 | 2 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 1 | | time | ceil_a | ceil_b | | --- | --- | --- | | time | ceil_a | ceil_b | | 2023-01-01T00:00:00Z | 1 | -0 | | 2023-01-01T00:01:00Z | -0 | 1 | | 2023-01-01T00:02:00Z | -0 | -0 | | 2023-01-01T00:03:00Z | -0 | -0 | | 2023-01-01T00:04:00Z | -0 | 2 | | 2023-01-01T00:05:00Z | -0 | 1 | | time | ceil | | --- | --- | | time | ceil | | 2023-01-01T00:00:00Z | 1 | | 2023-01-01T00:10:00Z | -0 | | 2023-01-01T00:20:00Z | -0 | | 2023-01-01T00:30:00Z | -0 | | 2023-01-01T00:40:00Z | -0 | | 2023-01-01T00:50:00Z | 1 | | time | b | cos | | --- | --- | --- | | time | b | cos | | 2023-01-01T00:00:00Z | -0.163643058925645 | 0.9866403278718959 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0.9906254752128878 | | 2023-01-01T00:02:00Z | -0.482943221384294 | 0.8856319645801471 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | 0.9973386305831397 | | 2023-01-01T00:04:00Z | 1.77857552719844 | -0.20628737691395405 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 0.7376943643170851 | | time | cos_a | cos_b | | --- | --- | --- | | time | cos_a | cos_b | | 2023-01-01T00:00:00Z | 0.9430573869206459 | 0.9866403278718959 | | 2023-01-01T00:01:00Z | 0.7144321674550146 | 0.9906254752128878 | | 2023-01-01T00:02:00Z | 0.6049946586273094 | 0.8856319645801471 | | 2023-01-01T00:03:00Z | 0.7392720891861374 | 0.9973386305831397 | | 2023-01-01T00:04:00Z | 0.616914561474936 | -0.20628737691395405 | | 2023-01-01T00:05:00Z | 0.6285065034701617 | 0.7376943643170851 | | time | cos | | --- | --- | | time | cos | | 2023-01-01T00:00:00Z | 0.9914907269510592 | | 2023-01-01T00:10:00Z | 0.9918765457796455 | | 2023-01-01T00:20:00Z | 0.9997307399250498 | | 2023-01-01T00:30:00Z | 0.7850670342365872 | | 2023-01-01T00:40:00Z | 0.9947779847618986 | | 2023-01-01T00:50:00Z | 0.9938532355205111 | | time | cumulative_sum | | --- | --- | | time | cumulative_sum | | 2023-01-01T00:00:00Z | -0.163643058925645 | | 2023-01-01T00:01:00Z | -0.02660869487169601 | | 2023-01-01T00:02:00Z | -0.5095519162559901 | | 2023-01-01T00:03:00Z | -0.5825252091316577 | | 2023-01-01T00:04:00Z | 1.1960503180667823 | | 2023-01-01T00:05:00Z | 1.9371977632810204 | | time | cumulative_sum_a | cumulative_sum_b | | --- | --- | --- | | time | cumulative_sum_a | cumulative_sum_b | | 2023-01-01T00:00:00Z | 0.33909108671076 | -0.163643058925645 | | 2023-01-01T00:01:00Z | -0.43589300185042595 | -0.02660869487169601 | | 2023-01-01T00:02:00Z | -1.3569301695708769 | -0.5095519162559901 | | 2023-01-01T00:03:00Z | -2.095737718004657 | -0.5825252091316577 | | 2023-01-01T00:04:00Z | -3.001717750172909 | 1.1960503180667823 | | 2023-01-01T00:05:00Z | -3.892882502804326 | 1.9371977632810204 | | time | cumulative_sum_a | cumulative_sum_b | | --- | --- | --- | | time | cumulative_sum_a | cumulative_sum_b | | 2023-01-01T00:00:00Z | 0.33909108671076 | -0.163643058925645 | | 2023-01-01T00:01:00Z | -0.43589300185042595 | -0.02660869487169601 | | 2023-01-01T00:02:00Z | -1.3569301695708769 | -0.5095519162559901 | | 2023-01-01T00:03:00Z | -2.095737718004657 | -0.5825252091316577 | | 2023-01-01T00:04:00Z | -3.001717750172909 | 1.1960503180667823 | | 2023-01-01T00:05:00Z | -3.892882502804326 | 1.9371977632810204 | | time | cumulative_sum | | --- | --- | | time | cumulative_sum | | 2023-01-01T00:00:00Z | 1.3054783385851743 | | 2023-01-01T00:10:00Z | 0.029980276948385454 | | 2023-01-01T00:20:00Z | -0.20208529969578404 | | 2023-01-01T00:30:00Z | -6.882005145666267 | | 2023-01-01T00:40:00Z | -7.904410787756402 | | 2023-01-01T00:50:00Z | -6.795080184131271 | | time | derivative | | --- | --- | | time | derivative | | 2023-01-01T00:01:00Z | 0.005011290382993233 | | 2023-01-01T00:02:00Z | -0.01033295975730405 | | 2023-01-01T00:03:00Z | 0.006832832141810439 | | 2023-01-01T00:04:00Z | 0.03085914700123513 | | 2023-01-01T00:05:00Z | -0.017290468033070033 | | 2023-01-01T00:06:00Z | -0.007557890705063634 | | time | derivative | | --- | --- | | time | derivative | | 2023-01-01T00:01:00Z | 1.5033871148979698 | | 2023-01-01T00:02:00Z | -3.0998879271912148 | | 2023-01-01T00:03:00Z | 2.0498496425431316 | | 2023-01-01T00:04:00Z | 9.257744100370537 | | 2023-01-01T00:05:00Z | -5.187140409921009 | | 2023-01-01T00:06:00Z | -2.26736721151909 | | time | derivative_a | derivative_b | | --- | --- | --- | | time | derivative_a | derivative_b | | 2023-01-01T00:01:00Z | -0.018567919587865765 | 0.005011290382993233 | | 2023-01-01T00:02:00Z | -0.0024342179859877505 | -0.01033295975730405 | | 2023-01-01T00:03:00Z | 0.0030371603214445152 | 0.006832832141810439 | | 2023-01-01T00:04:00Z | -0.0027862080622411984 | 0.03085914700123513 | | 2023-01-01T00:05:00Z | 0.00024692132561391543 | -0.017290468033070033 | | 2023-01-01T00:06:00Z | 0.016704951104985283 | -0.007557890705063634 | | time | derivative_a | derivative_b | | --- | --- | --- | | time | derivative_a | derivative_b | | 2023-01-01T00:01:00Z | -0.018567919587865765 | 0.005011290382993233 | | 2023-01-01T00:02:00Z | -0.0024342179859877505 | -0.01033295975730405 | | 2023-01-01T00:03:00Z | 0.0030371603214445152 | 0.006832832141810439 | | 2023-01-01T00:04:00Z | -0.0027862080622411984 | 0.03085914700123513 | | 2023-01-01T00:05:00Z | 0.00024692132561391543 | -0.017290468033070033 | | 2023-01-01T00:06:00Z | 0.016704951104985283 | -0.007557890705063634 | | time | derivative | | --- | --- | | time | derivative | | 2023-01-01T00:10:00Z | -0.025809764002219633 | | 2023-01-01T00:20:00Z | 0.010434324849926194 | | 2023-01-01T00:30:00Z | -0.06447854269326314 | | 2023-01-01T00:40:00Z | 0.05657514203880348 | | 2023-01-01T00:50:00Z | 0.021317362457152655 | | time | difference | | --- | --- | | time | difference | | 2023-01-01T00:01:00Z | 0.300677422979594 | | 2023-01-01T00:02:00Z | -0.619977585438243 | | 2023-01-01T00:03:00Z | 0.40996992850862635 | | 2023-01-01T00:04:00Z | 1.8515488200741077 | | 2023-01-01T00:05:00Z | -1.0374280819842019 | | 2023-01-01T00:06:00Z | -0.45347344230381803 | | time | difference_a | difference_b | | --- | --- | --- | | time | difference_a | difference_b | | 2023-01-01T00:01:00Z | -1.114075175271946 | 0.300677422979594 | | 2023-01-01T00:02:00Z | -0.14605307915926502 | -0.619977585438243 | | 2023-01-01T00:03:00Z | 0.18222961928667092 | 0.40996992850862635 | | 2023-01-01T00:04:00Z | -0.1671724837344719 | 1.8515488200741077 | | 2023-01-01T00:05:00Z | 0.014815279536834924 | -1.0374280819842019 | | 2023-01-01T00:06:00Z | 1.002297066299117 | -0.45347344230381803 | | time | difference_a | difference_b | | --- | --- | --- | | time | difference_a | difference_b | | 2023-01-01T00:01:00Z | -1.114075175271946 | 0.300677422979594 | | 2023-01-01T00:02:00Z | -0.14605307915926502 | -0.619977585438243 | | 2023-01-01T00:03:00Z | 0.18222961928667092 | 0.40996992850862635 | | 2023-01-01T00:04:00Z | -0.1671724837344719 | 1.8515488200741077 | | 2023-01-01T00:05:00Z | 0.014815279536834924 | -1.0374280819842019 | | 2023-01-01T00:06:00Z | 1.002297066299117 | -0.45347344230381803 | | time | difference | | --- | --- | | time | difference | | 2023-01-01T00:10:00Z | -0.2580976400221963 | | 2023-01-01T00:20:00Z | 0.10434324849926194 | | 2023-01-01T00:30:00Z | -0.6447854269326314 | | 2023-01-01T00:40:00Z | 0.5657514203880348 | | 2023-01-01T00:50:00Z | 0.21317362457152655 | | time | elapsed | | --- | --- | | time | elapsed | | 2023-01-01T00:01:00Z | 60000000000 | | 2023-01-01T00:02:00Z | 60000000000 | | 2023-01-01T00:03:00Z | 60000000000 | | 2023-01-01T00:04:00Z | 60000000000 | | 2023-01-01T00:05:00Z | 60000000000 | | 2023-01-01T00:06:00Z | 60000000000 | | time | elapsed | | --- | --- | | time | elapsed | | 2023-01-01T00:01:00Z | 60 | | 2023-01-01T00:02:00Z | 60 | | 2023-01-01T00:03:00Z | 60 | | 2023-01-01T00:04:00Z | 60 | | 2023-01-01T00:05:00Z | 60 | | 2023-01-01T00:06:00Z | 60 | | time | elapsed_a | elapsed_b | | --- | --- | --- | | time | elapsed_a | elapsed_b | | 2023-01-01T00:01:00Z | 60000000000 | 60000000000 | | 2023-01-01T00:02:00Z | 60000000000 | 60000000000 | | 2023-01-01T00:03:00Z | 60000000000 | 60000000000 | | 2023-01-01T00:04:00Z | 60000000000 | 60000000000 | | 2023-01-01T00:05:00Z | 60000000000 | 60000000000 | | 2023-01-01T00:06:00Z | 60000000000 | 60000000000 | | time | elapsed_a | elapsed_b | | --- | --- | --- | | time | elapsed_a | elapsed_b | | 2023-01-01T00:01:00Z | 60 | 60 | | 2023-01-01T00:02:00Z | 60 | 60 | | 2023-01-01T00:03:00Z | 60 | 60 | | 2023-01-01T00:04:00Z | 60 | 60 | | 2023-01-01T00:05:00Z | 60 | 60 | | 2023-01-01T00:06:00Z | 60 | 60 | | time | a | exp | | --- | --- | --- | | time | a | exp | | 2023-01-01T00:00:00Z | 0.33909108671076 | 1.4036711951820788 | | 2023-01-01T00:01:00Z | -0.774984088561186 | 0.460711111517308 | | 2023-01-01T00:02:00Z | -0.921037167720451 | 0.39810592427186076 | | 2023-01-01T00:03:00Z | -0.73880754843378 | 0.4776831901055915 | | 2023-01-01T00:04:00Z | -0.905980032168252 | 0.40414561525252984 | | 2023-01-01T00:05:00Z | -0.891164752631417 | 0.4101777188333968 | | time | exp_a | exp_b | | --- | --- | --- | | time | exp_a | exp_b | | 2023-01-01T00:00:00Z | 1.4036711951820788 | 0.8490450268435884 | | 2023-01-01T00:01:00Z | 0.460711111517308 | 1.14686755886191 | | 2023-01-01T00:02:00Z | 0.39810592427186076 | 0.6169648527893578 | | 2023-01-01T00:03:00Z | 0.4776831901055915 | 0.929625657322271 | | 2023-01-01T00:04:00Z | 0.40414561525252984 | 5.921415512753404 | | 2023-01-01T00:05:00Z | 0.4101777188333968 | 2.09834186598405 | | time | exp | | --- | --- | | time | exp | | 2023-01-01T00:00:00Z | 0.6475413743155294 | | 2023-01-01T00:10:00Z | 1.137246608416461 | | 2023-01-01T00:20:00Z | 1.030627830373793 | | 2023-01-01T00:30:00Z | 1.029720078241656 | | 2023-01-01T00:40:00Z | 1.0223606806499268 | | 2023-01-01T00:50:00Z | 1.1680137850180072 | | time | b | floor | | --- | --- | --- | | time | b | floor | | 2023-01-01T00:00:00Z | -0.163643058925645 | -1 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0 | | 2023-01-01T00:02:00Z | -0.482943221384294 | -1 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | -1 | | 2023-01-01T00:04:00Z | 1.77857552719844 | 1 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 0 | | time | floor_a | floor_b | | --- | --- | --- | | time | floor_a | floor_b | | 2023-01-01T00:00:00Z | 0 | -1 | | 2023-01-01T00:01:00Z | -1 | 0 | | 2023-01-01T00:02:00Z | -1 | -1 | | 2023-01-01T00:03:00Z | -1 | -1 | | 2023-01-01T00:04:00Z | -1 | 1 | | 2023-01-01T00:05:00Z | -1 | 0 | | time | floor | | --- | --- | | time | floor | | 2023-01-01T00:00:00Z | -5 | | 2023-01-01T00:10:00Z | 1 | | 2023-01-01T00:20:00Z | 0 | | 2023-01-01T00:30:00Z | 0 | | 2023-01-01T00:40:00Z | 0 | | 2023-01-01T00:50:00Z | 1 | | time | b | ln | | --- | --- | --- | | time | b | ln | | 2023-01-01T00:00:00Z | -0.163643058925645 | | | 2023-01-01T00:01:00Z | 0.137034364053949 | -1.98752355209665 | | 2023-01-01T00:02:00Z | -0.482943221384294 | | | 2023-01-01T00:03:00Z | -0.0729732928756677 | | | 2023-01-01T00:04:00Z | 1.77857552719844 | 0.5758127783016702 | | 2023-01-01T00:05:00Z | 0.741147445214238 | -0.2995556920844895 | | time | ln_a | ln_b | | --- | --- | --- | | time | ln_a | ln_b | | 2023-01-01T00:00:00Z | -1.0814865153308908 | | | 2023-01-01T00:01:00Z | | -1.98752355209665 | | 2023-01-01T00:02:00Z | | | | 2023-01-01T00:03:00Z | | | | 2023-01-01T00:04:00Z | | 0.5758127783016702 | | 2023-01-01T00:05:00Z | | -0.2995556920844895 | | time | ln | | --- | --- | | time | ln | | 2023-01-01T00:00:00Z | | | 2023-01-01T00:10:00Z | 0.25161504572793725 | | 2023-01-01T00:20:00Z | -1.1983831026157092 | | 2023-01-01T00:30:00Z | -1.2280265702380913 | | 2023-01-01T00:40:00Z | -1.5089436474159283 | | 2023-01-01T00:50:00Z | 0.4402187212890264 | | time | b | log | | --- | --- | --- | | time | b | log | | 2023-01-01T00:00:00Z | -0.163643058925645 | | | 2023-01-01T00:01:00Z | 0.137034364053949 | -1.8091219009630797 | | 2023-01-01T00:02:00Z | -0.482943221384294 | | | 2023-01-01T00:03:00Z | -0.0729732928756677 | | | 2023-01-01T00:04:00Z | 1.77857552719844 | 0.5241273780031629 | | 2023-01-01T00:05:00Z | 0.741147445214238 | -0.2726673414946528 | | time | log_a | log_b | | --- | --- | --- | | time | log_a | log_b | | 2023-01-01T00:00:00Z | -0.6719653532302217 | | | 2023-01-01T00:01:00Z | | -1.2349178161776593 | | 2023-01-01T00:02:00Z | | | | 2023-01-01T00:03:00Z | | | | 2023-01-01T00:04:00Z | | 0.3577725949246566 | | 2023-01-01T00:05:00Z | | -0.18612441633827553 | | time | log | | --- | --- | | time | log | | 2023-01-01T00:00:00Z | | | 2023-01-01T00:10:00Z | 0.10927502592347751 | | 2023-01-01T00:20:00Z | -0.5204511686721008 | | 2023-01-01T00:30:00Z | -0.5333251630849791 | | 2023-01-01T00:40:00Z | -0.6553258995757036 | | 2023-01-01T00:50:00Z | 0.1911845614863297 | | time | b | log2 | | --- | --- | --- | | time | b | log2 | | 2023-01-01T00:00:00Z | -0.163643058925645 | | | 2023-01-01T00:01:00Z | 0.137034364053949 | -2.8673903722598544 | | 2023-01-01T00:02:00Z | -0.482943221384294 | | | 2023-01-01T00:03:00Z | -0.0729732928756677 | | | 2023-01-01T00:04:00Z | 1.77857552719844 | 0.8307222397363156 | | 2023-01-01T00:05:00Z | 0.741147445214238 | -0.4321675114403543 | | time | log2_a | log2_b | | --- | --- | --- | | time | log2_a | log2_b | | 2023-01-01T00:00:00Z | -1.560255232456162 | | | 2023-01-01T00:01:00Z | | -2.8673903722598544 | | 2023-01-01T00:02:00Z | | | | 2023-01-01T00:03:00Z | | | | 2023-01-01T00:04:00Z | | 0.8307222397363156 | | 2023-01-01T00:05:00Z | | -0.4321675114403543 | | time | log2 | | --- | --- | | time | log2 | | 2023-01-01T00:00:00Z | | | 2023-01-01T00:10:00Z | 0.36300377868474476 | | 2023-01-01T00:20:00Z | -1.7289013592288134 | | 2023-01-01T00:30:00Z | -1.7716678429623767 | | 2023-01-01T00:40:00Z | -2.1769455171078644 | | 2023-01-01T00:50:00Z | 0.6351013661101591 | | time | b | log10 | | --- | --- | --- | | time | b | log10 | | 2023-01-01T00:00:00Z | -0.163643058925645 | | | 2023-01-01T00:01:00Z | 0.137034364053949 | -0.8631705113283253 | | 2023-01-01T00:02:00Z | -0.482943221384294 | | | 2023-01-01T00:03:00Z | -0.0729732928756677 | | | 2023-01-01T00:04:00Z | 1.77857552719844 | 0.25007231222579585 | | 2023-01-01T00:05:00Z | 0.741147445214238 | -0.1300953840950034 | | time | log10_a | log10_b | | --- | --- | --- | | time | log10_a | log10_b | | 2023-01-01T00:00:00Z | -0.46968362586098245 | | | 2023-01-01T00:01:00Z | | -0.8631705113283253 | | 2023-01-01T00:02:00Z | | | | 2023-01-01T00:03:00Z | | | | 2023-01-01T00:04:00Z | | 0.25007231222579585 | | 2023-01-01T00:05:00Z | | -0.1300953840950034 | | time | log10 | | --- | --- | | time | log10 | | 2023-01-01T00:00:00Z | | | 2023-01-01T00:10:00Z | 0.10927502592347751 | | 2023-01-01T00:20:00Z | -0.520451168672101 | | 2023-01-01T00:30:00Z | -0.5333251630849791 | | 2023-01-01T00:40:00Z | -0.6553258995757036 | | 2023-01-01T00:50:00Z | 0.19118456148632973 | | time | moving_average | | --- | --- | | time | moving_average | | 2023-01-01T00:02:00Z | -0.4523100565236256 | | 2023-01-01T00:03:00Z | -0.8116096015718056 | | 2023-01-01T00:04:00Z | -0.8552749161074944 | | 2023-01-01T00:05:00Z | -0.8453174444111498 | | 2023-01-01T00:06:00Z | -0.5620041570439896 | | 2023-01-01T00:07:00Z | -0.3569778402485757 | | time | moving_average_a | moving_average_b | | --- | --- | --- | | time | moving_average_a | moving_average_b | | 2023-01-01T00:02:00Z | -0.4523100565236256 | -0.16985063875199669 | | 2023-01-01T00:03:00Z | -0.8116096015718056 | -0.13962738340200423 | | 2023-01-01T00:04:00Z | -0.8552749161074944 | 0.40755300431282615 | | 2023-01-01T00:05:00Z | -0.8453174444111498 | 0.815583226512337 | | 2023-01-01T00:06:00Z | -0.5620041570439896 | 0.9357989917743662 | | 2023-01-01T00:07:00Z | -0.3569778402485757 | 0.15985821845558748 | | time | moving_average_a | moving_average_b | | --- | --- | --- | | time | moving_average_a | moving_average_b | | 2023-01-01T00:02:00Z | -0.4523100565236256 | -0.16985063875199669 | | 2023-01-01T00:03:00Z | -0.8116096015718056 | -0.13962738340200423 | | 2023-01-01T00:04:00Z | -0.8552749161074944 | 0.40755300431282615 | | 2023-01-01T00:05:00Z | -0.8453174444111498 | 0.815583226512337 | | 2023-01-01T00:06:00Z | -0.5620041570439896 | 0.9357989917743662 | | 2023-01-01T00:07:00Z | -0.3569778402485757 | 0.15985821845558748 | | time | moving_average | | --- | --- | | time | moving_average | | 2023-01-01T00:20:00Z | -0.9193144769987766 | | 2023-01-01T00:30:00Z | 0.626884141339178 | | 2023-01-01T00:40:00Z | 0.27189834404638374 | | 2023-01-01T00:50:00Z | 0.6890200973149928 | | time | non_negative_derivative | | --- | --- | | time | non_negative_derivative | | 2023-01-01T00:01:00Z | 0.005011290382993233 | | 2023-01-01T00:03:00Z | 0.006832832141810439 | | 2023-01-01T00:04:00Z | 0.03085914700123513 | | 2023-01-01T00:08:00Z | 0.0227877053636946 | | 2023-01-01T00:10:00Z | 0.001676063810538834 | | 2023-01-01T00:11:00Z | 0.014999637478226817 | | time | non_negative_derivative | | --- | --- | | time | non_negative_derivative | | 2023-01-01T00:01:00Z | 1.5033871148979698 | | 2023-01-01T00:03:00Z | 2.0498496425431316 | | 2023-01-01T00:04:00Z | 9.257744100370537 | | 2023-01-01T00:08:00Z | 6.836311609108379 | | 2023-01-01T00:10:00Z | 0.5028191431616502 | | 2023-01-01T00:11:00Z | 4.499891243468045 | | time | non_negative_derivative_a | non_negative_derivative_b | | --- | --- | --- | | time | non_negative_derivative_a | non_negative_derivative_b | | 2023-01-01T00:01:00Z | | 0.005011290382993233 | | 2023-01-01T00:03:00Z | 0.0030371603214445152 | 0.006832832141810439 | | 2023-01-01T00:04:00Z | | 0.03085914700123513 | | 2023-01-01T00:05:00Z | 0.00024692132561391543 | | | 2023-01-01T00:06:00Z | 0.016704951104985283 | | | 2023-01-01T00:08:00Z | | 0.0227877053636946 | | 2023-01-01T00:09:00Z | 0.018437240876186967 | | | 2023-01-01T00:10:00Z | | 0.001676063810538834 | | 2023-01-01T00:11:00Z | | 0.014999637478226817 | | 2023-01-01T00:13:00Z | 0.006694752202850366 | | | 2023-01-01T00:14:00Z | 0.011836797386191167 | | | time | non_negative_derivative_a | non_negative_derivative_b | | --- | --- | --- | | time | non_negative_derivative_a | non_negative_derivative_b | | 2023-01-01T00:01:00Z | | 0.005011290382993233 | | 2023-01-01T00:03:00Z | 0.0030371603214445152 | 0.006832832141810439 | | 2023-01-01T00:04:00Z | | 0.03085914700123513 | | 2023-01-01T00:05:00Z | 0.00024692132561391543 | | | 2023-01-01T00:06:00Z | 0.016704951104985283 | | | 2023-01-01T00:08:00Z | | 0.0227877053636946 | | 2023-01-01T00:09:00Z | 0.018437240876186967 | | | 2023-01-01T00:10:00Z | | 0.001676063810538834 | | 2023-01-01T00:11:00Z | | 0.014999637478226817 | | 2023-01-01T00:13:00Z | 0.006694752202850366 | | | 2023-01-01T00:14:00Z | 0.011836797386191167 | | | time | non_negative_derivative | | --- | --- | | time | non_negative_derivative | | 2023-01-01T00:20:00Z | 0.010434324849926194 | | 2023-01-01T00:40:00Z | 0.05657514203880348 | | 2023-01-01T00:50:00Z | 0.021317362457152655 | | time | non_negative_difference | | --- | --- | | time | non_negative_difference | | 2023-01-01T00:01:00Z | 0.300677422979594 | | 2023-01-01T00:03:00Z | 0.40996992850862635 | | 2023-01-01T00:04:00Z | 1.8515488200741077 | | 2023-01-01T00:08:00Z | 1.367262321821676 | | 2023-01-01T00:10:00Z | 0.10056382863233004 | | 2023-01-01T00:11:00Z | 0.899978248693609 | | time | non_negative_difference_a | non_negative_difference_b | | --- | --- | --- | | time | non_negative_difference_a | non_negative_difference_b | | 2023-01-01T00:01:00Z | | 0.300677422979594 | | 2023-01-01T00:03:00Z | 0.18222961928667092 | 0.40996992850862635 | | 2023-01-01T00:04:00Z | | 1.8515488200741077 | | 2023-01-01T00:05:00Z | 0.014815279536834924 | | | 2023-01-01T00:06:00Z | 1.002297066299117 | | | 2023-01-01T00:08:00Z | | 1.367262321821676 | | 2023-01-01T00:09:00Z | 1.106234452571218 | | | 2023-01-01T00:10:00Z | | 0.10056382863233004 | | 2023-01-01T00:11:00Z | | 0.899978248693609 | | 2023-01-01T00:13:00Z | 0.401685132171022 | | | 2023-01-01T00:14:00Z | 0.71020784317147 | | | time | non_negative_difference_a | non_negative_difference_b | | --- | --- | --- | | time | non_negative_difference_a | non_negative_difference_b | | 2023-01-01T00:01:00Z | | 0.300677422979594 | | 2023-01-01T00:03:00Z | 0.18222961928667092 | 0.40996992850862635 | | 2023-01-01T00:04:00Z | | 1.8515488200741077 | | 2023-01-01T00:05:00Z | 0.014815279536834924 | | | 2023-01-01T00:06:00Z | 1.002297066299117 | | | 2023-01-01T00:08:00Z | | 1.367262321821676 | | 2023-01-01T00:09:00Z | 1.106234452571218 | | | 2023-01-01T00:10:00Z | | 0.10056382863233004 | | 2023-01-01T00:11:00Z | | 0.899978248693609 | | 2023-01-01T00:13:00Z | 0.401685132171022 | | | 2023-01-01T00:14:00Z | 0.71020784317147 | | | time | non_negative_difference | | --- | --- | | time | non_negative_difference | | 2023-01-01T00:20:00Z | 0.10434324849926194 | | 2023-01-01T00:40:00Z | 0.5657514203880348 | | 2023-01-01T00:50:00Z | 0.21317362457152655 | | time | b | pow | | --- | --- | --- | | time | b | pow | | 2023-01-01T00:00:00Z | -0.163643058925645 | -0.004382205777325515 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0.002573288422171338 | | 2023-01-01T00:02:00Z | -0.482943221384294 | -0.1126388541916811 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | -0.0003885901893904874 | | 2023-01-01T00:04:00Z | 1.77857552719844 | 5.626222933751733 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 0.4071119474284653 | | time | pow_a | pow_b | | --- | --- | --- | | time | pow_a | pow_b | | 2023-01-01T00:00:00Z | 0.004483135555212479 | -0.00011735131084020357 | | 2023-01-01T00:01:00Z | -0.2795528536239978 | 0.000048322282876973225 | | 2023-01-01T00:02:00Z | -0.6628050073932118 | -0.026271227986693114 | | 2023-01-01T00:03:00Z | -0.22011853819169455 | -0.000002069282189962477 | | 2023-01-01T00:04:00Z | -0.6103699296012646 | 17.797604890097084 | | 2023-01-01T00:05:00Z | -0.5620694808926487 | 0.22362640363833164 | | time | pow | | --- | --- | | time | pow | | 2023-01-01T00:00:00Z | 2402278.159218532 | | 2023-01-01T00:10:00Z | 12.380844221267186 | | 2023-01-01T00:20:00Z | 0.000006244365466732681 | | 2023-01-01T00:30:00Z | 0.0000046424621235691315 | | 2023-01-01T00:40:00Z | 2.7973126174031977e-7 | | 2023-01-01T00:50:00Z | 81.6292140233699 | | time | b | round | | --- | --- | --- | | time | b | round | | 2023-01-01T00:00:00Z | -0.163643058925645 | -0 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0 | | 2023-01-01T00:02:00Z | -0.482943221384294 | -0 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | -0 | | 2023-01-01T00:04:00Z | 1.77857552719844 | 2 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 1 | | time | round_a | round_b | | --- | --- | --- | | time | round_a | round_b | | 2023-01-01T00:00:00Z | 0 | -0 | | 2023-01-01T00:01:00Z | -1 | 0 | | 2023-01-01T00:02:00Z | -1 | -0 | | 2023-01-01T00:03:00Z | -1 | -0 | | 2023-01-01T00:04:00Z | -1 | 2 | | 2023-01-01T00:05:00Z | -1 | 1 | | time | round | | --- | --- | | time | round | | 2023-01-01T00:00:00Z | -4 | | 2023-01-01T00:10:00Z | 1 | | 2023-01-01T00:20:00Z | 0 | | 2023-01-01T00:30:00Z | 0 | | 2023-01-01T00:40:00Z | 0 | | 2023-01-01T00:50:00Z | 2 | | time | b | sin | | --- | --- | --- | | time | b | sin | | 2023-01-01T00:00:00Z | -0.163643058925645 | -0.1629136686003898 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0.13660588515594851 | | 2023-01-01T00:02:00Z | -0.482943221384294 | -0.4643877941052164 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | -0.0729085450859347 | | 2023-01-01T00:04:00Z | 1.77857552719844 | 0.9784914502058565 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 0.6751348197618099 | | time | sin_a | sin_b | | --- | --- | --- | | time | sin_a | sin_b | | 2023-01-01T00:00:00Z | 0.3326300722640741 | -0.1629136686003898 | | 2023-01-01T00:01:00Z | -0.6997047077914582 | 0.13660588515594851 | | 2023-01-01T00:02:00Z | -0.7962295291135749 | -0.4643877941052164 | | 2023-01-01T00:03:00Z | -0.673406844448706 | -0.0729085450859347 | | 2023-01-01T00:04:00Z | -0.7870301289278495 | 0.9784914502058565 | | 2023-01-01T00:05:00Z | -0.7778043295686337 | 0.6751348197618099 | | time | sin | | --- | --- | | time | sin | | 2023-01-01T00:00:00Z | 0.933528830283535 | | 2023-01-01T00:10:00Z | 0.9597472276784815 | | 2023-01-01T00:20:00Z | 0.29712628761434723 | | 2023-01-01T00:30:00Z | 0.2887011711003489 | | 2023-01-01T00:40:00Z | 0.21934537994884437 | | 2023-01-01T00:50:00Z | 0.9998424824522808 | | time | b | sqrt | | --- | --- | --- | | time | b | sqrt | | 2023-01-01T00:00:00Z | -0.163643058925645 | | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0.370181528515334 | | 2023-01-01T00:02:00Z | -0.482943221384294 | | | 2023-01-01T00:03:00Z | -0.0729732928756677 | | | 2023-01-01T00:04:00Z | 1.77857552719844 | 1.3336324558132349 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 0.860899207349059 | | time | sqrt_a | sqrt_b | | --- | --- | --- | | time | sqrt_a | sqrt_b | | 2023-01-01T00:00:00Z | 0.5823152811928947 | | | 2023-01-01T00:01:00Z | | 0.370181528515334 | | 2023-01-01T00:02:00Z | | | | 2023-01-01T00:03:00Z | | | | 2023-01-01T00:04:00Z | | 1.3336324558132349 | | 2023-01-01T00:05:00Z | | 0.860899207349059 | | time | sqrt | | --- | --- | | time | sqrt | | 2023-01-01T00:00:00Z | | | 2023-01-01T00:10:00Z | 1.134063865909604 | | 2023-01-01T00:20:00Z | 0.5492555015405052 | | 2023-01-01T00:30:00Z | 0.5411746169982342 | | 2023-01-01T00:40:00Z | 0.4702589287652642 | | 2023-01-01T00:50:00Z | 1.2462130097934059 | | time | b | tan | | --- | --- | --- | | time | b | tan | | 2023-01-01T00:00:00Z | -0.163643058925645 | -0.16511961248511045 | | 2023-01-01T00:01:00Z | 0.137034364053949 | 0.13789861917955581 | | 2023-01-01T00:02:00Z | -0.482943221384294 | -0.5243575352718546 | | 2023-01-01T00:03:00Z | -0.0729732928756677 | -0.07310309943905952 | | 2023-01-01T00:04:00Z | 1.77857552719844 | -4.743341375725582 | | 2023-01-01T00:05:00Z | 0.741147445214238 | 0.9151958486043346 | | time | tan_a | tan_b | | --- | --- | --- | | time | tan_a | tan_b | | 2023-01-01T00:00:00Z | 0.3527145610408791 | -0.16511961248511045 | | 2023-01-01T00:01:00Z | -0.9793857830953787 | 0.13789861917955581 | | 2023-01-01T00:02:00Z | -1.3160934857179802 | -0.5243575352718546 | | 2023-01-01T00:03:00Z | -0.9109052733075013 | -0.07310309943905952 | | 2023-01-01T00:04:00Z | -1.2757522322802637 | -4.743341375725582 | | 2023-01-01T00:05:00Z | -1.2375438046768912 | 0.9151958486043346 | | time | tan | | --- | --- | | time | tan | | 2023-01-01T00:00:00Z | -2.603968631156288 | | 2023-01-01T00:10:00Z | 3.4171098358131733 | | 2023-01-01T00:20:00Z | 0.31117972731464494 | | 2023-01-01T00:30:00Z | 0.30154101138968664 | | 2023-01-01T00:40:00Z | 0.22482036866737865 | | 2023-01-01T00:50:00Z | 56.3338223288096 | --- # GROUP BY clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/group-by/ Use the `GROUP BY` clause to group data by one or more specified[tags](/influxdb3/enterprise/reference/glossary/#tag) or into specified time intervals.`GROUP BY` requires an [aggregate](/influxdb3/enterprise/reference/influxql/functions/aggregates/)or [selector](/influxdb3/enterprise/reference/influxql/functions/selectors/)function in the `SELECT` statement. ## Syntax ```sql SELECT_clause FROM_clause [WHERE_clause] GROUP BY group_expression[, ..., group_expression_n] ``` * **group\_expression**: Expression to identify tags or time intervals to group by. Can be a [tag key](/influxdb3/enterprise/reference/glossary/#tag-key), constant, [regular expression](/influxdb3/enterprise/reference/influxql/regular-expressions/), wildcard (`*`), or [function expression](/influxdb3/enterprise/reference/influxql/functions/). ## GROUP BY clause behaviors * `GROUP BY tag_key` - Groups data by a specific tag * `GROUP BY tag_key1, tag_key2` - Groups data by more than one tag * `GROUP BY *` - Groups data by all [tags](/influxdb3/enterprise/reference/glossary/#tag) * `GROUP BY /regex/` - Groups data by tag keys that match the regular expression * `GROUP BY time()` - Groups data into time intervals (windows) > [!Note] > If a query includes `WHERE` and `GROUP BY`, the `GROUP BY` clause must appear after > the `WHERE` clause. ## GROUP BY tags Groups data by one or more tag columns. ### GROUP BY tags examples The following examples use the[Bitcoin price sample data](/influxdb3/enterprise/reference/sample-data/#bitcoin-price-data). [](#group-data-by-a-single-tag) Group data by a single tag ```sql SELECT MEAN(*) FROM bitcoin GROUP BY code ``` name: bitcoin tags: code=EUR | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|27328.848667840004| name: bitcoin tags: code=GBP | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|23441.832453919982| name: bitcoin tags: code=USD | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|28054.160950480004| [](#group-data-by-more-than-one-tag) Group data by more than one tag ```sql SELECT MEAN(*) FROM bitcoin GROUP BY code, description ``` name: bitcoin tags: code=EUR, description=Euro | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|27328.848667840004| name: bitcoin tags: code=GBP, description=British Pound Sterling | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|23441.832453919982| name: bitcoin tags: code=USD, description=United States Dollar | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|28054.160950480004| [](#group-data-by-all-tags) Group data by all tags ```sql SELECT MEAN(*) FROM bitcoin GROUP BY * ``` name: bitcoin tags: code=EUR, crypto=bitcoin, description=Euro, symbol=€ | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|27328.848667840004| name: bitcoin tags: code=GBP, crypto=bitcoin, description=British Pound Sterling, symbol=£ | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|23441.832453919982| name: bitcoin tags: code=USD, crypto=bitcoin, description=United States Dollar, symbol=$ | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|28054.160950480004| [](#group-data-by-tag-keys-that-match-a-regular-expression) Group data by tag keys that match a regular expression ```sql SELECT MEAN(*) FROM bitcoin GROUP BY /^[cd]/ ``` name: bitcoin tags: code=EUR, crypto=bitcoin, description=Euro | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|27328.848667840004| name: bitcoin tags: code=GBP, crypto=bitcoin, description=British Pound Sterling | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|23441.832453919982| name: bitcoin tags: code=USD, crypto=bitcoin, description=United States Dollar | time | mean\_price | |--------------------|------------------| |1970-01-01T00:00:00Z|28054.160950480004| ## GROUP BY time `GROUP BY time()` groups data by into specified time intervals, also known as “windows”, and applies the [aggregate](/influxdb3/enterprise/reference/influxql/functions/aggregates/)and [selector](/influxdb3/enterprise/reference/influxql/functions/selectors/)functions in the `SELECT` clause to each interval. Use the [`time()` function](/influxdb3/enterprise/reference/influxql/functions/date-time/#time)to specify the time interval to group by. ```sql SELECT_clause FROM_clause WHERE GROUP BY time(time_interval[, offset])[, group_expression (...)] [fill(behavior)] ``` `GROUP BY time()` intervals use preset round-number time boundaries that are independent of time conditions in the [`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/).**Output data uses window start boundaries as the aggregate timestamps.**Use the `offset` argument of the `time()` function to shift time boundaries forward or backward in time. ### GROUP by time and fill gaps When grouping by time, if a window in the queried time range does not contain data, results return a row for the empty window containing the timestamp of the empty window and *null* values for each queried field. Use the [`fill()` function](/influxdb3/enterprise/reference/influxql/functions/misc/#fill)*at the end of the `GROUP BY` clause* to replace *null* field values. If no `FILL` clause is included, the default behavior is `fill(null)`. `fill()` provides the following behaviors for filling values: * **numeric literal**: Replaces null values with the specified numeric literal. * **linear**: Uses linear interpolation between existing values to replace null values. * **none**: Removes rows with null field values. * **null**: Keeps null values and associated timestamps. * **previous**: Replaces null values with the most recent non-null value. *See the [`fill()` documentation](/influxdb3/enterprise/reference/influxql/functions/misc/#fill)for detailed examples.* ### GROUP BY time examples The following examples use the[Bitcoin price sample data](/influxdb3/enterprise/reference/sample-data/#bitcoin-price-data). [](#group-and-aggregate-query-results-into-1-hour-windows) Group and aggregate query results into 1 hour windows ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-15T00:00:00Z' GROUP BY time(1h) ``` name: bitcoin | time | mean | |--------------------|------------------| |2023-05-01T00:00:00Z| 24494.27265 | |2023-05-01T01:00:00Z| 24452.1698 | |2023-05-01T02:00:00Z|23902.666124999996| |2023-05-01T03:00:00Z|23875.211349999998| |2023-05-01T04:00:00Z| 23855.6441 | | … | … | [](#group-and-aggregate-query-results-into-1-week-intervals-by-tag) Group and aggregate query results into 1 week intervals by tag ```sql SELECT MEAN(price) FROM bitcoin WHERE time >= '2023-05-01T00:00:00Z' AND time < '2023-05-15T00:00:00Z' GROUP BY time(1w), code ``` name: bitcoin tags: code=EUR | time | mean | |--------------------|------------------| |2023-04-27T00:00:00Z|27681.21808576779 | |2023-05-04T00:00:00Z|27829.413580354256| |2023-05-11T00:00:00Z|26210.24799033149 | name: bitcoin tags: code=GBP | time | mean | |--------------------|------------------| |2023-04-27T00:00:00Z|23744.083925842704| |2023-05-04T00:00:00Z|23871.201395652173| |2023-05-11T00:00:00Z|22482.33174723755 | name: bitcoin tags: code=USD | time | mean | |--------------------|------------------| |2023-04-27T00:00:00Z|28415.88231123595 | |2023-05-04T00:00:00Z|28568.010941384844| |2023-05-11T00:00:00Z|26905.87242099449 | ###### GROUP BY time with offset [](#group-and-aggregate-query-results-into-1-hour-intervals-and-offset-time-boundaries-by-15-minutes) Group and aggregate query results into 1 hour intervals and offset time boundaries by +15 minutes ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-02T00:00:00Z' GROUP BY time(1h, 15m) ``` name: bitcoin | time | mean | |--------------------|------------------| |2023-04-30T23:15:00Z| | |2023-05-01T00:15:00Z| 29313.6754 | |2023-05-01T01:15:00Z| 28932.0882 | |2023-05-01T02:15:00Z|28596.375225000003| |2023-05-01T03:15:00Z| 28578.915075 | | … | … | *Note that `offset` forces the first time boundary to be outside the queried time range so the query returns no results for that first interval.* [](#group-and-aggregate-query-results-into-1-hour-intervals-and-offset-time-boundaries-by--15-minutes) Group and aggregate query results into 1 hour intervals and offset time boundaries by -15 minutes ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-02T00:00:00Z' GROUP BY time(1h, -15m) ``` name: bitcoin | time | mean | |--------------------|------------------| |2023-04-30T23:45:00Z| 29319.9092 | |2023-05-01T00:45:00Z| 29285.3651 | |2023-05-01T01:45:00Z|28607.202666666668| |2023-05-01T02:45:00Z| 28576.056175 | |2023-05-01T03:45:00Z| 28566.96315 | | … | … | ###### GROUP BY time and fill gaps [](#group-and-aggregate-query-results-into-30-minute-intervals-and-fill-gaps-with-0) Group and aggregate query results into 30 minute intervals and fill gaps with `0` ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(0) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z| 0 | |2023-05-01T01:30:00Z|29263.2886| [](#group-and-aggregate-query-results-into-30-minute-intervals-and-fill-gaps-using-linear-interpolation) Group and aggregate query results into 30 minute intervals and fill gaps using linear interpolation ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(linear) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z|29285.3651| |2023-05-01T01:30:00Z|29263.2886| [](#group-and-aggregate-query-results-into-30-minute-intervals-and-fill-gaps-with-previous-values) Group and aggregate query results into 30 minute intervals and fill gaps with previous values ```sql SELECT MEAN(price) FROM bitcoin WHERE code = 'USD' AND time >= '2023-05-01T00:00:00Z' AND time < '2023-05-01T02:00:00Z' GROUP BY time(30m) fill(previous) ``` name: bitcoin | time | mean | |--------------------|----------| |2023-05-01T00:00:00Z|29319.9092| |2023-05-01T00:30:00Z|29307.4416| |2023-05-01T01:00:00Z|29307.4416| |2023-05-01T01:30:00Z|29263.2886| ## Result set If at least one row satisfies the query, InfluxDB 3 Enterprise returns row data in the query result set. If a query uses a `GROUP BY` clause, the result set includes the following: * Columns listed in the query’s `SELECT` clause * A `time` column that contains the timestamp for the record or the group * An `iox::measurement` column that contains the record’s measurement (table) name * Columns listed in the query’s `GROUP BY` clause; each row in the result set contains the values used for grouping ### Default time range If a query doesn’t specify a time range in the[`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/), InfluxDB uses the[default time range](/influxdb3/enterprise/reference/influxql/#default-time-range) for filtering and grouping by time. If a query includes the `GROUP BY` clause and doesn’t specify a time range in the`WHERE` clause, the default time group is the[default time range](/influxdb3/enterprise/reference/influxql/#default-time-range), and the `time` column in the result set contains the start of the range–for example: ```sql SELECT mean(temp) FROM home GROUP BY room ``` name: home tags: room=Kitchen | time | mean | |--------------------|------------------| |1970-01-01T00:00:00Z|22.623076923076926| name: home tags: room=Living Room | time | mean | |--------------------|-----------------| |1970-01-01T00:00:00Z|22.16923076923077| ## Notable behaviors of the GROUP BY clause ### Cannot group by fields InfluxQL does not support grouping data by **fields**. ### Tag order does not matter The order that tags are listed in the `GROUP BY` clause does not affect how data is grouped. ### Grouping by tag and no time range returns unexpected timestamps The `time` column contains the start of the [default time range](#default-time-range). ### Data grouped by time may return unexpected timestamps Because `GROUP BY time()` intervals use preset round-number time boundaries that are independent of time conditions in the [`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/), results may include timestamps outside of the queried time range. Results represent only data with timestamps in the specified time range, but output timestamps are determined by by the preset time boundaries. The following example groups data by 1-hour intervals, but the time range defined in the `WHERE` clause covers only part of a window: ``` SELECT MEAN(field) FROM example WHERE time >= '2022-01-01T00:30:00Z' AND time <= '2022-01-01T01:30:00Z' GROUP BY time(1h) ``` > [!Note] > **Note**: The timestamp in the first row of query results data occurs before the start of > the queried time range.[See why](#why-do-these-results-include-timestamps-outside-of-the-queried-time-range). #### Example data | time |field| |--------------------|-----| |2022-01-01T00:00:00Z| 8 | |2022-01-01T00:15:00Z| 4 | |2022-01-01T00:30:00Z| 0 | |2022-01-01T00:45:00Z| 8 | |2022-01-01T01:00:00Z| 5 | |2022-01-01T01:15:00Z| 0 | |2022-01-01T01:30:00Z| 8 | |2022-01-01T01:45:00Z| 8 | |2022-01-01T02:00:00Z| 9 | |2022-01-01T02:15:00Z| 6 | |2022-01-01T02:30:00Z| 3 | |2022-01-01T02:45:00Z| 0 | #### Query results | time |field| |--------------------|-----| |2022-01-01T00:00:00Z| 4 | |2022-01-01T01:00:00Z|5.25 | |2022-01-01T02:00:00Z| 6 | [](#why-do-these-results-include-timestamps-outside-of-the-queried-time-range) Why do these results include timestamps outside of the queried time range? `GROUP BY time()` creates windows with predefined time boundaries based on the specified interval. These boundaries are not determined by the queried time range, however, aggregate values in query results are calculated using only values that are in the queried time range. | time |field| |:-------------------|----:| |2022-01-01T00:00:00Z| 8 | |2022-01-01T00:15:00Z| 4 | |2022-01-01T00:30:00Z| 0 | |2022-01-01T00:45:00Z| 8 | |2022-01-01T01:00:00Z| 5 | |2022-01-01T01:15:00Z| 0 | |2022-01-01T01:30:00Z| 8 | |2022-01-01T01:45:00Z| 8 | |2022-01-01T02:00:00Z| 9 | |2022-01-01T02:15:00Z| 6 | |2022-01-01T02:30:00Z| 3 | |2022-01-01T02:45:00Z| 0 | 2022-01-01T00:00:00Z 2022-01-01T01:00:00Z 2022-01-01T02:00:00Z ### Fill with no data in the queried time range Queries ignore `fill()` if no data exists in the queried time range. This is the expected behavior. ### Fill with previous if no previous value exists `fill(previous)` doesn’t fill null values if there is no previous value in the queried time range. ### Fill with linear interpolation if there are not two values to interpolate between `fill(linear)` doesn’t fill null values if there are no values before or after the null value in the queried time range. | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 27328.848667840004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 23441.832453919982 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 28054.160950480004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 27328.848667840004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 23441.832453919982 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 28054.160950480004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 27328.848667840004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 23441.832453919982 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 28054.160950480004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 27328.848667840004 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 23441.832453919982 | | time | mean_price | | --- | --- | | time | mean_price | | 1970-01-01T00:00:00Z | 28054.160950480004 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 24494.27265 | | 2023-05-01T01:00:00Z | 24452.1698 | | 2023-05-01T02:00:00Z | 23902.666124999996 | | 2023-05-01T03:00:00Z | 23875.211349999998 | | 2023-05-01T04:00:00Z | 23855.6441 | | … | … | | time | mean | | --- | --- | | time | mean | | 2023-04-27T00:00:00Z | 27681.21808576779 | | 2023-05-04T00:00:00Z | 27829.413580354256 | | 2023-05-11T00:00:00Z | 26210.24799033149 | | time | mean | | --- | --- | | time | mean | | 2023-04-27T00:00:00Z | 23744.083925842704 | | 2023-05-04T00:00:00Z | 23871.201395652173 | | 2023-05-11T00:00:00Z | 22482.33174723755 | | time | mean | | --- | --- | | time | mean | | 2023-04-27T00:00:00Z | 28415.88231123595 | | 2023-05-04T00:00:00Z | 28568.010941384844 | | 2023-05-11T00:00:00Z | 26905.87242099449 | | time | mean | | --- | --- | | time | mean | | 2023-04-30T23:15:00Z | | | 2023-05-01T00:15:00Z | 29313.6754 | | 2023-05-01T01:15:00Z | 28932.0882 | | 2023-05-01T02:15:00Z | 28596.375225000003 | | 2023-05-01T03:15:00Z | 28578.915075 | | … | … | | time | mean | | --- | --- | | time | mean | | 2023-04-30T23:45:00Z | 29319.9092 | | 2023-05-01T00:45:00Z | 29285.3651 | | 2023-05-01T01:45:00Z | 28607.202666666668 | | 2023-05-01T02:45:00Z | 28576.056175 | | 2023-05-01T03:45:00Z | 28566.96315 | | … | … | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | 0 | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | 29285.3651 | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 2023-05-01T00:00:00Z | 29319.9092 | | 2023-05-01T00:30:00Z | 29307.4416 | | 2023-05-01T01:00:00Z | 29307.4416 | | 2023-05-01T01:30:00Z | 29263.2886 | | time | mean | | --- | --- | | time | mean | | 1970-01-01T00:00:00Z | 22.623076923076926 | | time | mean | | --- | --- | | time | mean | | 1970-01-01T00:00:00Z | 22.16923076923077 | | time | field | | --- | --- | | time | field | | 2022-01-01T00:00:00Z | 8 | | 2022-01-01T00:15:00Z | 4 | | 2022-01-01T00:30:00Z | 0 | | 2022-01-01T00:45:00Z | 8 | | 2022-01-01T01:00:00Z | 5 | | 2022-01-01T01:15:00Z | 0 | | 2022-01-01T01:30:00Z | 8 | | 2022-01-01T01:45:00Z | 8 | | 2022-01-01T02:00:00Z | 9 | | 2022-01-01T02:15:00Z | 6 | | 2022-01-01T02:30:00Z | 3 | | 2022-01-01T02:45:00Z | 0 | | time | field | | --- | --- | | time | field | | 2022-01-01T00:00:00Z | 4 | | 2022-01-01T01:00:00Z | 5.25 | | 2022-01-01T02:00:00Z | 6 | | time | field | | --- | --- | | time | field | | 2022-01-01T00:00:00Z | 8 | | 2022-01-01T00:15:00Z | 4 | | 2022-01-01T00:30:00Z | 0 | | 2022-01-01T00:45:00Z | 8 | | 2022-01-01T01:00:00Z | 5 | | 2022-01-01T01:15:00Z | 0 | | 2022-01-01T01:30:00Z | 8 | | 2022-01-01T01:45:00Z | 8 | | 2022-01-01T02:00:00Z | 9 | | 2022-01-01T02:15:00Z | 6 | | 2022-01-01T02:30:00Z | 3 | | 2022-01-01T02:45:00Z | 0 | --- # InfluxQL internals Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/internals/ Learn about the implementation of InfluxQL to understand how results are processed and how to create efficient queries: ## Query life cycle 1. InfluxQL query string is tokenized and then parsed into an abstract syntax tree (AST). This is the code representation of the query itself. 2. The AST is passed to the `QueryExecutor` which directs queries to the appropriate handlers. For example, queries related to meta data are executed by the **meta service** and `SELECT` statements are executed by the shards themselves. 3. The query engine then determines the shards that match the `SELECT`statement’s time range. From these shards, iterators are created for each field in the statement. 4. Iterators are passed to the emitter which drains them and joins the resulting points. The emitter’s job is to convert simple time/value points into the more complex result objects that are returned to the client. ### Understanding iterators Iterators provide a simple interface for looping over a set of points. For example, this is an iterator over Float points: ``` type FloatIterator interface { Next() *FloatPoint } ``` These iterators are created through the `IteratorCreator` interface: ``` type IteratorCreator interface { CreateIterator(opt *IteratorOptions) (Iterator, error) } ``` The `IteratorOptions` provide arguments about field selection, time ranges, and dimensions that the iterator creator can use when planning an iterator. The `IteratorCreator` interface is used at many levels such as the `Shards`,`Shard`, and `Engine`. This allows optimizations to be performed when applicable such as returning a precomputed `COUNT()`. Iterators aren’t just for reading raw data from storage, though. Iterators can be composed so that they provide additional functionality around an input iterator. For example, a `DistinctIterator` can compute the distinct values for each time window for an input iterator. Or a `FillIterator` can generate additional points that are missing from an input iterator. This composition also lends itself well to aggregation. For example, in the following SQL, `MEAN(value)` is a `MeanIterator` that wraps an iterator from the underlying shards: ```sql SELECT MEAN(value) FROM cpu GROUP BY time(10m) ``` The following example wraps `MEAN(value)` with an additional iterator (`DERIVATIVE()`) to determine the derivative of the mean: ```sql SELECT DERIVATIVE(MEAN(value), 20m) FROM cpu GROUP BY time(10m) ``` ### Cursors A **cursor** identifies data by shard in tuples (time, value) for a single series (measurement, tag set and field). The cursor traverses data stored as a log-structured merge-tree and handles deduplication across levels, tombstones for deleted data, and merging the cache (Write Ahead Log). A cursor sorts the `(time, value)` tuples by time in ascending or descending order. For example, a query that evaluates one field for 1,000 series over 3 shards constructs a minimum of 3,000 cursors (1,000 per shard). ### Auxiliary fields Because InfluxQL allows users to use selector functions such as `FIRST()`,`LAST()`, `MIN()`, and `MAX()`, the engine must provide a way to return related data at the same time with the selected point. Let’s look at the following query: ```sql SELECT FIRST(value), host FROM cpu GROUP BY time(1h) ``` We are selecting the first `value` that occurs every hour but we also want to retrieve the `host` associated with that point. Since the `Point` types only specify a single typed `Value` for efficiency, we push the `host` into the auxiliary fields of the point. These auxiliary fields are attached to the point until it is passed to the emitter where the fields get split off to their own iterator. ### Built-in iterators There are many helper iterators that let us build queries: * Merge Iterator - This iterator combines one or more iterators into a single new iterator of the same type. This iterator guarantees that all points within a window will be output before starting the next window, but does not provide ordering guarantees within the window. This allows for fast access for aggregate queries that don’t need stronger sorting guarantees. * Sorted Merge Iterator - Like `MergeIterator`, this iterator combines one or more iterators into a new iterator of the same type. However, this iterator guarantees time ordering of every point. This makes it slower than the `MergeIterator`but this ordering guarantee is required for non-aggregate queries which return the raw data points. * Limit Iterator - This iterator limits the number of points per name or tag group. This is the implementation of the `LIMIT` & `OFFSET` syntax. * Fill Iterator - This iterator injects extra points if they are missing from the input iterator. It can provide `null` points, points with the previous value, or points with a specific value. * Buffered Iterator - This iterator provides the ability to “unread” a point back onto a buffer so it can be read again next time. This is used extensively to provide lookahead for windowing. * Reduce Iterator - This iterator calls a reduction function for each point in a window. When the window is complete, then all points for that window are output. This is used for simple aggregate functions such as `COUNT()`. * Reduce Slice Iterator - This iterator collects all points for a window first, and then passes them all to a reduction function at once. The results are returned from the iterator. This is used for aggregate functions such as`DERIVATIVE()`. * Transform Iterator - This iterator calls a transform function for each point from an input iterator. This is used for executing binary expressions. * Dedupe Iterator - This iterator only outputs unique points. Because it is resource-intensive, this iterator is only used for small queries such as meta query statements. ### Call iterators Function calls in InfluxQL are implemented at two levels: * Some calls can be wrapped at multiple layers to improve efficiency. For example, a `COUNT()` can be performed at the shard level and then multiple `CountIterator`s can be wrapped with another `CountIterator` to compute the count of all shards. These iterators can be created using `NewCallIterator()`. * Some iterators are more complex or need to be implemented at a higher level. For example, the `DERIVATIVE()` function needs to retrieve all points for a window before performing the calculation. This iterator is created by the engine itself and is never requested to be created by the lower levels. --- # LIMIT and SLIMIT clauses Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/limit-and-slimit/ Use `LIMIT` to limit the number of **rows** returned per InfluxQL group. Use `SLIMIT` to limit the number of [series](/influxdb3/enterprise/reference/glossary/#series)returned in query results. ## LIMIT clause The `LIMIT` clause limits the number of rows to return from each InfluxQL group. If the query doesn’t include a [`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/), the entire result set is considered a single group. If a query [groups data by time](/influxdb3/enterprise/reference/influxql/group-by/#group-by-time), limits are applied after aggregate and selector operations are applied to each time window. ### Syntax ```sql SELECT_clause FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] LIMIT N ``` #### Arguments * **N**: Maximum number of points to return from each InfluxQL group. If `N` is greater than the number of points in a group, all points from the group are returned. ### Examples The following examples use the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). [](#limit-the-number-of-rows-returned) Limit the number of rows returned ```sql SELECT * FROM home LIMIT 3 ``` Name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T08:00:00Z| 0 |35.9| Kitchen | 21 | |2022-01-01T08:00:00Z| 0 |35.9|Living Room|21.1| |2022-01-01T09:00:00Z| 0 |36.2| Kitchen | 23 | [](#limit-the-number-of-rows-returned-from-each-influxql-group) Limit the number of rows returned from each InfluxQL group ```sql SELECT MEAN(*) FROM home GROUP BY time(2h), room LIMIT 3 ``` name: home tags: room=Kitchen | time |mean\_co|mean\_hum| mean\_temp | |--------------------|--------|---------|------------------| |2022-01-01T08:00:00Z| 0 | 36.05 | 22 | |2022-01-01T10:00:00Z| 0 | 36.05 |22.549999999999997| |2022-01-01T12:00:00Z| 0.5 | 36.25 | 22.65 | name: home tags: room=Living Room | time |mean\_co|mean\_hum| mean\_temp | |--------------------|--------|---------|------------------| |2022-01-01T08:00:00Z| 0 | 35.9 | 21.25 | |2022-01-01T10:00:00Z| 0 | 36 | 22 | |2022-01-01T12:00:00Z| 0 | 35.95 |22.299999999999997| ## SLIMIT clause > [!Important] > InfluxQL is being rearchitected to work with the InfluxDB 3 storage engine. > This process is ongoing and some InfluxQL features, such as `SLIMIT` are still > being implemented. For more information, see[InfluxQL feature support](/influxdb3/enterprise/reference/influxql/feature-support/). | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T08:00:00Z | 0 | 35.9 | Living Room | 21.1 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | time | mean_co | mean_hum | mean_temp | | --- | --- | --- | --- | | time | mean_co | mean_hum | mean_temp | | 2022-01-01T08:00:00Z | 0 | 36.05 | 22 | | 2022-01-01T10:00:00Z | 0 | 36.05 | 22.549999999999997 | | 2022-01-01T12:00:00Z | 0.5 | 36.25 | 22.65 | | time | mean_co | mean_hum | mean_temp | | --- | --- | --- | --- | | time | mean_co | mean_hum | mean_temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | 21.25 | | 2022-01-01T10:00:00Z | 0 | 36 | 22 | | 2022-01-01T12:00:00Z | 0 | 35.95 | 22.299999999999997 | --- # InfluxQL math operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/math-operators/ Use InfluxQL mathematical operators to perform mathematical operations in InfluxQL queries. Mathematical operators follow the [standard order of operations](https://golang.org/ref/spec#Operator_precedence). Parentheses take precedence to division and multiplication, which takes precedence to addition and subtraction. For example `5 / 2 + 3 * 2 = (5 / 2) + (3 * 2)` and `5 + 2 * 3 - 2 = 5 + (2 * 3) - 2`. ## Addition Add two numeric operands. Operands may be an identifier, constant, or literal numeric value. ```sql SELECT A + 5 FROM example SELECT A + B FROM example SELECT * FROM example WHERE A + 5 > 10 SELECT * FROM example WHERE A + B > 10 ``` ## Subtraction Subtract one numeric operand from another. Operands may be an identifier, constant, or literal numeric value. ```sql SELECT 1 - A FROM example SELECT B - A FROM example SELECT * FROM example WHERE 1 - A <= 3 SELECT * FROM example WHERE B - A <= 3 ``` ## Multiplication Multiply two numeric operands. Operands may be an identifier, constant, or literal numeric value. ```sql SELECT A * 10 FROM example SELECT A * B FROM example SELECT * FROM example WHERE A * 10 >= 20 SELECT * FROM example WHERE A * B >= 20 ``` Multiplication distributes across other operators. ```sql SELECT 10 * (A + B + C) FROM example SELECT 10 * (A - B - C) FROM example SELECT 10 * (A + B - C) FROM example ``` ## Division Divide one numeric operand by another. Operands may be an identifier, constant, or literal numeric value. ```sql SELECT A / 10 FROM example SELECT A / B FROM example SELECT * FROM example WHERE A / 10 <= 2 SELECT * FROM example WHERE A / B <= 2 ``` Division distributes across other operators. ```sql SELECT 10 / (A + B + C) FROM example SELECT 10 / (A - B - C) FROM example SELECT 10 / (A + B - C) FROM example ``` ## Modulo Perform a modulo operation with two numeric operands. Operands may be an identifier, constant, or literal numeric value. ```sql SELECT A % 2 FROM example SELECT A % B FROM example SELECT A FROM example WHERE A % 2 = 0 SELECT A, B FROM example WHERE A % B = 0 ``` ## Bitwise AND Perform a bitwise `AND` operation on two operands *of the same type*. Supported types are **integers** and **booleans**. Operands may be an identifier, constant, literal integer value, or literal boolean value. ```sql SELECT A & 255 FROM example SELECT A & B FROM example SELECT (A ^ true) & B FROM example SELECT * FROM example WHERE A & 15 > 0 ``` ## Bitwise OR Perform a bitwise `OR` operation on two operands *of the same type*. Supported types are **integers** and **booleans**. Operands may be an identifier, constant, literal integer value, or literal boolean value. ```sql SELECT A | 5 FROM example SELECT A | B FROM example SELECT * FROM example WHERE "bitfield" | 12 = 12 ``` ## Bitwise Exclusive-OR Perform a bitwise `Exclusive-OR` operation on two operands *of the same type*. Supported types are **integers** and **booleans**. Operands may be an identifier, constant, literal integer value, or literal boolean value. ```sql SELECT A ^ 255 FROM example SELECT A ^ B FROM example SELECT * FROM example WHERE "bitfield" ^ 6 > 0 ``` ## Unsupported Operators ### Inequalities Using any of `=`,`!=`,`<`,`>`,`<=`,`>=`,`<>` in the `SELECT` statement yields empty results for all types. Comparison operators can only be used in the `WHERE` clause. ### Logical Operators Using any of `!|`,`NAND`,`XOR`,`NOR` yield a parser error. Additionally using `AND`, `OR` in the `SELECT` clause of a query will not behave as mathematical operators and simply yield empty results, as they are InfluxQL tokens. However, you can apply the bitwise operators `&`, `|` and `^` to boolean values. ### Bitwise Not There is no bitwise-not operator, because the results you expect depend on the width of your bitfield. InfluxQL does not know how wide your bitfield is, so cannot implement a suitable bitwise-not operator. For example, if your bitfield is 8 bits wide, then the integer 1 represents the bits `0000 0001`. The bitwise-not of this should return the bits `1111 1110` (that is, the integer 254) However, if your bitfield is 16 bits wide, then the integer 1 represents the bits `0000 0000 0000 0001`. The bitwise-not of this should return the bits `1111 1111 1111 1110` (that is, the integer 65534) #### Solution You can implement a bitwise-not operation by using the `^` (bitwise xor) operator together with the number representing all-ones for your word-width: For 8-bit data: ```sql SELECT A ^ 255 FROM example ``` For 16-bit data: ```sql SELECT A ^ 65535 FROM example ``` For 32-bit data: ```sql SELECT A ^ 4294967295 FROM example ``` In each case, the constant you need can be calculated as `(2 ** width) - 1`. ## Notable behaviors of mathematical operators ### Mathematical operators with wildcards and regular expressions InfluxQL does not support combining mathematical operations with a wildcard (`*`) or [regular expression](/influxdb3/enterprise/reference/influxql/regular-expressions/)in the `SELECT` clause. The following queries are invalid and the output is an error: Perform a mathematical operation on a wildcard. ```sql SELECT * + 2 FROM "nope" -- ERR: unsupported expression with wildcard: * + 2 ``` Perform a mathematical operation on a wildcard within a function. ```sql SELECT COUNT(*) / 2 FROM "nope" -- ERR: unsupported expression with wildcard: count(*) / 2 ``` Perform a mathematical operation on a regular expression. ```sql SELECT /A/ + 2 FROM "nope" -- ERR: error parsing query: found +, expected FROM at line 1, char 12 ``` Perform a mathematical operation on a regular expression within a function. ```sql SELECT COUNT(/A/) + 2 FROM "nope" -- ERR: unsupported expression with regex field: count(/A/) + 2 ``` ### Mathematical operators with functions InfluxQL does not support mathematical operators inside of function calls. Note that InfluxQL only allows functions in the `SELECT` clause. For example, the following will work: ```sql SELECT 10 * mean("value") FROM "cpu" ``` However, the following query will return a parse error: ```sql SELECT mean(10 * "value") FROM "cpu" -- Error: expected field argument in mean() ``` > [!Tip] > InfluxQL supports [subqueries](/influxdb/v2/query-data/influxql/explore-data/subqueries/)which offer similar functionality to using mathematical operators inside a function call. --- # OFFSET and SOFFSET clauses Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/offset-and-soffset/ Use `OFFSET` to specify the number of [rows](/influxdb3/enterprise/reference/glossary/#series)to skip in each InfluxQL group before returning results. Use `SOFFSET` to specify the number of [series](/influxdb3/enterprise/reference/glossary/#series)to skip before returning results. ## `OFFSET` clause The `OFFSET` clause skips `N` rows in each InfluxQL group before returning results. Offsets honor row limits specified in the[`LIMIT` clause](/influxdb3/enterprise/reference/influxql/limit-and-slimit/#limit-clause)and display the limited number of rows after the specified offset. Use `LIMIT` and `OFFSET` together to paginate query results. ### Syntax ```sql SELECT_clause FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] OFFSET N [SLIMIT_clause] [SOFFSET_clause] ``` #### Arguments * **N**: Number of rows to skip in each InfluxQL group before returning results. ### Notable OFFSET clause behaviors * If there is no `LIMIT` clause in a query with an `OFFSET` clause, the query returns a single row per InfluxQL group at the specified offset. * If the query doesn’t include a [`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/), the entire result set is considered a single group and is returned in full. * If a query [groups data by time](/influxdb3/enterprise/reference/influxql/group-by/#group-by-time), the offset is applied after aggregate and selector operations are applied to each time window. * If the [`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/)includes a time range and the `OFFSET` clause causes InfluxQL to return points with timestamps outside of that time range, InfluxQL returns no results. ### Examples The following examples use the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). [](#return-the-nth-row) Return the nth row ```sql SELECT * FROM home OFFSET 3 ``` Name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T09:00:00Z| 0 |35.9|Living Room|21.4| [](#paginate-results-by-3-and-return-the-2nd-page-of-results) Paginate results by 3 and return the 2nd page of results ```sql SELECT * FROM home WHERE room = 'Kitchen' LIMIT 3 OFFSET 3 ``` Name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| |2022-01-01T13:00:00Z| 1 |36.5|Kitchen|22.8| [](#paginate-results-from-each-series-by-3-and-return-the-2nd-page-of-each-series) Paginate results from each series by 3 and return the 2nd page of each series ```sql SELECT * FROM home GROUP BY * LIMIT 3 OFFSET 3 ``` name: home tags: room=Kitchen | time |co |hum |temp| |--------------------|---|----|----| |2022-01-01T11:00:00Z| 0 | 36 |22.4| |2022-01-01T12:00:00Z| 0 | 36 |22.5| |2022-01-01T13:00:00Z| 1 |36.5|22.8| name: home tags: room=Living Room | time |co |hum |temp| |--------------------|---|----|----| |2022-01-01T11:00:00Z| 0 | 36 |22.2| |2022-01-01T12:00:00Z| 0 |35.9|22.2| |2022-01-01T13:00:00Z| 0 | 36 |22.4| ## `SOFFSET` clause > [!Important] > InfluxQL is being rearchitected to work with the InfluxDB 3 storage engine. > This process is ongoing and some InfluxQL features, such as `SOFFSET` are still > being implemented. For more information, see[InfluxQL feature support](/influxdb3/enterprise/reference/influxql/feature-support/). | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T09:00:00Z | 0 | 35.9 | Living Room | 21.4 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | 2022-01-01T13:00:00Z | 1 | 36.5 | Kitchen | 22.8 | | time | co | hum | temp | | --- | --- | --- | --- | | time | co | hum | temp | | 2022-01-01T11:00:00Z | 0 | 36 | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | 22.5 | | 2022-01-01T13:00:00Z | 1 | 36.5 | 22.8 | | time | co | hum | temp | | --- | --- | --- | --- | | time | co | hum | temp | | 2022-01-01T11:00:00Z | 0 | 36 | 22.2 | | 2022-01-01T12:00:00Z | 0 | 35.9 | 22.2 | | 2022-01-01T13:00:00Z | 0 | 36 | 22.4 | --- # ORDER BY clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/order-by/ Use the `ORDER BY` clause to sort data by time in ascending or descending order. InfluxQL only supports sorting data by `time`. ## Syntax ```sql SELECT_clause FROM_clause [WHERE_clause] [GROUP_BY_clause] ORDER BY time [ASC|DESC] ``` * If the the `ORDER BY` clause is not included, the default behavior is to sort data by time in **ascending** order: `ORDER BY time ASC`. * If the query includes [`WHERE`](/influxdb3/enterprise/reference/influxql/where/)and [`GROUP BY`](/influxdb3/enterprise/reference/influxql/group-by/) clauses, the `ORDER BY` clause must come **after** these clauses. #### Sort orders * **ASC (ascending)**: The first row in the results has the oldest timestamp. The last row in the results has the most recent timestamp. * **DESC (descending)**: The first row in the results has the most recent timestamp. The last row in the results has the oldest timestamp. ## Examples The following examples use the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). [](#sort-data-with-the-oldest-points-first) Sort data with the oldest points first > [!Tip] > Ordering data by time in ascending order is the default behavior. > Including `ORDER BY time ASC` in the query isn’t necessary, but it is supported. ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' ORDER BY time ASC ``` Name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T08:00:00Z| 0 |35.9|Kitchen| 21 | |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T10:00:00Z| 0 |36.1|Kitchen|22.7| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| [](#sort-data-with-the-newest-points-first) Sort data with the newest points first ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' ORDER BY time DESC ``` | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T10:00:00Z| 0 |36.1|Kitchen|22.7| |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T08:00:00Z| 0 |35.9|Kitchen| 21 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | --- # Quotation Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/quoting/ InfluxQL supports both single and double quotation marks, but they each function differently and serve different purposes. ## Single quotes Use single quotation marks (`'`) in the[string literal](/influxdb3/enterprise/reference/influxql/#strings) and[timestamp literal](/influxdb3/enterprise/reference/influxql/#dates--times) syntax. In the following example, the `WHERE` clause evaluates the value of the `room` tag. Tags only contain string values, so the right operand of the predicate expression should be a string literal. ```sql ... WHERE room = 'Kitchen' ``` ## Double quotes Use double quotation marks (`"`) to quote [identifiers](/influxdb3/enterprise/reference/influxql/#identifiers). Identifiers **must** be double-quoted in the following cases: * The identifier contains non-word characters. Word characters are defined as `[a-z,A-Z,0-9,_]`. * The identifier is case-sensitive. * The identifier contains [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords). * The identifier begins with a digit. Double-quoted identifiers can also: * Contain any Unicode character except for a new line. * Contain escaped `"` characters (for example: `\"`). * Include [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords). **While not always necessary, we recommend that you double quote identifiers.** > [!Important] > InfluxQL quoting guidelines differ from[line protocol quoting guidelines](/influxdb3/enterprise/reference/syntax/line-protocol/#quotes). --- # Regular expressions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/regular-expressions/ Regular expressions are a sequence of characters used to identify patterns in identifiers and string values. InfluxQL supports regular expressions in the following operations: * Identifying **fields** and **tags** to query in the[`SELECT` clause](/influxdb3/enterprise/reference/influxql/select/). * Identifying **measurements** to query in the[`FROM` clause](/influxdb3/enterprise/reference/influxql/select/#from-clause). * Testing **tag values** and **string field values** in the[`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/). * Identifying **tag keys** to group by in the[`GROUP BY` clause](/influxdb3/enterprise/reference/influxql/group-by/) > [!Important] > #### Query performance > > Regular expression comparisons are more computationally intensive than exact > string comparisons. Queries with regular expressions are not as performant > as those without. ## Regular expression syntax InfluxQL Regular expressions are surrounded by `/` characters and use the[Go regular expression syntax](http://golang.org/pkg/regexp/syntax/). ```sql /regular_expression/ ``` ### Regular expression flags Regular expression flags modify the pattern-matching behavior of the expression. InfluxQL supports the following regular expression flags: |Flag| Description | |----|-------------------------------------------------------------------------------| | i | case-insensitive | | m |multi-line mode: `^` and `$` match begin/end line in addition to begin/end text| | s | let `.` match `\n` | | U | ungreedy: swap meaning of `x*` and `x*?`, `x+` and `x+?`, etc. | Include regular expression flags at the beginning of your regular expression pattern enclosed in parentheses (`()`) and preceded by a question mark (`?`). ```sql /(?iU)foo*/ ``` ## Regular expression operators InfluxQL provides the following regular expression operators that test if a string operand matches a regular expression: * `=~`: Returns true if the string matches the regular expression * `!~`: Returns true if the string does not match the regular expression InfluxQL regular expression operators are used to test string column values in the [`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/). ## Regular expression examples The examples below use the following sample data sets: * [NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data) * [Get started home sensor data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data) [](#use-a-regular-expression-to-specify-field-keys-and-tag-keys-in-the-select-clause) Use a regular expression to specify field keys and tag keys in the SELECT clause ```sql SELECT /^t/ FROM weather ``` name: weather | time |temp\_avg|temp\_max|temp\_min| |--------------------|---------|---------|---------| |2020-01-01T00:00:00Z| 52 | 66 | 44 | |2020-01-01T00:00:00Z| 53 | 59 | 47 | |2020-01-01T00:00:00Z| 50 | 57 | 44 | |2020-01-02T00:00:00Z| 54 | 61 | 49 | |2020-01-02T00:00:00Z| 51 | 60 | 44 | |2020-01-02T00:00:00Z| 53 | 66 | 42 | | … | … | … | … | [](#use-a-regular-expression-to-specify-measurements-in-the-from-clause) Use a regular expression to specify measurements in the FROM clause ```sql SELECT /^t/ FROM /^[hw]/ ``` name: weather | time |temp|temp\_avg|temp\_max|temp\_min| |--------------------|----|---------|---------|---------| |2020-01-01T00:00:00Z| | 52 | 66 | 44 | |2020-01-01T00:00:00Z| | 53 | 59 | 47 | |2020-01-01T00:00:00Z| | 50 | 57 | 44 | |2020-01-02T00:00:00Z| | 54 | 61 | 49 | |2020-01-02T00:00:00Z| | 51 | 60 | 44 | |2020-01-02T00:00:00Z| | 53 | 66 | 42 | | … | … | … | … | … | [](#use-a-regular-expression-to-specify-tag-values-in-the-where-clause) Use a regular expression to specify tag values in the WHERE clause ```sql SELECT * FROM weather WHERE location !~ /^[S]/ ``` name: weather | time |location|precip|temp\_avg|temp\_max|temp\_min|wind\_avg| |--------------------|--------|------|---------|---------|---------|---------| |2020-01-01T00:00:00Z|Concord | 0 | 52 | 66 | 44 | 3.13 | |2020-01-01T00:00:00Z|Hayward | 0 | 50 | 57 | 44 | 2.24 | |2020-01-02T00:00:00Z|Concord | 0 | 53 | 66 | 42 | 3.13 | |2020-01-02T00:00:00Z|Hayward | 0 | 51 | 60 | 44 | 3.8 | |2020-01-03T00:00:00Z|Concord | 0 | 49 | 60 | 38 | 2.68 | |2020-01-03T00:00:00Z|Hayward | 0 | 50 | 62 | 41 | 3.13 | | … | … | … | … | … | … | … | [](#use-a-regular-expression-to-specify-a-tag-with-no-value-in-the-where-clause) Use a regular expression to specify a tag with no value in the WHERE clause ```sql SELECT * FROM home, weather WHERE location !~ /./ ``` name: weather | time |co |hum |location|precip| room |temp|temp\_avg|temp\_max|temp\_min|wind\_avg| |--------------------|---|----|--------|------|-----------|----|---------|---------|---------|---------| |2022-01-01T08:00:00Z| 0 |35.9| | | Kitchen | 21 | | | | | |2022-01-01T08:00:00Z| 0 |35.9| | |Living Room|21.1| | | | | |2022-01-01T09:00:00Z| 0 |36.2| | | Kitchen | 23 | | | | | |2022-01-01T09:00:00Z| 0 |35.9| | |Living Room|21.4| | | | | |2022-01-01T10:00:00Z| 0 |36.1| | | Kitchen |22.7| | | | | |2022-01-01T10:00:00Z| 0 | 36 | | |Living Room|21.8| | | | | | … | … | … | … | … | … | … | … | … | … | … | [](#use-a-regular-expression-to-specify-tag-keys-in-the-group-by-clause) Use a regular expression to specify tag keys in the GROUP BY clause ```sql SELECT MAX(precip) FROM weather GROUP BY /^l/ ``` name: weather tags: location=Concord | time |max | |--------------------|----| |2021-10-24T00:00:00Z|4.53| name: weather tags: location=Hayward | time |max | |--------------------|----| |2022-12-31T00:00:00Z|4.34| name: weather tags: location=San Francisco | time |max | |--------------------|----| |2021-10-24T00:00:00Z|4.02| | Flag | Description | | --- | --- | | Flag | Description | | i | case-insensitive | | m | multi-line mode: ^ and $ match begin/end line in addition to begin/end text | | s | let . match \n | | U | ungreedy: swap meaning of x* and x*? , x+ and x+? , etc. | | time | temp_avg | temp_max | temp_min | | --- | --- | --- | --- | | time | temp_avg | temp_max | temp_min | | 2020-01-01T00:00:00Z | 52 | 66 | 44 | | 2020-01-01T00:00:00Z | 53 | 59 | 47 | | 2020-01-01T00:00:00Z | 50 | 57 | 44 | | 2020-01-02T00:00:00Z | 54 | 61 | 49 | | 2020-01-02T00:00:00Z | 51 | 60 | 44 | | 2020-01-02T00:00:00Z | 53 | 66 | 42 | | … | … | … | … | | time | temp | temp_avg | temp_max | temp_min | | --- | --- | --- | --- | --- | | time | temp | temp_avg | temp_max | temp_min | | 2020-01-01T00:00:00Z | | 52 | 66 | 44 | | 2020-01-01T00:00:00Z | | 53 | 59 | 47 | | 2020-01-01T00:00:00Z | | 50 | 57 | 44 | | 2020-01-02T00:00:00Z | | 54 | 61 | 49 | | 2020-01-02T00:00:00Z | | 51 | 60 | 44 | | 2020-01-02T00:00:00Z | | 53 | 66 | 42 | | … | … | … | … | … | | time | location | precip | temp_avg | temp_max | temp_min | wind_avg | | --- | --- | --- | --- | --- | --- | --- | | time | location | precip | temp_avg | temp_max | temp_min | wind_avg | | 2020-01-01T00:00:00Z | Concord | 0 | 52 | 66 | 44 | 3.13 | | 2020-01-01T00:00:00Z | Hayward | 0 | 50 | 57 | 44 | 2.24 | | 2020-01-02T00:00:00Z | Concord | 0 | 53 | 66 | 42 | 3.13 | | 2020-01-02T00:00:00Z | Hayward | 0 | 51 | 60 | 44 | 3.8 | | 2020-01-03T00:00:00Z | Concord | 0 | 49 | 60 | 38 | 2.68 | | 2020-01-03T00:00:00Z | Hayward | 0 | 50 | 62 | 41 | 3.13 | | … | … | … | … | … | … | … | | time | co | hum | location | precip | room | temp | temp_avg | temp_max | temp_min | wind_avg | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | time | co | hum | location | precip | room | temp | temp_avg | temp_max | temp_min | wind_avg | | 2022-01-01T08:00:00Z | 0 | 35.9 | | | Kitchen | 21 | | | | | | 2022-01-01T08:00:00Z | 0 | 35.9 | | | Living Room | 21.1 | | | | | | 2022-01-01T09:00:00Z | 0 | 36.2 | | | Kitchen | 23 | | | | | | 2022-01-01T09:00:00Z | 0 | 35.9 | | | Living Room | 21.4 | | | | | | 2022-01-01T10:00:00Z | 0 | 36.1 | | | Kitchen | 22.7 | | | | | | 2022-01-01T10:00:00Z | 0 | 36 | | | Living Room | 21.8 | | | | | | … | … | … | … | … | … | … | … | … | … | … | | time | max | | --- | --- | | time | max | | 2021-10-24T00:00:00Z | 4.53 | | time | max | | --- | --- | | time | max | | 2022-12-31T00:00:00Z | 4.34 | | time | max | | --- | --- | | time | max | | 2021-10-24T00:00:00Z | 4.02 | --- # SELECT statement Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/select/ Use the `SELECT` statement to query data from one or more[measurements](/influxdb3/enterprise/reference/glossary/#measurement). The `SELECT` statement **requires** a [`SELECT` clause](#select-clause) and a[`FROM` clause](#from-clause). ## Syntax ```sql SELECT field_expression[, ..., field_expression_n[, tag_expression[, ..., tag_expression_n]]] FROM measurement_expression[, ..., measurement_expression_n] ``` ### SELECT clause The `SELECT` clause supports several formats for identifying data to query. It requires one or more **field expressions** and optional **tag expressions**. * **field\_expression**: Expression to identify one or more fields to return in query results. Can be a [field key](/influxdb3/enterprise/reference/glossary/#field-key), constant, [regular expression](/influxdb3/enterprise/reference/influxql/regular-expressions/),[wildcard (`*`)](#wildcard-expressions), or[function expression](/influxdb3/enterprise/reference/influxql/functions/) and any combination of arithmetic operators. * **tag\_expression**: Expression to identify one or more tags to return in query results. Can be a [tag key](/influxdb3/enterprise/reference/glossary/#tag-key) or constant. #### Select clause behaviors * `SELECT field_key` - Returns a specific field. * `SELECT field_key1, field_key2` - Returns two specific fields. * `SELECT field_key, tag_key` - Returns a specific field and tag. * `SELECT *` - Returns all [fields](/influxdb3/enterprise/reference/glossary/#field)and [tags](/influxdb3/enterprise/reference/glossary/#tag).*See [Wildcard expressions](#wildcard-expressions).* * `SELECT /^[t]/` - Returns all [fields](/influxdb3/enterprise/reference/glossary/#field)and [tags](/influxdb3/enterprise/reference/glossary/#tag) with keys that match the regular expression. At least one field key must match the regular expression. If no field keys match the regular expression, no results are returned. ### FROM clause The `FROM` clause specifies the[measurement](/influxdb3/enterprise/reference/glossary/#measurement) or[subquery](/influxdb3/enterprise/reference/influxql/subqueries/) to query. It requires one or more comma-delimited[measurement expressions](#measurement_expression) or [subqueries](#subquery). #### measurement\_expression A measurement expression identifies a measurement to query. It can be a measurement name, fully-qualified measurement, constant, or a [regular expression](/influxdb3/enterprise/reference/influxql/regular-expressions/). * **Measurement name**: When using just the measurement name, InfluxQL assumes the default retention policy of the database specified in the query request. ``` FROM measurement ``` * **Fully-qualified measurement**: A fully qualified measurement includes a database name, retention policy name, and measurement name, each separated by a period (`.`). If the retention policy is not specified, InfluxQL uses the default retention policy for the specified database. ```sql FROM database.retention_policy.measurement -- Fully-qualified measurement with default retention policy FROM database..measurement ``` > [!Important] > #### InfluxQL retention policies > > In InfluxDB 3 Enterprise, **retention policies** are not part of the data model > like they are in InfluxDB 1.x. > Each InfluxDB 3 Enterprise database has a **retention period** which defines the > maximum age of data to retain in the database. To use fully-qualified > measurements in InfluxQL queries, use the following naming convention when[creating a database](/influxdb3/enterprise/admin/databases/create/): > > ``` > database_name/retention_policy > ``` #### Subquery An InfluxQL subquery is a query nested in the `FROM` clause of an InfluxQL query. The outer query queries results returned by the inner query (subquery). For more information, see [InfluxQL subqueries](/influxdb3/enterprise/reference/influxql/subqueries/). ## Notable SELECT statement behaviors ### Must query at least one field A query requires at least one [field key](/influxdb3/enterprise/reference/glossary/#field-key)in the `SELECT` clause to return data. If the `SELECT` clause includes only [tag keys](/influxdb3/enterprise/reference/glossary/#tag-key), the query returns an empty result. When using regular expressions in the `SELECT` clause, if regular expression matches only tag keys and no field keys, the query returns an empty result. To return data associated with tag keys, include at least one field key in the`SELECT` clause. ### Wildcard expressions When using a wildcard expression (`*`) in the `SELECT` clause, the query returns all tags and fields. If a [function](/influxdb3/enterprise/reference/influxql/functions/) is applied to a wildcard expression, the query returns all *fields* with the function applied, but does not return *tags* unless they are included in the `SELECT` clause. ### Cannot include both aggregate and non-aggregate field expressions The `SELECT` statement cannot include an aggregate field expression (one that uses an [aggregate](/influxdb3/enterprise/reference/influxql/functions/aggregates/)or [selector](/influxdb3/enterprise/reference/influxql/functions/aggregates/)function) **and** a non-aggregate field expression. For example, in the following query, an aggregate function is applied to one field, but not the other: ```sql SELECT mean(temp), hum FROM home ``` This query returns an error. For more information, see [error about mixing aggregate and non-aggregate queries](/enterprise_influxdb/v1/troubleshooting/errors/#error-parsing-query-mixing-aggregate-and-non-aggregate-queries-is-not-supported). ## Data types and casting operations The [`SELECT` clause](#select-clause) supports specifying a[field’s](/influxdb3/enterprise/reference/glossary/#field) type and basic casting operations with the `::` syntax. ```sql SELECT field_expression::type FROM measurement_expression ``` The `::` syntax allows users to perform basic cast operations in queries. Currently, InfluxQL supports casting *numeric* [field values](/influxdb3/enterprise/reference/glossary/#field-value)to other numeric types. Casting to an **identifier type** acts as a filter on results and returns only columns of that specific identifier type along with the `time` column. ##### Numeric types * `float` * `integer` * `unsigned` ##### Non-numeric types * `string` * `boolean` ##### Identifier types * `field` * `tag` > [!Important] > InfluxQL returns no data if the query attempts to cast a numeric value to a > non-numeric type and vice versa. When casting a float value to an integer or unsigned integer, the float value is truncated at the decimal point. No rounding is performed. ## SELECT statement examples The examples below use the following sample data sets: * [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data) * [NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data) [](#select-all-fields-and-tags-from-a-measurement) Select all fields and tags from a measurement ```sql SELECT * FROM home ``` Name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T08:00:00Z| 0 |35.9| Kitchen | 21 | |2022-01-01T08:00:00Z| 0 |35.9|Living Room|21.1| |2022-01-01T09:00:00Z| 0 |36.2| Kitchen | 23 | |2022-01-01T09:00:00Z| 0 |35.9|Living Room|21.4| |2022-01-01T10:00:00Z| 0 |36.1| Kitchen |22.7| |2022-01-01T10:00:00Z| 0 | 36 |Living Room|21.8| | … | … | … | … | … | [](#select-specific-tags-and-fields-from-a-measurement) Select specific tags and fields from a measurement ```sql SELECT temp, hum, room FROM home ``` Name: home | time |temp|hum | room | |--------------------|----|----|-----------| |2022-01-01T08:00:00Z| 21 |35.9| Kitchen | |2022-01-01T08:00:00Z|21.1|35.9|Living Room| |2022-01-01T09:00:00Z| 23 |36.2| Kitchen | |2022-01-01T09:00:00Z|21.4|35.9|Living Room| |2022-01-01T10:00:00Z|22.7|36.1| Kitchen | |2022-01-01T10:00:00Z|21.8| 36 |Living Room| | … | … | … | … | [](#select-all-fields-from-a-measurement) Select all fields from a measurement ```sql SELECT *::field FROM home ``` Name: home | time |co |hum |temp| |--------------------|---|----|----| |2022-01-01T08:00:00Z| 0 |35.9| 21 | |2022-01-01T08:00:00Z| 0 |35.9|21.1| |2022-01-01T09:00:00Z| 0 |36.2| 23 | |2022-01-01T09:00:00Z| 0 |35.9|21.4| |2022-01-01T10:00:00Z| 0 |36.1|22.7| |2022-01-01T10:00:00Z| 0 | 36 |21.8| | … | … | … | … | [](#select-a-field-from-a-measurement-and-perform-basic-arithmetic) Select a field from a measurement and perform basic arithmetic ```sql SELECT (temp * (9 / 5)) + 32 FROM home ``` Name: home | time | temp | |--------------------|-----------------| |2022-01-01T08:00:00Z|69.80000000000001| |2022-01-01T08:00:00Z| 69.98 | |2022-01-01T09:00:00Z| 73.4 | |2022-01-01T09:00:00Z| 70.52 | |2022-01-01T10:00:00Z| 72.86 | |2022-01-01T10:00:00Z|71.24000000000001| | … | … | > [!Note] > **Note:** InfluxDB follows the standard order of operations. > See [InfluxQL mathematical operators](/influxdb3/enterprise/reference/influxql/math-operators/)for more on supported operators. [](#select-all-data-from-more-than-one-measurement) Select all data from more than one measurement ```sql SELECT * FROM home, weather ``` Name: weather | time |co |hum| location |precip|room|temp|temp\_avg|temp\_max|temp\_min|wind\_avg| |--------------------|---|---|-------------|------|----|----|---------|---------|---------|---------| |2020-01-01T00:00:00Z| | | Concord | 0 | | | 52 | 66 | 44 | 3.13 | |2020-01-01T00:00:00Z| | |San Francisco| 0 | | | 53 | 59 | 47 | 14.32 | |2020-01-01T00:00:00Z| | | Hayward | 0 | | | 50 | 57 | 44 | 2.24 | |2020-01-02T00:00:00Z| | |San Francisco| 0 | | | 54 | 61 | 49 | 5.82 | |2020-01-02T00:00:00Z| | | Hayward | 0 | | | 51 | 60 | 44 | 3.8 | |2020-01-02T00:00:00Z| | | Concord | 0 | | | 53 | 66 | 42 | 3.13 | | … | … | … | … | … | … | … | … | … | … | … | [](#select-all-data-from-a-fully-qualified-measurement-with-default-retention-policy) Select all data from a fully-qualified measurement (with default retention policy) ```sql SELECT * FROM "get-started"..home ``` Name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T08:00:00Z| 0 |35.9| Kitchen | 21 | |2022-01-01T08:00:00Z| 0 |35.9|Living Room|21.1| |2022-01-01T09:00:00Z| 0 |36.2| Kitchen | 23 | |2022-01-01T09:00:00Z| 0 |35.9|Living Room|21.4| |2022-01-01T10:00:00Z| 0 |36.1| Kitchen |22.7| |2022-01-01T10:00:00Z| 0 | 36 |Living Room|21.8| | … | … | … | … | … | ### Type-casting examples [](#cast-an-integer-field-to-a-float) Cast an integer field to a float ```sql SELECT co::float FROM home ``` [](#cast-a-float-field-to-an-integer) Cast a float field to an integer ```sql SELECT temp::integer FROM home ``` [](#cast-a-float-field-to-an-unsigned-integer) Cast a float field to an unsigned integer ```sql SELECT temp::unsigned FROM home ``` | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T08:00:00Z | 0 | 35.9 | Living Room | 21.1 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T09:00:00Z | 0 | 35.9 | Living Room | 21.4 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T10:00:00Z | 0 | 36 | Living Room | 21.8 | | … | … | … | … | … | | time | temp | hum | room | | --- | --- | --- | --- | | time | temp | hum | room | | 2022-01-01T08:00:00Z | 21 | 35.9 | Kitchen | | 2022-01-01T08:00:00Z | 21.1 | 35.9 | Living Room | | 2022-01-01T09:00:00Z | 23 | 36.2 | Kitchen | | 2022-01-01T09:00:00Z | 21.4 | 35.9 | Living Room | | 2022-01-01T10:00:00Z | 22.7 | 36.1 | Kitchen | | 2022-01-01T10:00:00Z | 21.8 | 36 | Living Room | | … | … | … | … | | time | co | hum | temp | | --- | --- | --- | --- | | time | co | hum | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | 21 | | 2022-01-01T08:00:00Z | 0 | 35.9 | 21.1 | | 2022-01-01T09:00:00Z | 0 | 36.2 | 23 | | 2022-01-01T09:00:00Z | 0 | 35.9 | 21.4 | | 2022-01-01T10:00:00Z | 0 | 36.1 | 22.7 | | 2022-01-01T10:00:00Z | 0 | 36 | 21.8 | | … | … | … | … | | time | temp | | --- | --- | | time | temp | | 2022-01-01T08:00:00Z | 69.80000000000001 | | 2022-01-01T08:00:00Z | 69.98 | | 2022-01-01T09:00:00Z | 73.4 | | 2022-01-01T09:00:00Z | 70.52 | | 2022-01-01T10:00:00Z | 72.86 | | 2022-01-01T10:00:00Z | 71.24000000000001 | | … | … | | time | co | hum | location | precip | room | temp | temp_avg | temp_max | temp_min | wind_avg | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | time | co | hum | location | precip | room | temp | temp_avg | temp_max | temp_min | wind_avg | | 2020-01-01T00:00:00Z | | | Concord | 0 | | | 52 | 66 | 44 | 3.13 | | 2020-01-01T00:00:00Z | | | San Francisco | 0 | | | 53 | 59 | 47 | 14.32 | | 2020-01-01T00:00:00Z | | | Hayward | 0 | | | 50 | 57 | 44 | 2.24 | | 2020-01-02T00:00:00Z | | | San Francisco | 0 | | | 54 | 61 | 49 | 5.82 | | 2020-01-02T00:00:00Z | | | Hayward | 0 | | | 51 | 60 | 44 | 3.8 | | 2020-01-02T00:00:00Z | | | Concord | 0 | | | 53 | 66 | 42 | 3.13 | | … | … | … | … | … | … | … | … | … | … | … | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T08:00:00Z | 0 | 35.9 | Living Room | 21.1 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T09:00:00Z | 0 | 35.9 | Living Room | 21.4 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T10:00:00Z | 0 | 36 | Living Room | 21.8 | | … | … | … | … | … | --- # InfluxQL SHOW statements Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/show/ Use InfluxQL `SHOW` statements to query schema information from a database. ## SHOW RETENTION POLICIES Use the `SHOW RETENTION POLICIES` statement to list retention policies associated with a database. Each database has a single retention policy–`autogen`. > [!Note] > In InfluxDB 3 Enterprise, retention policies are not part of the > data structure but are expected by InfluxQL and used in fully-qualified measurements > in the [`FROM` clause](/influxdb3/enterprise/reference/influxql/select/#from-clause). > The data returned for each retention policy does not represent the actual > retention-related attributes of the database. The values are placeholder values > meant only for InfluxQL feature parity. > > For information about database data retention, see[List databases](/influxdb3/enterprise/admin/databases/list/). ```sql SHOW RETENTION POLICIES [ON ] ``` #### Examples ```sql -- Show retention policies in the database specified in the query request SHOW RETENTION POLICIES -- Show retention policies in a specific database SHOW RETENTION POLICIES ON "example-database" ``` ## SHOW MEASUREMENTS Use the `SHOW MEASUREMENTS` statement to list measurements in a database. ```sql SHOW MEASUREMENTS [with_measurement_clause] [where_clause] [limit_clause] [offset_clause] ``` #### Examples ```sql -- Show all measurements SHOW MEASUREMENTS -- Show measurements where region tag = 'uswest' AND host tag = 'serverA' SHOW MEASUREMENTS WHERE "region" = 'uswest' AND "host" = 'serverA' -- Show measurements that start with 'h2o' SHOW MEASUREMENTS WITH MEASUREMENT =~ /h2o.*/ ``` ## SHOW FIELD KEYS Use the `SHOW FIELD KEYS` statement to list all field keys in a measurement. ```sql SHOW FIELD KEYS [from_clause] ``` #### Examples ```sql -- Show field keys and field value data types from all measurements SHOW FIELD KEYS -- Show field keys and field value data types from specified measurement SHOW FIELD KEYS FROM "cpu" ``` ## SHOW TAG KEYS Use the `SHOW TAG KEYS` statement to list tag keys in a measurement. ```sql SHOW TAG KEYS [from_clause] [where_clause] [limit_clause] [offset_clause] ``` #### Examples ```sql -- Show all tag keys SHOW TAG KEYS -- Show all tag keys from the cpu measurement SHOW TAG KEYS FROM "cpu" -- Show all tag keys from the cpu measurement where the region key = 'uswest' SHOW TAG KEYS FROM "cpu" WHERE "region" = 'uswest' -- Show all tag keys where the host key = 'serverA' SHOW TAG KEYS WHERE "host" = 'serverA' ``` ## SHOW TAG VALUES Use the `SHOW TAG VALUES` statement to list values of specified tags in a database. ```sql SHOW TAG VALUES [from_clause] WITH KEY = [where_clause] [limit_clause] [offset_clause] ``` By default, the `SHOW TAG VALUES` statement only returns unique tag values from**the last day**. To modify the time range, include a[`WHERE` clause with a time-based predicate](/influxdb3/enterprise/reference/influxql/where/#time-ranges). > [!Important] > #### Include a FROM clause > > We strongly recommend including a `FROM` clause with the `SHOW TAG VALUES`statement that specifies 1-50 tables to query. > Without a `FROM` clause, the InfluxDB query engine must read data from all > tables and return unique tag values from each. > > Depending on the number of tables in your database and the number of unique tag > values in each table, excluding a `FROM` clause can result in poor query performance, > query timeouts, or unnecessary resource allocation that may affect other queries. #### Examples ```sql -- Show tag values from the cpu measurement for the region tag SHOW TAG VALUES FROM "cpu" WITH KEY = "region" -- Show tag values from the cpu measurement for the region tag for a custom time range SHOW TAG VALUES FROM "cpu" WITH KEY = "region" WHERE time > -7d -- Show tag values from multiple measurements for the region tag SHOW TAG VALUES FROM "cpu", "memory", "disk" WITH KEY = "region" -- Show tag values from the cpu measurement for all tag keys that do not include the letter c SHOW TAG VALUES FROM "cpu" WITH KEY !~ /.*c.*/ -- Show tag values from the cpu measurement for region & host tag keys where service = 'redis' SHOW TAG VALUES FROM "cpu" WITH KEY IN ("region", "host") WHERE "service" = 'redis' ``` #### Related * [Explore your schema with InfluxQL](/influxdb3/enterprise/query-data/influxql/explore-schema/) --- # InfluxQL subqueries Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/subqueries/ An InfluxQL subquery is a query nested in the `FROM` clause of an InfluxQL query. The outer query queries results returned by the inner query (subquery). > [!Important] > InfluxQL does not support a `HAVING` clause, however InfluxQL subqueries offer > functionality similar to the [SQL `HAVING` clause](/influxdb3/enterprise/reference/sql/having/). ## Syntax ```sql SELECT_clause FROM ( SELECT_statement ) [...] ``` When using subqueries, InfluxQL **performs the inner query first**, then performs the outer query. The outer query requires a [`SELECT` clause](/influxdb3/enterprise/reference/influxql/select/#select-clause)and a [`FROM` clause](/influxdb3/enterprise/reference/influxql/select/#from-clause). The inner query is enclosed in parentheses in the outer query’s `FROM` clause. InfluxQL supports multiple nested subqueries: ```sql SELECT_clause FROM ( SELECT_clause FROM ( SELECT_statement ) [...] ) [...] ``` ## Examples > [!Note] > #### Sample data > > The examples below use the following sample data sets: > > * [Get started home sensor data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data) > * [Random numbers sample data](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data) [](#apply-an-aggregate-function-to-an-aggregated-result-set) Apply an aggregate function to an aggregated result set ```sql SELECT SUM(max) FROM ( SELECT MAX(temp) FROM home GROUP BY room ) ``` Table: home | time |sum | |--------------------|----| |1970-01-01T00:00:00Z|46.1| [](#calculate-the-average-difference-between-two-fields) Calculate the average difference between two fields ```sql SELECT MEAN(difference) FROM ( SELECT a - b AS difference FROM numbers ) ``` Table: numbers | time | mean | |--------------------|---------------------| |1970-01-01T00:00:00Z|\-0.03629771779732732| [](#filter-aggregate-values-based-on-a-threshold) Filter aggregate values based on a threshold ```sql SELECT co_change FROM ( SELECT SPREAD(co) AS co_change FROM home GROUP BY room, time(2h) ) WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' AND co_change >= 4 ``` Table: home | time |co\_chnage| |--------------------|----------| |2022-01-01T18:00:00Z| 4 | |2022-01-01T18:00:00Z| 5 | [](#perform-additional-aggregate-operations-on-aggregate-values) Perform additional aggregate operations on aggregate values ```sql SELECT SUM(co_derivative) AS sum_derivative FROM ( SELECT DERIVATIVE(MEAN(co)) AS co_derivative FROM home GROUP BY time(12m), room ) WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY room ``` Table: home | time | room |sum\_derivative| |--------------------|-----------|---------------| |1970-01-01T00:00:00Z| Kitchen | 5.2 | |1970-01-01T00:00:00Z|Living Room| 3.4 | ## Notable subquery behaviors ### Apply time bounds to the outer query to improve performance To improve the performance of InfluxQL queries that use subqueries and a specified time range, apply the `WHERE` clause with time-based predicates to the outer query rather than the inner query. For example–the following queries return the same results, but **the query with time-based predicate on the outer query is more performant than the query with time-based predicate on the inner query**: #### Time bounds on the outer query ```sql SELECT inner_value AS value FROM ( SELECT raw_value as inner_value ) WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' ``` #### Time bounds on the inner query ```sql SELECT inner_value AS value FROM ( SELECT raw_value as inner_value WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' ) ``` ### Cannot use multiple SELECT statements in a subquery InfluxQL does not support multiple[`SELECT` statements](/influxdb3/enterprise/reference/influxql/select/)per subquery: ```sql SELECT_clause FROM (SELECT_statement; SELECT_statement) [...] ``` However, InfluxQL does support multiple nested subqueries per outer query: ```sql SELECT_clause FROM ( SELECT_clause FROM ( SELECT_statement ) [...] ) [...] ------------------ ---------------- Subquery 1 Subquery 2 ``` | time | sum | | --- | --- | | time | sum | | 1970-01-01T00:00:00Z | 46.1 | | time | mean | | --- | --- | | time | mean | | 1970-01-01T00:00:00Z | -0.03629771779732732 | | time | co_chnage | | --- | --- | | time | co_chnage | | 2022-01-01T18:00:00Z | 4 | | 2022-01-01T18:00:00Z | 5 | | time | room | sum_derivative | | --- | --- | --- | | time | room | sum_derivative | | 1970-01-01T00:00:00Z | Kitchen | 5.2 | | 1970-01-01T00:00:00Z | Living Room | 3.4 | --- # Time and time zones Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/time-and-timezone/ InfluxQL is designed for working with time series data and includes features specifically for working with time. You can review the following ways to work with time and timestamps in your InfluxQL queries: ## Time syntax InfluxQL supports the following timestamp literal syntaxes: ```sql '2006-01-02T15:04:05.00Z' -- RFC3339 date-time string '2006-01-02 15:04:05.00' -- RFC3339-like date-time string 1136189045000000000 -- Unix nanosecond epoch time 1136189045s -- Unix epoch time ``` * **RFC3339 date-time string**:[String literal](/influxdb3/enterprise/reference/influxql/#strings) using the RFC3339 timestamp format, `YYYY-MM-DDTHH:MM:SS.nnnnnnnnnZ`. * **RFC3339-like date-time string**:[String literal](/influxdb3/enterprise/reference/influxql/#strings) using the RFC3339-like timestamp format, `YYYY-MM-DD HH:MM:SS.nnnnnnnnn`. * **Unix nanosecond epoch time**:[Integer](/influxdb3/enterprise/reference/influxql/#integers) that represents the number of nanoseconds elapsed since the[Unix epoch](/influxdb3/enterprise/reference/glossary/#unix-epoch). * **Unix epoch time**:[Duration literal](/influxdb3/enterprise/reference/influxql/#durations)that represents the number of specified time units elapsed since the[Unix epoch](/influxdb3/enterprise/reference/glossary/#unix-epoch).*[View supported duration units](/influxdb3/enterprise/reference/influxql/#durations)*. ##### Supported timestamp values | | RFC3339 |Unix nanosecond time | |-----------|------------------------------|---------------------| |**Maximum**|2262-04-11T23:47:16.854775807Z| 9223372036854775807 | |**Minimum**|1677-09-21T00:12:43.145224193Z|\-9223372036854775807| ### Add and subtract time values Timestamp values support addition and subtraction operations with[duration literals](/influxdb3/enterprise/reference/influxql/#durations). Add (`+`) or subtract (`-`) a duration to or from a timestamp to return an updated timestamp. ```sql '2023-01-01T00:00:00Z' + 2h -- Resolves to 2023-01-01T02:00:00Z '2023-01-01 00:00:00' - 20h -- Resolves to 2022-12-31T04:00:00Z 1672531200000000000 + 1y -- Resolves to 2024-01-01T00:00:00Z ``` > [!Important] > InfluxQL requires a whitespace between the `+` operators `-` and the duration literal. ## Query time range To specify the time range of a query, use conditional expressions in the[`WHERE` clause](/influxdb3/enterprise/reference/influxql/where/) that compare the value of the `time` column to an absolute timestamp or a relative timestamp. * **Absolute time range**: Define query time bounds with timestamp literals ``` WHERE time >= '2023-01-01T00:00:00Z' AND time <= '2023-07-01T00:00:00Z' WHERE time >= '2023-01-01 00:00:00' AND time <= '2023-07-01 00:00:00' WHERE time >= 1672531200000000000 AND time <= 1688169600000000000 WHERE time >= 1672531200s and time <= 1688169600000ms ``` * **Relative time range**: Define query time bounds with a duration literal added to or subtracted from timestamp literals. > [!Tip] > Use `now()` to return the current system time (UTC). ``` -- Query data from the last day WHERE time >= now() - 1d -- Query data from the previous week WHERE time >= now() - 1w AND time <= now() - 2w -- Query data relative to a specific time WHERE time >= '2023-01-01' - 1w AND time <= '2023-01-01' + 1w ``` ### Supported operators Conditional expressions with time operands support the following comparison operators: |Operator| Meaning | |--------|------------------------| | `=` | equal to | | `<>` | not equal to | | `!=` | not equal to | | `>` | greater than | | `>=` |greater than or equal to| | `<` | less than | | `<=` | less than or equal to | > [!Important] > InfluxQL supports the `AND` logical operator to define query time bounds, but > does not support using the `OR` logical operator to query multiple time ranges. ## Query examples The following examples use the[Home sensor sample dataset](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). [](#specify-a-time-range-with-rfc3339-date-time-strings) Specify a time range with RFC3339 date-time strings ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T08:00:00Z| 0 |35.9|Kitchen| 21 | |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T10:00:00Z| 0 |36.1|Kitchen|22.7| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| [](#specify-a-time-range-with-rfc3339-like-date-time-strings) Specify a time range with RFC3339-like date-time strings ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01 08:00:00' AND time <= '2022-01-01 12:00:00' ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T08:00:00Z| 0 |35.9|Kitchen| 21 | |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T10:00:00Z| 0 |36.1|Kitchen|22.7| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| [](#specify-a-time-range-with-nanosecond-epoch-timestamps) Specify a time range with nanosecond epoch timestamps ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= 1641024000000000000 AND time <= 1641038400000000000 ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T08:00:00Z| 0 |35.9|Kitchen| 21 | |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T10:00:00Z| 0 |36.1|Kitchen|22.7| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| [](#specify-a-time-range-with-second-precision-epoch-timestamps) Specify a time range with second-precision epoch timestamps ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= 1641024000s AND time <= 1641038400s ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T08:00:00Z| 0 |35.9|Kitchen| 21 | |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T10:00:00Z| 0 |36.1|Kitchen|22.7| |2022-01-01T11:00:00Z| 0 | 36 |Kitchen|22.4| |2022-01-01T12:00:00Z| 0 | 36 |Kitchen|22.5| [](#specify-a-time-range-relative-to-a-timestamp) Specify a time range relative to a timestamp ```sql SELECT * FROM home WHERE time >= '2022-01-01T20:00:00Z' - 2h ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T18:00:00Z|18 |36.9| Kitchen |23.3| |2022-01-01T18:00:00Z| 9 |36.2|Living Room|22.8| |2022-01-01T19:00:00Z|22 |36.6| Kitchen |23.1| |2022-01-01T19:00:00Z|14 |36.3|Living Room|22.5| |2022-01-01T20:00:00Z|26 |36.5| Kitchen |22.7| |2022-01-01T20:00:00Z|17 |36.4|Living Room|22.2| [](#specify-a-time-range-relative-to-now) Specify a time range relative to now ```sql SELECT * FROM home WHERE time >= now() - 2h ``` ## Time zone clause By default, InfluxDB stores and returns timestamps in UTC. Use the time zone clause and the `tz()` function to apply a time zone offset to UTC times and return timestamps in the specified time zone including any applicable seasonal offset such as Daylight Savings Time (DST) or British Summer Time (BST). ```sql SELECT_clause FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] [LIMIT_clause] [OFFSET_clause] [SLIMIT_clause] [SOFFSET_clause] tz('time_zone') ``` * **time\_zone**: Time zone string literal to adjust times to. Uses time zone names defined in the[Internet Assigned Numbers Authority time zone database](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List). ### Time zone example [](#return-the-utc-offset-for-chicagos-time-zone) Return the UTC offset for Chicago’s time zone The following example uses the[Home sensor sample dataset](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ```sql SELECT * FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' tz('America/Chicago') ``` name: home | time |co |hum | room |temp| |-------------------------|---|----|-------|----| |2022-01-01T02:00:00-06:00| 0 |35.9|Kitchen| 21 | |2022-01-01T03:00:00-06:00| 0 |36.2|Kitchen| 23 | |2022-01-01T04:00:00-06:00| 0 |36.1|Kitchen|22.7| |2022-01-01T05:00:00-06:00| 0 | 36 |Kitchen|22.4| |2022-01-01T06:00:00-06:00| 0 | 36 |Kitchen|22.5| ## Notable behaviors ### Cannot query multiple time ranges InfluxDB does not support using `OR` in the `WHERE` clause to query multiple time ranges. For example, the following query returns no results: ```sql SELECT * FROM home WHERE (time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z') OR (time >= '2022-01-01T18:00:00Z' AND time <= '2022-01-01T20:00:00Z') ``` ### Querying future data with a `GROUP BY time()` clause Queries that do not specify time bounds in the `WHERE` clause and do not include a `GROUP BY time()` clause use the [minimum and maximum timestamps](#supported-timestamp-values)as the default time range. If the query includes a `GROUP BY time()` clause, the default time range is between `1677-09-21T00:12:43.145224193Z` and[`now()`](/influxdb3/enterprise/reference/influxql/functions/date-time/#now). To query data with timestamps that occur in the future (after `now()`), provide an explicit upper bound in the `WHERE` clause. ### Cannot use parameters for durations Currently, InfluxDB doesn’t support using parameters for durations in[parameterized queries](/influxdb3/enterprise/query-data/influxql/parameterized-queries/). | | RFC3339 | Unix nanosecond time | | --- | --- | --- | | | RFC3339 | Unix nanosecond time | | Maximum | 2262-04-11T23:47:16.854775807Z | 9223372036854775807 | | Minimum | 1677-09-21T00:12:43.145224193Z | -9223372036854775807 | | Operator | Meaning | | --- | --- | | Operator | Meaning | | = | equal to | | <> | not equal to | | != | not equal to | | > | greater than | | >= | greater than or equal to | | < | less than | | <= | less than or equal to | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T11:00:00Z | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T12:00:00Z | 0 | 36 | Kitchen | 22.5 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T18:00:00Z | 18 | 36.9 | Kitchen | 23.3 | | 2022-01-01T18:00:00Z | 9 | 36.2 | Living Room | 22.8 | | 2022-01-01T19:00:00Z | 22 | 36.6 | Kitchen | 23.1 | | 2022-01-01T19:00:00Z | 14 | 36.3 | Living Room | 22.5 | | 2022-01-01T20:00:00Z | 26 | 36.5 | Kitchen | 22.7 | | 2022-01-01T20:00:00Z | 17 | 36.4 | Living Room | 22.2 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T02:00:00-06:00 | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T03:00:00-06:00 | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T04:00:00-06:00 | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T05:00:00-06:00 | 0 | 36 | Kitchen | 22.4 | | 2022-01-01T06:00:00-06:00 | 0 | 36 | Kitchen | 22.5 | --- # WHERE clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/influxql/where/ Use the `WHERE` clause to filter data based on[field values](/influxdb3/enterprise/reference/glossary/#field),[tag values](/influxdb3/enterprise/reference/glossary/#tag), and[timestamps](/influxdb3/enterprise/reference/glossary/#timestamp). ## Syntax ```sql SELECT_clause FROM_clause WHERE [(AND|OR) [...]] ``` * **conditional\_expression**: Comparison between two operands that evaluates to`true` or `false`. Comparison logic is determined by[operators](#operators) used in the expression. These expressions can operate on InfluxDB fields, tags, and timestamps. Use logical operators (`AND`, `OR`) to chain multiple conditional expressions together. ## Operators Operators evaluate the relationship between two operands and return`true` or `false`. ### Comparison operators |Operator| Meaning |Supported data types| |--------|----------------------------------|--------------------| | `=` | Equal to | all | | `<>` | Not equal to | all | | `!=` | Not equal to | all | | `>` | Greater than | numeric, timestamp | | `>=` | Greater than or equal to | numeric, timestamp | | `<` | Less than | numeric, timestamp | | `<=` | Less than or equal to | numeric, timestamp | | `=~` | Matches a regular expression | strings | | `!~` |Doesn’t match a regular expression| strings | ### Logical operators |Operator| Meaning | |--------|-----------------------------------------------------------------------| | `AND` |Returns `true` if both operands are `true`. Otherwise, returns `false`.| | `OR` | Returns `true` if any operand is `true`. Otherwise, returns `false`. | ## Time ranges Use the `WHERE` clause to specify a time range to query. If a time range isn’t specified in the `WHERE` clause, the [default time range](/influxdb3/enterprise/reference/influxql/#default-time-range) is used. Timestamps are stored in the `time` column. Use comparison operators to compare the value of the `time` column to a timestamp literal, integer (Unix nanosecond timestamp), or [expression](/influxdb3/enterprise/reference/glossary/#expression). [Timestamp](#timestamp)[Integer](#integer)[Expression](#expression) ```sql WHERE time >= '2023-01-01T00:00:00Z' AND time < '2023-07-01T00:00:00Z' ``` ```sql WHERE time >= 1672531200000000000 AND time < 1688169600000000000 ``` ```sql WHERE time >= now() - 1d AND time < now() ``` See [Time syntax](/influxdb3/enterprise/reference/influxql/time-and-timezone/#time-syntax)for information on how to specify alternative time ranges in the `WHERE` clause. > [!Important] > InfluxQL [does not support querying multiple time ranges](#cannot-query-multiple-time-ranges). ## Regular expressions Regular expressions can be used to evaluate *string* values in the `WHERE` clause using regular expression comparison operators: * `=~`: Matches a regular expression * `!~`: Doesn’t match a regular expression ```sql SELECT * FROM home WHERE room =~ /^K/ ``` For more information about InfluxQL regular expression syntax, see[InfluxQL regular expressions](/influxdb3/enterprise/reference/influxql/regular-expressions/). ## WHERE clause examples The following examples use the[Home sensor sample dataset](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). [](#select-data-with-a-specific-tag-value) Select data with a specific tag value ```sql SELECT * FROM home WHERE room = 'Living Room' ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T08:00:00Z| 0 |35.9|Living Room|21.1| |2022-01-01T09:00:00Z| 0 |35.9|Living Room|21.4| |2022-01-01T10:00:00Z| 0 | 36 |Living Room|21.8| |2022-01-01T11:00:00Z| 0 | 36 |Living Room|22.2| |2022-01-01T12:00:00Z| 0 |35.9|Living Room|22.2| | … | … | … | … | … | [](#select-data-from-a-specific-time-range) Select data from a specific time range ```sql SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T08:00:00Z| 0 |35.9| Kitchen | 21 | |2022-01-01T08:00:00Z| 0 |35.9|Living Room|21.1| |2022-01-01T09:00:00Z| 0 |36.2| Kitchen | 23 | |2022-01-01T09:00:00Z| 0 |35.9|Living Room|21.4| |2022-01-01T10:00:00Z| 0 |36.1| Kitchen |22.7| |2022-01-01T10:00:00Z| 0 | 36 |Living Room|21.8| [](#select-data-from-a-relative-time-range) Select data from a relative time range ```sql SELECT * FROM home WHERE time >= '2022-01-01T20:00:00Z' - 2h ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-----------|----| |2022-01-01T18:00:00Z|18 |36.9| Kitchen |23.3| |2022-01-01T18:00:00Z| 9 |36.2|Living Room|22.8| |2022-01-01T19:00:00Z|22 |36.6| Kitchen |23.1| |2022-01-01T19:00:00Z|14 |36.3|Living Room|22.5| |2022-01-01T20:00:00Z|26 |36.5| Kitchen |22.7| |2022-01-01T20:00:00Z|17 |36.4|Living Room|22.2| [](#select-field-values-above-a-threshold) Select field values above a threshold ```sql SELECT co FROM home WHERE co > 9 ``` name: home | time |co | |--------------------|---| |2022-01-01T18:00:00Z|18 | |2022-01-01T19:00:00Z|14 | |2022-01-01T19:00:00Z|22 | |2022-01-01T20:00:00Z|17 | |2022-01-01T20:00:00Z|26 | [](#select-specific-field-values) Select specific field values ```sql SELECT room, co FROM home WHERE co = 9 ``` name: home | time | room |co | |--------------------|-----------|---| |2022-01-01T17:00:00Z| Kitchen | 9 | |2022-01-01T18:00:00Z|Living Room| 9 | [](#select-field-values-based-on-arithmetic) Select field values based on arithmetic ```sql SELECT room, co FROM home WHERE co - 10 > 5 ``` name: home | time | room |co | |--------------------|-----------|---| |2022-01-01T18:00:00Z| Kitchen |18 | |2022-01-01T19:00:00Z| Kitchen |22 | |2022-01-01T20:00:00Z|Living Room|17 | |2022-01-01T20:00:00Z| Kitchen |26 | [](#select-data-with-field-values-above-a-threshold-and-a-specific-tag-value) Select data with field values above a threshold and a specific tag value ```sql SELECT * FROM home WHERE temp > 22.7 AND room = 'Kitchen' ``` name: home | time |co |hum | room |temp| |--------------------|---|----|-------|----| |2022-01-01T09:00:00Z| 0 |36.2|Kitchen| 23 | |2022-01-01T13:00:00Z| 1 |36.5|Kitchen|22.8| |2022-01-01T14:00:00Z| 1 |36.3|Kitchen|22.8| |2022-01-01T18:00:00Z|18 |36.9|Kitchen|23.3| |2022-01-01T19:00:00Z|22 |36.6|Kitchen|23.1| [](#select-data-based-on-the-relationship-between-columns) Select data based on the relationship between columns ```sql SELECT co, temp FROM home WHERE co > temp ``` name: home | time |co |temp| |--------------------|---|----| |2022-01-01T20:00:00Z|26 |22.7| ## Notable behaviors ### Single and double quotes In InfluxQL, single quotation marks (`'`) and double quotation marks (`"`) work differently and can alter the way a `WHERE` clause functions. Single quotes are used in [string](/influxdb3/enterprise/reference/influxql/#strings)and [timestamp](/influxdb3/enterprise/reference/influxql/#dates--times) literals. Double quotes are used to quote [identifiers](/influxdb3/enterprise/reference/influxql/#identifiers), (time, field, and tag column names). For example, the following conditional expression compares the value of the`location` *column* to the *literal string*, `London`: ```sql "location" = 'London' ``` The following conditional expression compares the value of the `location` *column*to the value of the `London` *column*: ```sql "location" = "London" ``` Misused double and single quotes in the `WHERE` clause often results in unexpected empty query results. For more information about quotation marks, see[InfluxQL quotation](/influxdb3/enterprise/reference/influxql/quoting/). ### Cannot query multiple time ranges InfluxDB does not support using `OR` in the `WHERE` clause to query multiple time ranges. For example, the following query returns no results: ```sql SELECT * FROM home WHERE (time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z') OR (time >= '2022-01-01T18:00:00Z' AND time <= '2022-01-01T20:00:00Z') ``` | Operator | Meaning | Supported data types | | --- | --- | --- | | Operator | Meaning | Supported data types | | = | Equal to | all | | <> | Not equal to | all | | != | Not equal to | all | | > | Greater than | numeric, timestamp | | >= | Greater than or equal to | numeric, timestamp | | < | Less than | numeric, timestamp | | <= | Less than or equal to | numeric, timestamp | | =~ | Matches a regular expression | strings | | !~ | Doesn’t match a regular expression | strings | | Operator | Meaning | | --- | --- | | Operator | Meaning | | AND | Returns true if both operands are true . Otherwise, returns false . | | OR | Returns true if any operand is true . Otherwise, returns false . | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Living Room | 21.1 | | 2022-01-01T09:00:00Z | 0 | 35.9 | Living Room | 21.4 | | 2022-01-01T10:00:00Z | 0 | 36 | Living Room | 21.8 | | 2022-01-01T11:00:00Z | 0 | 36 | Living Room | 22.2 | | 2022-01-01T12:00:00Z | 0 | 35.9 | Living Room | 22.2 | | … | … | … | … | … | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T08:00:00Z | 0 | 35.9 | Kitchen | 21 | | 2022-01-01T08:00:00Z | 0 | 35.9 | Living Room | 21.1 | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T09:00:00Z | 0 | 35.9 | Living Room | 21.4 | | 2022-01-01T10:00:00Z | 0 | 36.1 | Kitchen | 22.7 | | 2022-01-01T10:00:00Z | 0 | 36 | Living Room | 21.8 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T18:00:00Z | 18 | 36.9 | Kitchen | 23.3 | | 2022-01-01T18:00:00Z | 9 | 36.2 | Living Room | 22.8 | | 2022-01-01T19:00:00Z | 22 | 36.6 | Kitchen | 23.1 | | 2022-01-01T19:00:00Z | 14 | 36.3 | Living Room | 22.5 | | 2022-01-01T20:00:00Z | 26 | 36.5 | Kitchen | 22.7 | | 2022-01-01T20:00:00Z | 17 | 36.4 | Living Room | 22.2 | | time | co | | --- | --- | | time | co | | 2022-01-01T18:00:00Z | 18 | | 2022-01-01T19:00:00Z | 14 | | 2022-01-01T19:00:00Z | 22 | | 2022-01-01T20:00:00Z | 17 | | 2022-01-01T20:00:00Z | 26 | | time | room | co | | --- | --- | --- | | time | room | co | | 2022-01-01T17:00:00Z | Kitchen | 9 | | 2022-01-01T18:00:00Z | Living Room | 9 | | time | room | co | | --- | --- | --- | | time | room | co | | 2022-01-01T18:00:00Z | Kitchen | 18 | | 2022-01-01T19:00:00Z | Kitchen | 22 | | 2022-01-01T20:00:00Z | Living Room | 17 | | 2022-01-01T20:00:00Z | Kitchen | 26 | | time | co | hum | room | temp | | --- | --- | --- | --- | --- | | time | co | hum | room | temp | | 2022-01-01T09:00:00Z | 0 | 36.2 | Kitchen | 23 | | 2022-01-01T13:00:00Z | 1 | 36.5 | Kitchen | 22.8 | | 2022-01-01T14:00:00Z | 1 | 36.3 | Kitchen | 22.8 | | 2022-01-01T18:00:00Z | 18 | 36.9 | Kitchen | 23.3 | | 2022-01-01T19:00:00Z | 22 | 36.6 | Kitchen | 23.1 | | time | co | temp | | --- | --- | --- | | time | co | temp | | 2022-01-01T20:00:00Z | 26 | 22.7 | --- # InfluxDB 3 Enterprise internals Source: https://docs.influxdata.com/influxdb3/enterprise/reference/internals/ Learn about InfluxDB 3 Enterprise internal systems and architecture. Understanding these internals can help with performance optimization, troubleshooting, and advanced integrations. ## [Data retention in InfluxDB 3 Enterprise](/influxdb3/enterprise/reference/internals/data-retention/) InfluxDB 3 Enterprise enforces database and table retention periods at query time and, to optimize storage, routinely deletes expired data. ## [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) Learn about the InfluxDB 3 Enterprise storage engine, including the upgraded storage engine (the default for new clusters) and how to upgrade or downgrade an existing cluster. ## [InfluxDB 3 Enterprise authentication and authorization](/influxdb3/enterprise/reference/internals/authentication/) InfluxDB 3 Enterprise uses an Attribute-Based Access Control (ABAC) model to manage permissions for authentication (authn) and authorization (authz). ## [Role-based access control (RBAC)](/influxdb3/enterprise/reference/internals/rbac/) How InfluxDB 3 Enterprise role-based access control (RBAC) works: built-in roles and the permissions model for multi-user authentication. ## [Data durability](/influxdb3/enterprise/reference/internals/durability/) Learn how InfluxDB 3 Enterprise ensures data durability through a structured write path that balances performance, persistence, and query efficiency. --- # InfluxDB 3 Enterprise authentication and authorization Source: https://docs.influxdata.com/influxdb3/enterprise/reference/internals/authentication/ InfluxDB 3 Enterprise uses an Attribute-Based Access Control (ABAC) model to manage permissions and supports multiple token types for different authentication scenarios. This model allows for fine-grained control over access to resources and actions within an InfluxDB 3 Enterprise instance. The ABAC model includes the following components: * **Authentication (authn)**: The process through which a user verifies their identity. In InfluxDB 3 Enterprise, this occurs when a token is validated. Users may be human or machine (for example, through automation). InfluxDB 3 Enterprise tokens represent previously verified authenticated users that facilitate automation. * **Authorization (authz)**: The process that determines if an authenticated user can perform a requested action. In InfluxDB 3 Enterprise, authorization evaluates whether a token has permissions to perform actions on specific resources. * **Context**: The system may use contextual information, such as location or time, when evaluating permissions. * **Subject**: The identity requesting access to the system. In InfluxDB 3 Enterprise, the subject is a *token* (similar to an “API key” in other systems). Tokens include attributes such as identifier, name, description, and expiration date. * **Action**: The operations (for example, CRUD) that subjects may perform on resources. * **Permissions**: The set of actions that a specific subject can perform on a specific resource. Authorization compares the incoming request against the permissions set to decide if the request is allowed or not. In InfluxDB 3 Enterprise, *admin* tokens have all permissions, while *resource* tokens have specific permissions. Resource tokens have fine-grained permissions for specific resources of a specific type. For example, a database token can have permissions to read from a specific database but not write to it. * **Resource**: The objects that can be accessed or manipulated. Resources have attributes such as identifier and name. In InfluxDB 3 Enterprise, resources include databases and system information endpoints. * Database tokens provide access to specific databases for actions like writing and querying data. * System tokens provide access to system-level resources, such as API endpoints for server runtime statistics and health. Access controls for system information API endpoints help prevent information leaks and attacks (such as DoS). #### Related * [Manage tokens](/influxdb3/enterprise/admin/tokens/) --- # Data retention in InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/reference/internals/data-retention/ InfluxDB 3 Enterprise enforces database and table retention periods at query time. Any points with timestamps beyond a retention period are filtered out of query results, even though the data may still exist in storage. * [Database retention period](#database-retention-period) * [Table retention period](#table-retention-period) * [Retention period duration formats](#retention-period-duration-formats) * [Set database retention period](#set-database-retention-period) * [Update retention periods](#update-retention-periods) * [Retention period precedence](#retention-period-precedence) * [When does data actually get deleted?](#when-does-data-actually-get-deleted) ## Database retention period A **database retention period** is the duration of time that a database retains data. Retention periods are designed to automatically delete expired data and optimize storage without any user intervention. By default, data does not expire. When you [create a database](/influxdb3/enterprise/admin/databases/create/), you can optionally set a retention period. Retention periods can be as short as an hour or infinite (`none`). [Points](/influxdb3/enterprise/reference/glossary/#point) in a database with timestamps beyond the defined retention period (relative to now) are not queryable, but may still exist in storage until[fully deleted](#when-does-data-actually-get-deleted). Database retention periods serve as the default retention period for all tables in the database, unless a table has its own retention period defined. ## Table retention period In addition to database-level retention periods, InfluxDB 3 Enterprise supports**table-level retention periods**. A table retention period overrides the database retention period for that specific table. This allows you to: * Keep different types of data for different durations within the same database * Apply shorter retention periods to high-volume, low-value data * Apply longer retention periods to critical data that needs extended retention ## Retention period duration formats Retention periods are specified as [duration](/influxdb3/enterprise/reference/glossary/#duration)values using a numeric value plus a duration unit. The retention period value cannot be negative or contain whitespace. ### Valid duration units |Unit| Description | |----|---------------| |`h` | hour | |`d` | day | |`w` | week | |`mo`|month (30 days)| |`y` |year (365 days)| > [!Note] > Minute (`m`) and second (`s`) units are not supported for retention periods. > [!Warning] > #### Retention period constraints > > * **Minimum for data retention**: The practical minimum retention period is 1 hour (`1h`). > * **Zero-duration periods**: Setting a retention period to `0` (for example,`0d` or `0h`) is allowed but marks all data for immediate deletion at query time.*This differs from InfluxDB 1.x and 2.x where `0d` meant infinite retention.* > * **Infinite retention**: Use `none` to set an infinite retention period. ### Example retention period values |Value | Description | |------|-----------------------------| | `1h` | 1 hour | |`24h` | 24 hours (1 day) | | `7d` | 7 days | | `4w` | 4 weeks (28 days) | |`1mo` | 1 month (30 days) | |`90d` | 90 days | | `1y` | 1 year (365 days) | |`none`|Infinite - data never expires| You can combine multiple duration units in a single value: | Value | Description | |--------|--------------------------------| |`30d12h`|30 days and 12 hours (30.5 days)| | `2w3d` | 2 weeks and 3 days (17 days) | |`1y6mo` | 1 year and 6 months (545 days) | ### Set database retention period Use the [`influxdb3 create database` command](/influxdb3/enterprise/reference/cli/influxdb3/create/database/) or the [/api/v3/configure/database](/influxdb3/enterprise/api/database/#operation/PostConfigureDatabase) HTTP API endpoint to create a database with a retention period: [influxdb3 CLI](#influxdb3-cli-3)[HTTP API](#http-api-3) ```bash # Create a database with a 30-day retention period influxdb3 create database --retention-period 30d DATABASE_NAME # Create a database with infinite retention influxdb3 create database --retention-period none DATABASE_NAME # Create a database with a 90-day retention period using authentication influxdb3 create database \ --retention-period 90d \ --token AUTH_TOKEN \ DATABASE_NAME ``` ```bash # Create a database with a 30-day retention period curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "retention_period": "30d" }' # Create a database with infinite retention curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "retention_period": "none" }' # Create a database with a 90-day retention period curl --request POST "http://localhost:8181/api/v3/configure/database" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "retention_period": "90d" }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ### Set table retention period Use the [`influxdb3 create table` command](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) or the [/api/v3/configure/table](/influxdb3/enterprise/api/table/#operation/PostConfigureTable) HTTP API endpoint to create a table with a retention period: [influxdb3 CLI](#influxdb3-cli)[HTTP API](#http-api) ```bash # Create a table with a 7-day retention period influxdb3 create table \ --tags sensor_id,location \ --retention-period 7d \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME # Create a table with field definitions and retention period influxdb3 create table \ --tags room,sensor_id \ --fields temp:float64,hum:float64 \ --retention-period 30d \ --database DATABASE_NAME \ --token AUTH_TOKEN \ TABLE_NAME ``` ```bash # Create a table with a 7-day retention period curl --request POST "http://localhost:8181/api/v3/configure/table" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["sensor_id", "location"], "retention_period": "7d" }' # Create a table with field definitions and retention period curl --request POST "http://localhost:8181/api/v3/configure/table" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "db": "DATABASE_NAME", "table": "TABLE_NAME", "tags": ["room", "sensor_id"], "fields": [ {"name": "temp", "type": "float64"}, {"name": "hum", "type": "float64"} ], "retention_period": "30d" }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `TABLE_NAME`: the name of the table to create * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) ## Update retention periods InfluxDB 3 Enterprise allows you to update database retention periods after creation using the[`influxdb3 update database` command](/influxdb3/enterprise/reference/cli/influxdb3/update/database/). [influxdb3 CLI](#influxdb3-cli-2)[HTTP API](#http-api-2) ```bash # Update a database retention period to 60 days influxdb3 update database --retention-period 60d DATABASE_NAME # Clear a database retention period (set to infinite) influxdb3 update database --retention-period none DATABASE_NAME # Update with authentication influxdb3 update database \ --retention-period 90d \ --token AUTH_TOKEN \ DATABASE_NAME ``` ```bash # Update a database retention period to 60 days curl --request PATCH "http://localhost:8181/api/v3/configure/database/DATABASE_NAME" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "retention_period": "60d" }' # Clear a database retention period (set to infinite) curl --request PATCH "http://localhost:8181/api/v3/configure/database/DATABASE_NAME" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "retention_period": "none" }' # Update a database retention period to 90 days curl --request PATCH "http://localhost:8181/api/v3/configure/database/DATABASE_NAME" \ --header "Content-Type: application/json" \ --header "Authorization: Bearer AUTH_TOKEN" \ --data '{ "retention_period": "90d" }' ``` Replace the following: * `DATABASE_NAME`: the name of the database * `AUTH_TOKEN`: your [admin token](/influxdb3/enterprise/admin/tokens/) > [!Note] > InfluxDB 3 Enterprise does not currently support updating table retention periods > after creation. Table retention periods can only be set when creating the table. ## Retention period precedence When both database and table retention periods are defined, the **table retention period takes precedence** for that specific table. This allows fine-grained control over data retention within a database. ### Examples #### Example: Table retention shorter than database retention ```bash # Database with 90-day retention influxdb3 create database --retention-period 90d mydb # Table with 7-day retention (overrides database retention) influxdb3 create table \ --tags sensor_id \ --retention-period 7d \ --database mydb \ metrics ``` In this example: * Data in the `metrics` table expires after 7 days * Data in other tables in `mydb` expires after 90 days #### Example: Table retention longer than database retention ```bash # Database with 30-day retention influxdb3 create database --retention-period 30d mydb # Table with 1-year retention (overrides database retention) influxdb3 create table \ --tags device_id \ --retention-period 1y \ --database mydb \ audit_logs ``` In this example: * Data in the `audit_logs` table expires after 1 year * Data in other tables in `mydb` expires after 30 days #### Example: Database retention updated after table creation ```bash # Original database with 30-day retention influxdb3 create database --retention-period 30d mydb # Table with 7-day retention influxdb3 create table \ --tags sensor_id \ --retention-period 7d \ --database mydb \ metrics # Update database retention to 90 days influxdb3 update database --retention-period 90d mydb ``` After the update: * Data in the `metrics` table still expires after 7 days (table retention unchanged) * Data in other tables in `mydb` now expires after 90 days (affected by database update) ## When does data actually get deleted? InfluxDB 3 Enterprise routinely deletes expired data to optimize storage. The retention enforcement service runs periodically and: 1. **Identifies expired data**: Finds data with timestamps beyond the retention period 2. **Deletes expired files**: Permanently removes Parquet files containing only expired data 3. **Optimizes mixed files**: For files containing both expired and non-expired data, the system may compact and rewrite files to remove expired data The timing of physical deletion depends on: * The retention enforcement service schedule * The compaction strategy configured for your installation * The ratio of expired to non-expired data in Parquet files > [!Important] > Even though expired data may still exist in storage temporarily, it will never > appear in query results. Retention period enforcement at query time ensures > expired data is always filtered out. #### Related * [Create a database](/influxdb3/enterprise/admin/databases/create/) * [Create a table](/influxdb3/enterprise/admin/tables/create/) * [influxdb3 create database](/influxdb3/enterprise/reference/cli/influxdb3/create/database/) * [influxdb3 create table](/influxdb3/enterprise/reference/cli/influxdb3/create/table/) * [influxdb3 update database](/influxdb3/enterprise/reference/cli/influxdb3/update/database/) * [Create database API](/influxdb3/enterprise/api/database/#operation/PostConfigureDatabase) * [Create table API](/influxdb3/enterprise/api/table/#operation/PostConfigureTable) * [Update database API](/influxdb3/enterprise/api/database/#operation/update_database) [internals](/influxdb3/enterprise/tags/internals/)[retention](/influxdb3/enterprise/tags/retention/) | Unit | Description | | --- | --- | | Unit | Description | | h | hour | | d | day | | w | week | | mo | month (30 days) | | y | year (365 days) | | Value | Description | | --- | --- | | Value | Description | | 1h | 1 hour | | 24h | 24 hours (1 day) | | 7d | 7 days | | 4w | 4 weeks (28 days) | | 1mo | 1 month (30 days) | | 90d | 90 days | | 1y | 1 year (365 days) | | none | Infinite - data never expires | | Value | Description | | --- | --- | | Value | Description | | 30d12h | 30 days and 12 hours (30.5 days) | | 2w3d | 2 weeks and 3 days (17 days) | | 1y6mo | 1 year and 6 months (545 days) | --- # InfluxDB 3 Enterprise Internals Source: https://docs.influxdata.com/influxdb3/enterprise/reference/internals/durability/ ## How data flows through InfluxDB 3 Enterprise When data is written to InfluxDB 3 Enterprise, it progresses through multiple stages to ensure durability, optimize performance, and enable efficient querying. Configuration options at each stage affect system behavior, balancing reliability and resource usage. ## Data flow for writes As written data moves through InfluxDB 3 Enterprise, it follows a structured path to ensure durability, efficient querying, and optimized storage. Ingest path and data flow for InfluxDB 3 Core & EnterpriseFigure: Write request, response, and ingest flow for InfluxDB 3 Core and Enterprise 1. [Write validation and memory buffer](#write-validation-and-memory-buffer) 2. [Write-ahead log (WAL) persistence](#write-ahead-log-wal-persistence) 3. [Query availability](#query-availability) 4. [Parquet storage](#parquet-storage) 5. [In-memory cache](#in-memory-cache) ### Write validation and memory buffer * **Process**: InfluxDB validates incoming data before accepting it into the system. * **Impact**: Prevents malformed or unsupported data from entering the database. * **Details**: The database validates incoming data and stores it in the write buffer (in memory). If `no_sync=true`, the server sends a response to acknowledge the write [without waiting for persistence](/influxdb3/enterprise/reference/cli/influxdb3/write/#write-line-protocol-and-immediately-return-a-response). ### Write-ahead log (WAL) persistence * **Process**: The database flushes the write buffer to the WAL every second (default). * **Impact**: Ensures durability by persisting data to object storage. * **Tradeoff**: More frequent flushing improves durability but increases I/O overhead. * **Details**: Every second (default), the database flushes the write buffer to the Write-Ahead Log (WAL) for persistence in the object store. If `no_sync=false` (default), the server sends a response to acknowledge the write. ### Query availability * **Process**: The system moves data to the queryable buffer after WAL persistence. * **Impact**: Enables fast queries on recent data. * **Tradeoff**: A larger buffer speeds up queries but increases memory usage. * **Details**: After WAL persistence completes, data moves to the queryable buffer where it becomes available for queries. By default, the server keeps up to 900 WAL files (15 minutes of data) buffered. ### Parquet storage * **Process**: Every ten minutes (default), data is persisted to Parquet files in object storage. * **Impact**: Provides durable, long-term storage. * **Tradeoff**: More frequent persistence reduces reliance on the WAL but increases I/O costs. * **Memory usage**: The persistence process uses memory from the configured memory pool ([`exec-mem-pool-size`](/influxdb3/enterprise/reference/config-options/#exec-mem-pool-size)) when converting data to Parquet format. For write-heavy workloads, ensure adequate memory is allocated. * **Details**: Every ten minutes (default), InfluxDB 3 Enterprise persists the oldest data from the queryable buffer to the object store in Parquet format, and keeps the remaining data (the most recent 5 minutes) in memory. ### WAL tail The *WAL tail* is the most recent data in the WAL that InfluxDB 3 Enterprise has not yet durably persisted beyond the WAL. Because InfluxDB 3 Enterprise flushes the WAL to object storage every second, the WAL tail is durable, but it remains in the WAL until the next persistence step captures it: the next Parquet persistence on the Parquet engine, or the next snapshot on the upgraded storage engine (the default for new clusters in 3.11+). ### In-memory cache * **Process**: Recently persisted Parquet files are cached in memory. * **Impact**: Reduces query latency by minimizing object storage access. * **Details**: InfluxDB 3 Enterprise puts Parquet files into an in-memory cache so that queries against the most recently persisted data don’t have to go to object storage. ## Upgraded storage engine compaction The upgraded storage engine (the default for new clusters in 3.11+) compacts data differently from the Parquet engine described above. Incoming writes are buffered in the WAL, flushed to snapshots, and merged into Gen0 files. From there, InfluxDB 3 Enterprise uses **time-disjoint two-level compaction** by default: all leading-edge ingest funnels through a hot-tail L1 run set, and several concurrent L1 compaction jobs can run against the leading edge (or any heavily written range) at once. L1 run sets are allowed to transiently overlap in their assigned time range under load—this is a healthy, query-safe state—and the compactor reconciles back to a disjoint L1 as high-priority background work. Under low load, nothing overlaps and behavior matches the legacy layout. Clusters that started on 3.10 or earlier and have not yet upgraded keep the legacy four-level (L1 through L4) compaction layout, where compaction serializes on a single hot tail and concurrent leading-edge writes must wait. Newly created window/shards use the time-disjoint layout; existing checkpoints keep their recorded layout until explicitly upgraded. For per-level and per-engine tuning options, see[Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/). --- # Role-based access control (RBAC) Source: https://docs.influxdata.com/influxdb3/enterprise/reference/internals/rbac/ > [!Note] > #### RBAC applies to multi-user authentication > > Role-based access control governs users signed in through multi-user > authentication, which is **off by default** in InfluxDB 3 Enterprise. Existing`apiv3_` token workflows are unaffected. See[Manage users and authentication](/influxdb3/enterprise/admin/security/manage-users/)to enable it. Role-based access control (RBAC) governs what authenticated users can do in InfluxDB 3 Enterprise. Each user is assigned one or more built-in roles that determine their permissions. ## Built-in roles InfluxDB 3 Enterprise provides three built-in roles: * **Admin**: Full access to all resources, including creating and managing other users, roles, and tokens. * **Auditor**: Can list and describe databases and read tokens, users, and roles. Can’t query or write data. * **Member**: Can query and write data and create databases, and can create, read, and delete tokens. Can read users and roles. Can’t delete databases or manage users, roles, or admin tokens. > [!Important] > #### Auditor and Member grant less than their names suggest > > Only the **Admin** role (or an admin token) has full access. The**Auditor** role can’t read data, and neither **Auditor** nor **Member**can access system endpoints such as `/health`, `/metrics`, `/ping`, and`/ready`, or query the `_internal` database. Use an admin token for user > and role management. ## Assign roles Assign roles to a user with the `influxdb3 update user-roles` command. See[Manage users and authentication](/influxdb3/enterprise/admin/security/manage-users/)for the user-management workflow. ## Custom roles Authoring custom roles (creating roles and editing role permissions) is disabled by default (`--rbac-authoring-disabled` defaults to `true`). Listing roles and assigning the built-in roles to users remain available regardless of this setting. See[configuration options](/influxdb3/enterprise/reference/config-options/#rbac-authoring-disabled)for details. ## Limitations RBAC has the following known limitation in InfluxDB 3 Enterprise: * **Token scope can exceed role scope**: A user with token-creation permission (**Admin** or **Member**) can create a token with database-level permissions broader than their own role. InfluxDB 3 Enterprise only restricts non-admin users from creating tokens that grant system-resource permissions. #### Related * [Manage users and authentication](/influxdb3/enterprise/admin/security/manage-users/) * [InfluxDB 3 Enterprise authentication and authorization](/influxdb3/enterprise/reference/internals/authentication/) * [Manage users with InfluxDB 3 Explorer](/influxdb3/explorer/manage-users/) * [User roles and access in InfluxDB 3 Explorer](/influxdb3/explorer/manage-users/roles/) --- # Storage engine Source: https://docs.influxdata.com/influxdb3/enterprise/reference/internals/storage-engine/ > [!Important] > #### The upgraded storage engine is the default for new clusters > > New InfluxDB 3 Enterprise clusters default to the upgraded storage > engine—no flag is required. > Clusters that started on 3.10 or earlier keep the Parquet engine until you > run the storage engine upgrade by restarting the cluster with[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree). ## What is the storage engine upgrade? InfluxDB 3 Enterprise includes major upgrades to the storage layer that improve how data is written, stored, compressed, compacted, and queried. These upgrades touch every layer of the storage path—from a new on-disk file format to how fields are organized into column families and how compaction manages resources (see[Upgraded storage engine compaction](/influxdb3/enterprise/reference/internals/durability/#upgraded-storage-engine-compaction)for the compaction model). ## Why these upgrades The existing InfluxDB 3 storage layer uses [Apache Parquet](https://parquet.apache.org/)and is optimized for analytical workloads. Customers running high-cardinality, wide-schema, and query-intensive workloads need better single-series query performance, more predictable resource usage, and the schema flexibility that made InfluxDB v1 and v2 popular. These upgrades extend the storage layer to support those workloads while maintaining full compatibility with InfluxDB 3’s data model and query languages. Key improvements include: * **Faster single-series queries**: Optimized for highly selective time-series queries. * **Resource usage**: Bounded CPU and memory during persistence and compaction, using a fixed memory budget instead of unbounded growth during heavy ingestion or compaction. * **Wide-and-sparse table support**: Schemas with up to millions of columns and dynamic schema evolution without full-table rewrites. * **Column families**: Group related fields together on disk, so queries only read the data they need. * **Bulk data export**: Export compacted data as Parquet files for use with external tools. * **Automatic Parquet upgrade**: Existing data migrates automatically; hybrid query mode covers reads during the transition. ## Run the storage engine upgrade New clusters use the upgraded storage engine by default and do not need any flag. For clusters that started on 3.10 or earlier (Parquet engine), run the storage engine upgrade by including the `--upgrade-pacha-tree` flag in your[`influxdb3 serve` startup command](/influxdb3/enterprise/get-started/setup/): ```bash influxdb3 serve \ --node-id host01 \ --cluster-id cluster01 \ --object-store file \ --data-dir ~/.influxdb3 \ --upgrade-pacha-tree ``` You can also trigger the upgrade with an environment variable: ```bash export INFLUXDB3_UPGRADE_PACHA_TREE=true influxdb3 serve ... ``` > [!Note] > #### Upgrading the storage engine > > The `--use-pacha-tree` flag and the `INFLUXDB3_USE_PACHA_TREE` and`INFLUXDB3_ENTERPRISE_USE_PACHA_TREE` environment variables are > deprecated. > They are still accepted and start the same migration, but the server > logs a deprecation warning at startup. See [Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/)for tuning options, or[Query system data](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables)for system tables and telemetry. > [!Warning] > #### Existing clusters with Parquet data > > On clusters with existing Parquet data, running the storage engine upgrade**automatically converts Parquet files to `.pt` format** on startup, which > consumes additional CPU and memory while the migration runs. > Queries continue to work normally during this period. > See [Upgrade from Parquet](#upgrade-from-parquet) for details. > > Before you upgrade a production cluster, test the storage engine upgrade in > a staging or test environment first. ## What’s changed These upgrades touch every layer of the storage path—from the on-disk file format to how data is compressed and organized. ### New file format Data is stored in a new columnar file format (`.pt` files) optimized for time-series workloads. All data within a file is sorted by column family key,[series key](/influxdb3/enterprise/reference/glossary/#series-key), and timestamp, which enables efficient compaction, querying, and filtering. The format uses type-specific compression algorithms that adapt to data characteristics—delta-delta RLE for timestamps, Gorilla encoding for floats, dictionary encoding for low-cardinality strings, and more. ### Column families Column families let you group related fields together so that queries only read the data they need. Fields in the same family are stored together on disk. For wide tables with hundreds of fields, this dramatically reduces I/O. When writing [line protocol](/influxdb3/enterprise/reference/line-protocol/), use the `::` (double-colon) delimiter in field names to assign fields to a family. The portion before `::` is the family name; everything after is the field name. ```txt metrics,host=sA cpu::usage_user=55.2,cpu::usage_sys=12.1,cpu::usage_idle=32.7 1000000000 metrics,host=sA mem::free=2048i,mem::used=6144i,mem::cached=1024i 1000000000 metrics,host=sA disk::read_bytes=50000i,disk::write_bytes=32000i 1000000000 ``` This creates three column families: |Family| Fields | |------|---------------------------------------| |`cpu` |`usage_user`, `usage_sys`, `usage_idle`| |`mem` | `free`, `used`, `cached` | |`disk`| `read_bytes`, `write_bytes` | When a query references only `mem::free`, the storage layer reads only the`mem` family block and skips `cpu` and `disk` data entirely. > [!Note] > Only the first `::` is significant. > A field name like `a::b::c` creates family `a` with field `b::c`. Fields written without `::` are assigned to auto-generated families (named`__0`, `__1`, etc.), each holding up to 100 fields. ## Upgrade from Parquet Existing clusters with Parquet data can upgrade with zero manual migration. The upgrade is fully automatic and occurs on initial startup. When you restart a cluster with `--upgrade-pacha-tree`, the system: 1. Detects existing Parquet data and enters hybrid mode. 2. Clears the legacy WAL on ingest nodes and streams Parquet files through a conversion pipeline. 3. Integrates converted files into the new storage format through compaction. 4. Automatically transitions once all data is migrated. During hybrid mode, queries merge results from both the legacy and upgraded storage layers. If there is a conflict (same series key and timestamp), the upgraded data takes precedence. Each ingest node on the upgrade roster must complete its migration. An ingest node that remains `stopped` without completing its migration blocks the cluster upgrade. Before you start the upgrade, remove the ingest nodes you’ve retired, after the compactor has absorbed their Parquet files. See [Node removal on the Parquet engine](/influxdb3/enterprise/admin/node-lifecycle/#removal-on-the-parquet-engine-completes-after-the-storage-engine-upgrade). For the upgrade, use the latest 3.11.x or 3.12.x release. Starting with v3.11.8 and v3.12.2, the upgrade drops a removed ingest node from its roster only after the node’s Parquet files are converted or absorbed. In earlier releases, a removed ingest node holds the upgrade at `upgrading`. After the upgrade completes, Parquet engine options remaining in your configuration—including options you may have relied on long before the upgrade—no longer have any effect. On the next server start following completion of the upgrade, each one logs a warning indicating it can be removed. Use the startup log as a checklist for cleaning up your configuration. To monitor upgrade progress, see[Query system data](/influxdb3/enterprise/admin/query-system-data/#query-storage-engine-tables). ## Downgrade to Parquet If you need to revert an upgraded cluster back to standard Parquet storage, use the `influxdb3 downgrade-to-parquet` command. This command updates the catalog and deletes all files specific to the upgraded storage engine from object storage. > [!Note] > #### Downgrade impacts > > The downgrade deletes all `.pt` files, including data written > after the upgrade.**Only data that existed before the upgrade (original Parquet files) is > preserved.**You can run the storage engine upgrade again later by restarting with`--upgrade-pacha-tree`. Before you downgrade: 1. **Stop all nodes** in the cluster before running the downgrade command. The command checks for running nodes and refuses to proceed if any are active. ``` influxdb3 stop node --node-id ``` 2. **Verify table compatibility.**The downgrade validates that all tables can be represented in Parquet format. Tables that exceed the Parquet column limit or contain columns without legacy Parquet column IDs block the downgrade. Use the `--dry-run` flag to preview the downgrade—list files that would be deleted without making any changes: ```bash influxdb3 downgrade-to-parquet \ --cluster-id cluster01 \ --object-store file \ --data-dir ~/.influxdb3 \ --dry-run ``` Run the downgrade: ```bash influxdb3 downgrade-to-parquet \ --cluster-id cluster01 \ --object-store file \ --data-dir ~/.influxdb3 ``` After the downgrade completes, restart nodes without the`--upgrade-pacha-tree` flag to resume standard Parquet storage mode. For all available options, see[Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/#downgrade-options). ## Export to Parquet You can export compacted data as Parquet files for use with external tools. > [!Note] > Data must be compacted before it can be exported. > Uncompacted data is not available for export at this time. ```bash # Step 1: List available databases influxdb3 export databases # Step 2: List tables in a database influxdb3 export tables --database mydb # Step 3: List compacted 24-hour windows for a table influxdb3 export windows --database mydb --table cpu # Step 4: Export data as Parquet files influxdb3 export data \ --database mydb \ --table cpu \ --output-dir ./export_output ``` To export specific time windows only: ```bash influxdb3 export data \ --database mydb \ --table cpu \ --window 2026-01-15,2026-01-16 \ --output-dir ./export_output ``` ## Who should upgrade existing clusters New clusters use the upgraded storage engine by default. If your cluster started on 3.10 or earlier and still runs the Parquet engine, the storage engine upgrade especially benefits workloads with: * High cardinality or wide tables * Frequent backfill across time ranges * Query-heavy access patterns requiring low latency * Sparse schemas with dynamic column creation * Resource constraints where bounded memory and CPU usage matter #### Related * [Set up InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/setup/) * [Storage engine configuration reference](/influxdb3/enterprise/reference/storage-engine-config-options/) * [Query system data](/influxdb3/enterprise/admin/query-system-data/) * [InfluxDB 3 Enterprise Internals](/influxdb3/enterprise/reference/internals/durability/) [storage](/influxdb3/enterprise/tags/storage/)[performance](/influxdb3/enterprise/tags/performance/)[internals](/influxdb3/enterprise/tags/internals/) | Family | Fields | | --- | --- | | Family | Fields | | cpu | usage_user , usage_sys , usage_idle | | mem | free , used , cached | | disk | read_bytes , write_bytes | --- # Line protocol reference Source: https://docs.influxdata.com/influxdb3/enterprise/reference/line-protocol/ InfluxDB 3 Enterprise uses line protocol to write data points. It is a text-based format that provides the table, tag set, field set, and timestamp of a data point. * [Elements of line protocol](#elements-of-line-protocol) * [Data types and format](#data-types-and-format) * [Quotes](#quotes) * [Special characters](#special-characters) * [Comments](#comments) * [Naming restrictions](#naming-restrictions) * [Duplicate points](#duplicate-points) ```text # Syntax
[,=[,=]] =[,=] [] ``` ```lp myTable,tag1=value1,tag2=value2 fieldKey="fieldValue" 1556813561098000000 ``` Lines separated by the newline character `\n` represent a single point in InfluxDB. Line protocol is whitespace-sensitive. > [!Note] > Line protocol does not support the newline character `\n` in tag or field values. ## Elements of line protocol myTable,tag1=val1,tag2=val2 field1="v1",field2=1i 0000000000000000000 ### Table (Required) The table name. InfluxDB accepts one table per point.*Table names are case-sensitive and subject to [naming restrictions](#naming-restrictions).* ***Data type:** [String](#string)* > [!Note] > If familiar with previous InfluxDB versions, “**table**” is synonymous with > “**measurement**.” ### Tag set (***Optional***) All tag key-value pairs for the point. Key-value relationships are denoted with the `=` operand. Multiple tag key-value pairs are comma-delimited.*Tag keys and tag values are case-sensitive. Tag keys are subject to [naming restrictions](#naming-restrictions). Tag values cannot be empty; instead, omit the tag from the tag set.* ***Key data type:** [String](#string)* ***Value data type:** [String](#string)* ### Field set (Required) All field key-value pairs for the point. Points must have at least one field.*Field keys and string values are case-sensitive. Field keys are subject to [naming restrictions](#naming-restrictions).* ***Key data type:** [String](#string)* ***Value data type:** [Float](#float) | [Integer](#integer) | [UInteger](#uinteger) | [String](#string) | [Boolean](#boolean)* > [!Note] > *Always double quote string field values. More on quotes [below](#quotes).* > > ``` > tableName fieldKey="field string value" 1556813561098000000 > ``` ### Timestamp (***Optional***) The [Unix timestamp](/influxdb3/enterprise/reference/glossary/#unix-timestamp) for the data point. InfluxDB accepts one timestamp per point. If no timestamp is provided, InfluxDB uses the system time (UTC) of its host machine. ***Data type:** [Unix timestamp](#unix-timestamp)* > [!Note] > #### Important notes about timestamps > > * To ensure a data point includes the time a metric is observed (not received by InfluxDB), > include the timestamp. > * If your timestamps are not in nanoseconds, specify the precision of your timestamps > when writing the data to InfluxDB 3 Enterprise. ### Whitespace Whitespace in line protocol determines how InfluxDB interprets the data point. The **first unescaped space** delimits the table and the tag set from the field set. The **second unescaped space** delimits the field set from the timestamp. myTable,tag1=val1,tag2=val2 field1="v1",field2=1i 0000000000000000000 ## Data types and format ### Float IEEE-754 64-bit floating-point numbers. Default numerical type.*InfluxDB supports scientific notation in float field values.* ##### Float field value examples ```lp myTable fieldKey=1.0 myTable fieldKey=1 myTable fieldKey=-1.234456e+78 ``` ### Integer Signed 64-bit integers. Trailing `i` on the number specifies an integer. | Minimum integer | Maximum integer | |-----------------------|----------------------| |`-9223372036854775808i`|`9223372036854775807i`| ##### Integer field value examples ```lp myTable fieldKey=1i myTable fieldKey=12485903i myTable fieldKey=-12485903i ``` ### UInteger Unsigned 64-bit integers. Trailing `u` on the number specifies an unsigned integer. |Minimum uinteger| Maximum uinteger | |----------------|-----------------------| | `0u` |`18446744073709551615u`| ##### UInteger field value examples ```lp myTable fieldKey=1u myTable fieldKey=12485903u ``` ### String Plain text string. | Element |Maximum length| |-----------------------------------------|--------------| |Table name, tag key, tag value, field key| 64KB | | Field value | 1MB | ##### String example ```lp # String table name, field key, and field value myTable fieldKey="this is a string" ``` ### Boolean Stores `true` or `false` values. |Boolean value| Accepted syntax | |-------------|-----------------------------------| | True | `t`, `T`, `true`, `True`, `TRUE` | | False |`f`, `F`, `false`, `False`, `FALSE`| ##### Boolean field value examples ```lp myTable fieldKey=true myTable fieldKey=false myTable fieldKey=t myTable fieldKey=f myTable fieldKey=TRUE myTable fieldKey=FALSE ``` > [!Note] > Do not quote boolean field values. > Quoted field values are interpreted as strings. ### Unix timestamp Unix timestamp in a [specified precision](/influxdb3/enterprise/reference/glossary/#unix-timestamp). Default precision is nanoseconds (`ns`). | Minimum timestamp | Maximum timestamp | |----------------------|---------------------| |`-9223372036854775806`|`9223372036854775806`| ##### Unix timestamp example ```lp myTableName fieldKey="fieldValue" 1556813561098000000 ``` ## Quotes Line protocol supports single and double quotes as described in the following table: | Element | Double quotes | Single quotes | |-----------|---------------------------------------|---------------------------------------| | Table |*Limited* *|*Limited* *| | Tag key |*Limited* *|*Limited* *| | Tag value |*Limited* *|*Limited* *| | Field key |*Limited* *|*Limited* *| |Field value| **Strings only** | Never | | Timestamp | Never | Never | * *Line protocol accepts double and single quotes in table names, tag keys, tag values, and field keys, but interprets them as part of the name, key, or value.* ## Special Characters Line protocol supports special characters in [string elements](#string). In the following contexts, it requires escaping certain characters with a backslash (`\`): | Element | Escape characters | |-----------|-------------------------| | Table | Comma, Space | | Tag key |Comma, Equals Sign, Space| | Tag value |Comma, Equals Sign, Space| | Field key |Comma, Equals Sign, Space| |Field value| Double quote, Backslash | You do not need to escape other special characters. ##### Examples of special characters in line protocol ```lp # Table name with spaces my\ Table fieldKey="string value" # Double quotes in a string field value myTable fieldKey="\"string\" within a string" # Tag keys and values with spaces myTable,tag\ Key1=tag\ Value1,tag\ Key2=tag\ Value2 fieldKey=100 # Emojis myTable,tagKey=🍭 fieldKey="Launch 🚀" 1556813561098000000 ``` ### Escaping backslashes Line protocol supports both literal backslashes and backslashes as an escape character. With two contiguous backslashes, the first is interpreted as an escape character. For example: |Backslashes|Interpreted as| |-----------|--------------| | `\` | `\` | | `\\` | `\` | | `\\\` | `\\` | | `\\\\` | `\\` | | `\\\\\` | `\\\` | | `\\\\\\` | `\\\` | ## Comments Line protocol interprets `#` at the beginning of a line as a comment character and ignores all subsequent characters until the next newline `\n`. ```lp # This is a comment myTable fieldKey="string value" 1556813561098000000 ``` ## Naming restrictions Table names, tag keys, and field keys are alphanumeric and must begin with a letter or a number. They can contain dashes (`-`) and underscores (`_`). ## Duplicate points A point is uniquely identified by the table name, tag set, and timestamp. If you submit line protocol with the same table, tag set, and timestamp, InfluxDB stores a single point for that identity, merging the field sets of the duplicate writes. When duplicate writes set the same field, InfluxDB does not guarantee which value is retained. Read the following warning before relying on overwrites to maintain a last-value view. > [!Warning] > #### Overwrites are not deterministic > > Overwriting a point (same table, tag set, and timestamp) is not reliable in > InfluxDB 3 Enterprise, regardless of the delay between writes: queries may > return either version, and either version may be permanently stored. > Which version is retained depends on write rate, buffer and snapshot timing, > compaction state, and, in clusters with multiple ingest nodes, which node > received each write. > > To maintain a last-value view, use the [append-only patterns](#recommended-patterns-for-last-value-tracking)instead of overwrites. ### Recommended patterns for last-value tracking To reliably maintain a last-value view of your data, use one of these append-only patterns: #### Append-only with unique timestamps (recommended) Write each change as a new point with a unique timestamp using the actual event time. Query for the most recent point to get the current value. **Line protocol example**: ```text device_status,device_id=sensor01 status="active",temperature=72.5 1700000000000000000 device_status,device_id=sensor01 status="active",temperature=73.1 1700000300000000000 device_status,device_id=sensor01 status="inactive",temperature=73.1 1700000600000000000 ``` **SQL query to get latest state**: ```sql SELECT device_id, status, temperature, time FROM device_status WHERE time >= now() - INTERVAL '7 days' AND device_id = 'sensor01' ORDER BY time DESC LIMIT 1 ``` **InfluxQL query to get latest state**: ```influxql SELECT LAST(status), LAST(temperature) FROM device_status WHERE device_id = 'sensor01' AND time >= now() - 7d GROUP BY device_id ``` #### Append-only with change tracking field If you need to filter by “changes since a specific time,” add a dedicated `last_change_timestamp` field. **Line protocol example**: ```text device_status,device_id=sensor01 status="active",temperature=72.5,last_change_timestamp=1700000000000000000i 1700000000000000000 device_status,device_id=sensor01 status="active",temperature=73.1,last_change_timestamp=1700000300000000000i 1700000300000000000 device_status,device_id=sensor01 status="inactive",temperature=73.1,last_change_timestamp=1700000600000000000i 1700000600000000000 ``` **SQL query to get changes since a specific time**: ```sql SELECT device_id, status, temperature, time FROM device_status WHERE last_change_timestamp >= 1700000000000000000 ORDER BY time DESC ``` ### Anti-patterns to avoid The following patterns will produce non-deterministic results when duplicate points are flushed together: #### Don’t overwrite the same (time, tags) point If points with the same time and tag set are flushed to storage together, any of the values might be retained. For example, **don’t do this**: ```text -- All writes use the same timestamp device_status,device_id=sensor01 status="active",temperature=72.5 1700000000000000000 device_status,device_id=sensor01 status="active",temperature=73.1 1700000000000000000 device_status,device_id=sensor01 status="inactive",temperature=73.1 1700000000000000000 ``` #### Don’t add a field while overwriting data (time, tags) Adding a field doesn’t make points unique. Points with the same time and tag set are still considered duplicates–for example,**don’t do this**: ```text -- All writes use the same timestamp, but add a version field device_status,device_id=sensor01 status="active",temperature=72.5,version=1i 1700000000000000000 device_status,device_id=sensor01 status="active",temperature=73.1,version=2i 1700000000000000000 device_status,device_id=sensor01 status="inactive",temperature=73.1,version=3i 1700000000000000000 ``` #### Don’t rely on short write delays to force ordering Short delays don’t guarantee that duplicate points won’t be flushed together. The flush interval depends on buffer size, ingestion rate, and system load. In InfluxDB 3 Enterprise, overwrite resolution is never deterministic, regardless of delay length; use the append-only patterns instead. For example, **don’t do this**: ```text -- Writing with delays between each write device_status,device_id=sensor01 status="active" 1700000000000000000 # Wait 10 seconds... device_status,device_id=sensor01 status="inactive" 1700000000000000000 ``` Append-only patterns increase row count. See [Create a database](/influxdb3/enterprise/admin/databases/create/) to configure shorter retention for last-value data. For query and storage guidance, see [Performance tuning](/influxdb3/enterprise/admin/performance-tuning/). #### Related * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/) [write](/influxdb3/enterprise/tags/write/)[line protocol](/influxdb3/enterprise/tags/line-protocol/)[syntax](/influxdb3/enterprise/tags/syntax/) | Minimum integer | Maximum integer | | --- | --- | | Minimum integer | Maximum integer | | -9223372036854775808i | 9223372036854775807i | | Minimum uinteger | Maximum uinteger | | --- | --- | | Minimum uinteger | Maximum uinteger | | 0u | 18446744073709551615u | | Element | Maximum length | | --- | --- | | Element | Maximum length | | Table name, tag key, tag value, field key | 64KB | | Field value | 1MB | | Boolean value | Accepted syntax | | --- | --- | | Boolean value | Accepted syntax | | True | t , T , true , True , TRUE | | False | f , F , false , False , FALSE | | Minimum timestamp | Maximum timestamp | | --- | --- | | Minimum timestamp | Maximum timestamp | | -9223372036854775806 | 9223372036854775806 | | Element | Double quotes | Single quotes | | --- | --- | --- | | Element | Double quotes | Single quotes | | Table | Limited * | Limited * | | Tag key | Limited * | Limited * | | Tag value | Limited * | Limited * | | Field key | Limited * | Limited * | | Field value | Strings only | Never | | Timestamp | Never | Never | | Element | Escape characters | | --- | --- | | Element | Escape characters | | Table | Comma, Space | | Tag key | Comma, Equals Sign, Space | | Tag value | Comma, Equals Sign, Space | | Field key | Comma, Equals Sign, Space | | Field value | Double quote, Backslash | | Backslashes | Interpreted as | | --- | --- | | Backslashes | Interpreted as | | \ | \ | | \\ | \ | | \\\ | \\ | | \\\\ | \\ | | \\\\\ | \\\ | | \\\\\\ | \\\ | --- # Naming restrictions and conventions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/naming-restrictions/ InfluxDB 3 has specific naming restrictions and conventions that apply to databases, tables, tags, fields, and other identifiers. Understanding these restrictions helps ensure your data model works correctly with all query languages and avoids naming conflicts. ## Database names Database names must follow these restrictions: * **Length**: Maximum 511 ASCII characters * **Allowed characters**: ASCII alphanumeric characters (a-z, A-Z, 0-9), underscore (`_`), and dash (`-`). Multibyte (non-ASCII) characters are not allowed. * **Forward slash**: At most one forward slash (`/`), which separates the database name from a v1 retention policy name (for example, `mydb/autogen`). The slash can’t be the first or last character. * **Prohibited characters**: Whitespace and characters other than those listed above * **Starting character**: Must start with an ASCII letter or number * **Case sensitivity**: Database names are case-sensitive ### Examples **Valid database names:** ```text mydb sensor_data prod-metrics logs/application webserver123 ``` **Invalid database names:** ```text my database # Contains whitespace sensor.data # Contains a period app@server # Contains a special character _internal # Starts with underscore /mydb # Starts with a slash mydb/rp/extra # Contains more than one slash ``` ## Table (measurement) names Table names in InfluxDB 3 Enterprise follow line protocol measurement naming rules: * **Length**: No explicit limit, but practical limits apply for performance * **Allowed characters**: Alphanumeric characters (a-z, A-Z, 0-9), underscore (`_`), dash (`-`) * **Starting character**: Should start with a letter or number and should not start with underscore (`_`) * **Case sensitivity**: Table names are case-sensitive * **Quoting**: Use double quotes when names contain special characters or whitespace ### Examples **Valid table names:** ```text temperature cpu_usage http-requests sensor123 웹서버_메트릭스 # UTF-8 characters ``` **Names requiring quotes in queries:** ```text "my table" # Contains whitespace "cpu.usage" # Contains period "http@requests" # Contains special character ``` **Invalid table names:** ```text _internal # Starts with underscore (not recommended) ``` ## Tag keys and field keys Tag keys and field keys follow these restrictions: * **Length**: No explicit limit, but shorter names improve performance * **Allowed characters**: Alphanumeric characters (a-z, A-Z, 0-9), underscore (`_`), dash (`-`) * **Starting character**: Should start with a letter or number and should not start with underscore (`_`) * **Case sensitivity**: Tag and field keys are case-sensitive * **Quoting**: Use double quotes when names contain special characters or whitespace ### Examples **Valid tag and field keys:** ```text host region temperature cpu_usage http-status sensor123 ``` **Keys requiring quotes in queries:** ```text "host name" # Contains whitespace "cpu.usage" # Contains period "http@status" # Contains special character ``` **Invalid tag and field keys:** ```text _internal # Starts with underscore (not recommended) ``` ## Tag values and field values Tag and field values have different restrictions: ### Tag values * **Type**: Must be strings * **Length**: No explicit limit * **Characters**: Any UTF-8 characters allowed * **Case sensitivity**: Tag values are case-sensitive * **Null values**: Allowed (excluded from primary key) ### Field values * **Type**: Can be integers, floats, strings, booleans, or unsigned integers * **Length**: No explicit limit for strings * **Characters**: Any UTF-8 characters allowed for strings * **Case sensitivity**: String field values are case-sensitive * **Null values**: Allowed (but at least one field must be non-null per row) ## Query language specific considerations Different query languages have additional naming requirements: ### SQL identifiers When using SQL to query InfluxDB 3 Enterprise: * **Unquoted identifiers**: Must start with letter or underscore, contain only letters, digits, or underscores * **Quoted identifiers**: Use double quotes (`"`) for names with special characters, whitespace, or to preserve case * **Case sensitivity**: Unquoted identifiers are case-insensitive, quoted identifiers are case-sensitive * **Reserved keywords**: [SQL keywords](/influxdb3/enterprise/reference/sql/#keywords) must be quoted when used as identifiers ### InfluxQL identifiers When using InfluxQL to query InfluxDB 3 Enterprise: * **Unquoted identifiers**: Must start with ASCII letter or underscore, contain only ASCII letters, digits, or underscores * **Quoted identifiers**: Use double quotes (`"`) for names with special characters or whitespace * **Case sensitivity**: All identifiers are case-sensitive * **Reserved keywords**: [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords) must be quoted when used as identifiers * **Character encoding**: UTF-8 encoding supported ## Reserved namespaces ### System reserved prefixes The following prefixes may be reserved for system use: * **`_`** (underscore): May be reserved for system databases, measurements, and field keys * **`iox_`**: Reserved for InfluxDB internal metadata > [!Caution] > #### Using underscore-prefixed names > > While InfluxDB might not explicitly reject names starting with underscore (`_`), > using them risks conflicts with current or future system features and may > result in unexpected behavior or data loss. ### Common reserved keywords Avoid using these common reserved keywords as identifiers without quoting: **SQL keywords** (partial list): * `SELECT`, `FROM`, `WHERE`, `GROUP`, `ORDER`, `BY` * `CREATE`, `DROP`, `ALTER`, `INSERT`, `UPDATE`, `DELETE` * `TABLE`, `DATABASE`, `INDEX`, `VIEW` * `TIME`, `TIMESTAMP`, `INTERVAL` **InfluxQL keywords** (partial list): * `SELECT`, `FROM`, `WHERE`, `GROUP`, `ORDER`, `BY` * `SHOW`, `DROP`, `CREATE`, `DELETE` * `MEASUREMENT`, `TAG`, `FIELD`, `TIME` * `LIMIT`, `OFFSET`, `SLIMIT`, `SOFFSET` For complete lists, see: * [SQL keywords](/influxdb3/enterprise/reference/sql/#keywords) * [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords) ## Best practices ### Naming conventions 1. **Use descriptive names**: Choose names that clearly describe the data 2. **Keep names simple**: Avoid special characters when possible 3. **Use consistent casing**: Establish and follow a consistent case convention 4. **Avoid reserved keywords**: Don’t use SQL or InfluxQL keywords as identifiers 5. **Use underscores for separation**: Prefer `cpu_usage` over `cpu-usage` or `cpuUsage` ### Performance considerations 1. **Shorter names**: Shorter names improve query performance and reduce storage 2. **Avoid excessive tag cardinality**: Too many unique tag values can impact performance 3. **Limit column count**: Keep the number of columns (tags + fields) reasonable 4. **Consistent naming**: Use the same names across related tables ### Example naming strategy ```text # Database naming prod-metrics dev-logs sensor-data # Table naming cpu_usage memory_utilization http_requests disk_io # Tag keys host region service environment # Field keys value count duration_ms bytes_sent ``` ## Quoting identifiers When identifiers contain special characters, whitespace, or reserved keywords, they must be quoted in queries: ### SQL examples ```sql -- Quoted database and table names SELECT * FROM "my-database"."my table"; -- Quoted column names SELECT "cpu usage", "memory.available" FROM metrics; -- Reserved keyword as identifier SELECT "group" FROM "user-data"; ``` ### InfluxQL examples ```influxql -- Quoted measurement name SELECT * FROM "http requests"; -- Quoted tag key with special characters SELECT * FROM metrics WHERE "host.name" = 'server01'; -- Reserved keyword as field SELECT "time" FROM "system-metrics"; ``` ## Troubleshooting naming issues ### Common error patterns 1. **Unquoted special characters**: Use double quotes around identifiers with special characters 2. **Reserved keyword conflicts**: Quote reserved keywords when used as identifiers 3. **Case sensitivity issues**: Check case sensitivity rules for your query language 4. **Underscore prefix warnings**: Avoid starting names with underscore to prevent conflicts ### Validation tips 1. **Test names in queries**: Verify names work correctly in your target query language 2. **Check for reserved keywords**: Cross-reference names against keyword lists 3. **Validate character encoding**: Ensure UTF-8 characters are properly encoded 4. **Consider future compatibility**: Choose names that work across different query languages #### Related * [Create a database](/influxdb3/enterprise/admin/databases/create/) * [Line protocol reference](/influxdb3/enterprise/reference/line-protocol/) * [SQL reference documentation](/influxdb3/enterprise/reference/sql/) * [InfluxQL reference documentation](/influxdb3/enterprise/reference/influxql/) --- # Processing engine reference Source: https://docs.influxdata.com/influxdb3/enterprise/reference/processing-engine/ The Processing engine is an embedded Python virtual machine that runs inside an InfluxDB 3 Enterprise database server. It executes Python code in response to triggers and database events without requiring external application servers or middleware. ## Enable and disable the Processing Engine The Processing Engine activates when [`--plugin-dir`](/influxdb3/enterprise/reference/config-options/#plugin-dir) or `INFLUXDB3_PLUGIN_DIR` is configured. When not configured, the Python environment and PyO3 bindings aren’t initialized, and the server runs without Processing Engine functionality. ### Process mode and `--plugin-dir` Setting `--plugin-dir` automatically adds `process` mode to any node, regardless of the [`--mode`](/influxdb3/enterprise/reference/config-options/#mode) setting. You don’t need to explicitly set `--mode=process` when `--plugin-dir` is configured. Conversely, if you explicitly set `--mode=process`, you **must** also set `--plugin-dir`. A node with `--mode=process` but no `--plugin-dir` doesn’t start. #### Run the Processing Engine in a cluster In a cluster, we recommend running the Processing Engine on a standalone process node, a node started with `--mode=process` and `--plugin-dir`. The process node runs triggers; plugin queries go to query nodes, and plugin writes go to ingest nodes. Pin each trigger to the process node with [`--node-spec`](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/#options). For the complete setup, see [Configure a standalone Processing Engine](/influxdb3/enterprise/admin/processing-engine/). ### Default behavior by deployment type | Deployment |Default state| Configuration | |----------------|-------------|-----------------------------------------| | Docker images | **Enabled** | `INFLUXDB3_PLUGIN_DIR=/plugins` | |DEB/RPM packages| **Enabled** |`plugin-dir="/var/lib/influxdb3/plugins"`| | Binary/source | Disabled | No `plugin-dir` configured | ### Disable in Docker deployments Docker images set `INFLUXDB3_PLUGIN_DIR=/plugins` by default. > [!Warning] > Setting `INFLUXDB3_PLUGIN_DIR=""` (empty string) does **not** disable the Processing Engine. > You must unset the variable, not set it to empty. Use the `INFLUXDB3_UNSET_VARS` feature to unset inherited environment variables: ```bash docker run -e INFLUXDB3_UNSET_VARS="INFLUXDB3_PLUGIN_DIR" influxdb:3-enterprise ``` `INFLUXDB3_UNSET_VARS` accepts one or more environment variable names (for example, a comma-separated list) and unsets them before the server starts. Use it to clear any inherited variables that you don’t want the InfluxDB 3 Enterprise container to see (for example, `INFLUXDB3_PLUGIN_DIR`, `INFLUXDB3_LOG_LEVEL`) when you can’t modify the parent environment directly. This is useful in orchestration environments (Kubernetes, Docker Compose) where removing an inherited variable isn’t straightforward. ### Disable in systemd deployments (DEB/RPM) The post-install script sets `plugin-dir="/var/lib/influxdb3/plugins"` in the TOML configuration. To disable the Processing Engine: 1. Edit the configuration file: ``` sudo nano /etc/influxdb3/influxdb3-enterprise.conf ``` 2. Comment out or remove the `plugin-dir` line: ``` # plugin-dir="/var/lib/influxdb3/plugins" ``` > [!Warning] > Do not set `plugin-dir=""` (empty string)—you must remove or comment out the line. 3. Restart the service: ``` sudo systemctl restart influxdb3-enterprise ``` > [!Note] > The `/var/lib/influxdb3/plugins` directory can remain on disk. > The Processing Engine only activates based on the `plugin-dir` configuration, not directory existence. ### Benefits of disabling When the Processing Engine is disabled: * The Python environment and PyO3 bindings are not initialized * Plugin-related operations return a “No plugin directory configured” error * The server runs with reduced resource usage This is useful for deployments that don’t require plugin functionality and want a minimal server footprint. ## How it works ### Architecture The Processing engine runs Python code directly within a InfluxDB 3 Enterprise server process. This design provides high performance and direct access to database resources. * **Embedded execution**: Code runs in the same process space as the database server * **Direct data access**: Zero-copy access to data * **Event-driven**: Responds to database writes, scheduled events, and HTTP requests * **Cache integration**: Access to system caches including Last values and Distinct values > [!Note] > The Processing engine runs all plugins in the same Python process. Changes made by one plugin can affect other plugins. ### Event processing flow When specific events occur in the database, the Processing engine handles them through a consistent sequence: 1. A **trigger** specific to the event type activates its plugin. The event types include: * Data writes to specific tables or all tables * Scheduled events (time-based or cron expressions) * HTTP requests to configured endpoints 2. The engine loads the associated **plugin** specified in the trigger configuration 3. The plugin receives context data specific to the trigger type: * Write triggers: the written data and table information * Schedule triggers: the scheduled call time * HTTP triggers: the request object with methods, headers, and body 4. The plugin processes the received data, can query the database, call external tools, and write and cache data in the database 5. Execution completes and the engine returns to waiting state ## Key components ### Trigger system Triggers connect database events to Python code execution based on specific conditions: * **Data write triggers**: Execute on WAL flush events, when data is written to the object store, for a specific table or all tables in a database * **Scheduled triggers**: Run at intervals or according to cron expressions * **HTTP triggers**: Respond to HTTP requests to custom endpoints ### Plugin registry The registry manages all Python code available to the Processing engine: * Indexes plugins by filename and location * Tracks which plugins are used by which triggers * Manages plugin versioning and dependencies ### Memory management The Processing engine implements specialized memory handling to ensure stability and performance: * **Execution isolation**: Each plugin runs in its own context * **Cache system**: Maintains state between executions * **Resource limits**: Controls memory usage and execution time ## Performance characteristics The Processing engine is designed for high-performance operation with minimal overhead: * **Low latency**: Activates triggers in sub-millisecond time * **Efficient access**: Accesses database directly without network overhead * **Controlled resources**: Limits memory and CPU usage through configuration * **Execution policies**: Offers synchronous or asynchronous processing options ## Reliability features The Processing engine includes multiple features to ensure consistent and dependable execution: * **Error handling**: Configures behaviors for failure scenarios (log, retry, or disable) * **Execution tracking**: Tracks plugin performance and resource usage * **State persistence**: Persists cache state across server restarts ## Extension capabilities Extend and customize the Processing engine through several built-in mechanisms: * **Package management**: Installs custom Python dependencies * **Plugin distribution**: Distributes plugins via Git repositories * **Shared API**: Provides consistent interface for database operations For a step-by-step guide to setting up and using the Processing engine, see the [Getting started with plugins](/influxdb3/enterprise/plugins/) documentation. | Deployment | Default state | Configuration | | --- | --- | --- | | Deployment | Default state | Configuration | | Docker images | Enabled | INFLUXDB3_PLUGIN_DIR=/plugins | | DEB/RPM packages | Enabled | plugin-dir="/var/lib/influxdb3/plugins" | | Binary/source | Disabled | No plugin-dir configured | --- # Sample data Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sample-data/ Sample datasets are used throughout the InfluxDB 3 Enterprise documentation to demonstrate functionality. Use the following sample datasets to replicate provided examples. * [Home sensor data](#home-sensor-data) * [Home sensor actions data](#home-sensor-actions-data) * [NOAA Bay Area weather data](#noaa-bay-area-weather-data) * [European Union wind data](#european-union-wind-data) * [Bitcoin price data](#bitcoin-price-data) * [Random numbers sample data](#random-numbers-sample-data) ## Home sensor data Includes simulated hourly home sensor data with anomalous sensor readings to demonstrate processing and alerting on time series data. To customize timestamps in the dataset, use the button in the lower right corner of the page. ##### Time Range **2022-01-01T08:00:00Z**to**2022-01-01T20:00:00Z***(Customizable)* ##### Schema * home *(table)* * **tags**: * room * Kitchen * Living Room * **fields**: * co *(integer)* * temp *(float)* * hum *(float)* [](#write-home-sensor-data-to-influxdb) Write home sensor data to InfluxDB #### Write the home sensor data to InfluxDB Use the `influxdb3` CLI, InfluxDB v3 API, InfluxDB v2 API, or InfluxDB v1 API to write the home sensor sample data to InfluxDB 3 Enterprise. [influxdb3](#influxdb3)[v3 API](#v3-api)[v2 API](#v2-api)[v1 API](#v1-api) ```bash influxdb3 write \ --token AUTH_TOKEN \ --database DATABASE_NAME \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200' ``` ```bash curl --request POST \ http://localhost:8181/api/v3/write_lp?db=DATABASE_NAME&precision=auto&accept_partial=false \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-raw "home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200" ``` ```bash curl --request POST \ http://localhost:8181/api/v2/write?bucket=DATABASE_NAME&precision=s \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary " home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200 " ``` ```bash curl --request POST \ http://localhost:8181/write?db=DATABASE_NAME&precision=s \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary " home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200 " ``` Replace the following in the sample script: * `DATABASE_NAME`: the name of [database](/influxdb3/enterprise/admin/databases/) to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) ## Home sensor actions data Includes hypothetical actions triggered by data in the [home sensor data](#home-sensor-data)and is a companion dataset to that sample dataset. To customize timestamps in the dataset, use the button in the lower right corner of the page. This lets you modify the sample dataset to stay within the retention period of the database you write it to. ##### Time Range **2022-01-01T08:00:00Z**to**2022-01-01T20:00:00Z***(Customizable)* ##### Schema * home\_actions *(table)* * **tags**: * room * Kitchen * Living Room * action * alert * cool * level * ok * warn * **fields**: * description *(string)* [](#write-home-sensor-actions-data-to-influxdb) Write home sensor actions data to InfluxDB #### Write the home sensor actions data to InfluxDB Use the `influxdb3` CLI, InfluxDB v3 API, InfluxDB v2 API, or InfluxDB v1 API to write the home sensor actions sample data to InfluxDB 3 Enterprise. [influxdb3](#influxdb3-2)[v3 API](#v3-api-2)[v2 API](#v2-api-2)[v1 API](#v1-api-2) ```sh influxdb3 write \ --token AUTH_TOKEN \ --database DATABASE_NAME \ 'home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23°C). Cooling to 22°C." 1641027600 home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23.3°C). Cooling to 22°C." 1641060000 home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23.1°C). Cooling to 22°C." 1641063600 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 18 ppm." 1641060000 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 22 ppm." 1641063600 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 26 ppm." 1641067200 home_actions,room=Living\ Room,action=alert,level=warn description="Carbon monoxide level above normal: 14 ppm." 1641063600 home_actions,room=Living\ Room,action=alert,level=warn description="Carbon monoxide level above normal: 17 ppm." 1641067200' ``` ```sh curl --request POST \ http://localhost:8181/api/v3/write_lp?db=DATABASE_NAME&precision=auto&accept_partial=false \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-raw "home_actions,room=Kitchen,action=cool,level=ok description=\"Temperature at or above 23°C (23°C). Cooling to 22°C.\" 1739437200 home_actions,room=Kitchen,action=cool,level=ok description=\"Temperature at or above 23°C (23.3°C). Cooling to 22°C.\" 1739469600 home_actions,room=Kitchen,action=cool,level=ok description=\"Temperature at or above 23°C (23.1°C). Cooling to 22°C.\" 1739473200 home_actions,room=Kitchen,action=alert,level=warn description=\"Carbon monoxide level above normal: 18 ppm.\" 1739469600 home_actions,room=Kitchen,action=alert,level=warn description=\"Carbon monoxide level above normal: 22 ppm.\" 1739473200 home_actions,room=Kitchen,action=alert,level=warn description=\"Carbon monoxide level above normal: 26 ppm.\" 1739476800 home_actions,room=Living Room,action=alert,level=warn description=\"Carbon monoxide level above normal: 14 ppm.\" 1739473200 home_actions,room=Living Room,action=alert,level=warn description=\"Carbon monoxide level above normal: 17 ppm.\" 1739476800" ``` ```sh curl --request POST \ http://localhost:8181/api/v2/write?bucket=DATABASE_NAME&precision=s \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary ' home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23°C). Cooling to 22°C." 1641027600 home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23.3°C). Cooling to 22°C." 1641060000 home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23.1°C). Cooling to 22°C." 1641063600 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 18 ppm." 1641060000 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 22 ppm." 1641063600 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 26 ppm." 1641067200 home_actions,room=Living\ Room,action=alert,level=warn description="Carbon monoxide level above normal: 14 ppm." 1641063600 home_actions,room=Living\ Room,action=alert,level=warn description="Carbon monoxide level above normal: 17 ppm." 1641067200 ' ``` ```sh curl --request POST \ http://localhost:8181/write?db=DATABASE_NAME&precision=s \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary ' home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23°C). Cooling to 22°C." 1641027600 home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23.3°C). Cooling to 22°C." 1641060000 home_actions,room=Kitchen,action=cool,level=ok description="Temperature at or above 23°C (23.1°C). Cooling to 22°C." 1641063600 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 18 ppm." 1641060000 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 22 ppm." 1641063600 home_actions,room=Kitchen,action=alert,level=warn description="Carbon monoxide level above normal: 26 ppm." 1641067200 home_actions,room=Living\ Room,action=alert,level=warn description="Carbon monoxide level above normal: 14 ppm." 1641063600 home_actions,room=Living\ Room,action=alert,level=warn description="Carbon monoxide level above normal: 17 ppm." 1641067200 ' ``` Replace the following in the sample script: * `DATABASE_NAME`: the name of database to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) ## NOAA Bay Area weather data Includes daily weather metrics from three San Francisco Bay Area airports from**January 1, 2020 to December 31, 2022**. This sample dataset includes seasonal trends and is good for exploring time series use cases that involve seasonality. ##### Time Range **2020-01-01T00:00:00Z** to **2022-12-31T00:00:00Z** ##### Schema * weather *(table)* * **tags**: * location * Concord * Hayward * San Francisco * **fields** * precip *(float)* * temp\_avg *(float)* * temp\_max *(float)* * temp\_min *(float)* * wind\_avg *(float)* [](#write-the-noaa-bay-area-weather-data-to-influxdb) Write the NOAA Bay Area weather data to InfluxDB #### Write the NOAA Bay Area weather data to InfluxDB Use the `influxdb3` CLI, InfluxDB v3 API, InfluxDB v2 API, or InfluxDB v1 API to write the NOAA Bay Area weather sample data to InfluxDB 3 Enterprise. [influxdb3](#influxdb3-3)[v3 API](#v3-api-3)[v2 API](#v2-api-3)[v1 API](#v1-api-3) ```sh influxdb3 write \ --token AUTH_TOKEN \ --database DATABASE_NAME \ "$(curl --request GET https://docs.influxdata.com/downloads/bay-area-weather.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v3/write_lp?db=DATABASE_NAME&precision=auto&accept_partial=false \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/bay-area-weather.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v2/write?bucket=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/bay-area-weather.lp)" ``` ```sh curl --request POST \ http://localhost:8181/write?db=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/bay-area-weather.lp)" ``` Replace the following in the sample script: * `DATABASE_NAME`: the name of database to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) ## European Union wind data The European Union (EU) wind sample dataset provides hourly measurements of wind speed and wind direction from various cities in the EU. The dataset includes a hierarchical tag set of country, county, and city. ##### Time Range **2025-10-01T00:00:00Z** to **2025-10-01T23:00:00Z** ##### Schema * wind\_data *(table)* * **tags**: * country * *20 countries* * county * *111 counties* * city * *129 cities* * **fields**: * wind\_speed *(float)* * wind\_direction *(integer)* [](#write-the-eu-wind-sample-data-to-influxdb) Write the EU wind sample data to InfluxDB #### Write the EU wind sample data to InfluxDB Use the `influxdb3` CLI, InfluxDB v3 API, InfluxDB v2 API, or InfluxDB v1 API to write the EU wind sample data to InfluxDB 3 Enterprise. [influxdb3](#influxdb3-4)[v3 API](#v3-api-4)[v2 API](#v2-api-4)[v1 API](#v1-api-4) ```sh influxdb3 write \ --token AUTH_TOKEN \ --database DATABASE_NAME \ "$(curl --request GET https://docs.influxdata.com/downloads/eu-wind-data.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v3/write_lp?db=DATABASE_NAME&precision=auto&accept_partial=false \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/eu-wind-data.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v2/write?bucket=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/eu-wind-data.lp)" ``` ```sh curl --request POST \ http://localhost:8181/write?db=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/eu-wind-data.lp)" ``` Replace the following in the sample script: * `DATABASE_NAME`: the name of database to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) ## Bitcoin price data The Bitcoin price sample dataset provides Bitcoin prices from**2023-05-01T00:00:00Z to 2023-05-15T00:00:00Z**—*[Powered by CoinDesk](https://www.coindesk.com/price/bitcoin)*. ##### Time Range **2023-05-01T00:19:00Z** to **2023-05-14T23:48:00Z** ##### Schema * bitcoin *(table)* * **tags**: * code * EUR * GBP * USD * crypto * bitcoin * description * Euro * British Pound Sterling * United States Dollar * symbol * € *(€)* * £ *(£)* * $ *($)* * **fields** * price *(float)* [](#write-the-bitcoin-sample-data-to-influxdb) Write the Bitcoin sample data to InfluxDB #### Write the Bitcoin price sample data to InfluxDB Use the `influxdb3` CLI, InfluxDB v3 API, InfluxDB v2 API, or InfluxDB v1 API to write the Bitcoin price sample data to InfluxDB 3 Enterprise. [influxdb3](#influxdb3-5)[v3 API](#v3-api-5)[v2 API](#v2-api-5)[v1 API](#v1-api-5) ```sh influxdb3 write \ --token AUTH_TOKEN \ --database DATABASE_NAME \ "$(curl --request GET https://docs.influxdata.com/downloads/bitcoin.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v3/write_lp?db=DATABASE_NAME&precision=auto&accept_partial=false \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/bitcoin.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v2/write?bucket=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/bitcoin.lp)" ``` ```sh curl --request POST \ http://localhost:8181/write?db=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/bitcoin.lp)" ``` Replace the following in the sample script: * `DATABASE_NAME`: the name of database to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) ## Random numbers sample data Includes two fields with randomly generated numbers reported every minute. Each field has a specific range of randomly generated numbers. This sample dataset is used to demonstrate mathematic operations and transformation functions. ##### Time Range **2023-01-01T00:00:00Z** to **2023-01-01T12:00:00Z** ##### Schema * numbers *(table)* * **fields** * a *(float between -1 and 1)* * b *(float between -3 and 3)* [](#write-the-random-number-sample-data-to-influxdb) Write the random number sample data to InfluxDB #### Write the random number sample data to InfluxDB Use the `influxdb3` CLI, InfluxDB v3 API, InfluxDB v2 API, or InfluxDB v1 API to write the random number sample data to InfluxDB 3 Enterprise. [influxdb3](#influxdb3-6)[v3 API](#v3-api-6)[v2 API](#v2-api-6)[v1 API](#v1-api-6) ```sh influxdb3 write \ --token AUTH_TOKEN \ --database DATABASE_NAME \ "$(curl --request GET https://docs.influxdata.com/downloads/random-numbers.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v3/write_lp?db=DATABASE_NAME&precision=auto&accept_partial=false \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/random-numbers.lp)" ``` ```sh curl --request POST \ http://localhost:8181/api/v2/write?bucket=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Accept: application/json" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/random-numbers.lp)" ``` ```sh curl --request POST \ http://localhost:8181/write?db=DATABASE_NAME \ --header "Authorization: Bearer AUTH_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary "$(curl --request GET https://docs.influxdata.com/downloads/random-numbers.lp)" ``` Replace the following in the sample script: * `DATABASE_NAME`: the name of database to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) --- # SQL reference documentation Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/ InfluxDB 3 Enterprise uses the [Apache Arrow DataFusion](https://arrow.apache.org/datafusion/) implementation of SQL. ## Identifiers An identifier is a token which refers to the name of an InfluxDB database object, such as a **measurement** or a column name (**time**, **tag keys**, or **field keys**). ## Quoting Use double quotes on [identifiers](#identifiers) to treat them as case-sensitive. Use single quotes on string literals. General quoting guidelines: * Single quote RFC3339 and RFC3339-like time values. * Do *not* quote Unix epoch time values (integers cast to a timestamp). * Double-quote mixed case, [camel case](https://en.wikipedia.org/wiki/Camel_case) or case-sensitive identifiers. * Double-quote identifiers that contain special characters or whitespace characters. ##### Quoting examples ```sql -- Double-quote identifiers that contain whitespace SELECT "water temperature", "buoy location" FROM buoy -- Double-quote measurement names with special characters SELECT * FROM "h2o-temperature" -- Double-quote identifiers that should be treated as case-sensitive SELECT "pH" FROM "Water" ``` > [!Note] > **Note:** We recommend always double-quoting identifiers, regardless of case-sensitivity. Unquoted identifiers **are not** case-sensitive and match any measurement, tag key, or field key with the same characters, despite case. For example, if you have two fields in a measurement named `ph` and `pH`, the unquoted identifier, `pH` will match both. To query in a case-sensitive manner, double-quote identifiers. ## Literals A literal is an explicit value not represented by an identifier. ### String literals String literals are surrounded by single quotes. ```sql 'santa_monica' 'pH' 'average temperature' ``` ### Numeric literals Number literals are positive or negative numbers that are either exact numbers or floats. ```sql -- Integers 10 +10 -10 -- Unsigned integers 10::BIGINT UNSIGNED +10::BIGINT UNSIGNED -- Floats 10.78654 -100.56 ``` ### Date and time literals The following date and time literals are supported: ``` '2022-01-31T06:30:30.123Z' -- (RFC3339) '2022-01-31T06:30:30.123' -- (RFC3339-like) '2022-01-31 06:30:30.123' -- (RFC3339-like) '2022-01-31 06:30:30' -- ((RFC3339-like, no fractional seconds) 1643610630123000000::TIMESTAMP -- (Unix epoch nanosecond cast to a timestamp) ``` ### Boolean literals Boolean literals are either `TRUE` or `FALSE`. ## Duration units Interval literals specify a length or unit of time. ```sql INTERVAL '4 minutes' INTERVAL '12 days 6 hours 30 minutes' ``` The following units of time are supported: * nanoseconds * microseconds * milliseconds * seconds * minutes * hours * days * weeks * months * years * century ## Operators Operators are reserved words or characters which perform certain operations, including comparisons and arithmetic. ### Arithmetic operators Arithmetic operators take two numeric values (either literals or variables) and perform a calculation that returns a single numeric value. |Operator| Description |Example|Result| |--------|--------------|-------|------| | `+` | Addition |`2 + 2`| `4` | | `-` | Subtraction |`4 - 2`| `2` | | `*` |Multiplication|`2 * 3`| `6` | | `/` | Division |`6 / 3`| `2` | | `%` | Modulo |`7 % 2`| `1` | ### Comparison operators Comparison operators evaluate the relationship between the left and right operands and `TRUE` or `FALSE`. |Operator| Meaning | Example | |--------|--------------------------------------------------------|------------------| | `=` | Equal to | `123 = 123` | | `<>` | Not equal to | `123 <> 456` | | `!=` | Not equal to | `123 != 456` | | `>` | Greater than | `3 > 2` | | `>=` | Greater than or equal to | `3 >= 2` | | `<` | Less than | `1 < 2` | | `<=` | Less than or equal to | `1 <= 2` | | `~` | Matches a regular expression | `'abc' ~ 'a.*'` | | `~\*` | Matches a regular expression *(case-insensitive)* |`'Abc' ~\* 'A.*'` | | `!~` | Does not match a regular expression | `'abc' !~ 'd.*'` | | `!~\*` |Does not match a regular expression *(case-insensitive)*|`'Abc' !~\* 'a.*'`| ### Logical operators |Operator | Meaning | |---------|--------------------------------------------------------------------------| | `AND` | Returns true if both operands are true. Otherwise, returns false. | |`BETWEEN`|Returns true if the left operand is within the range of the right operand.| |`EXISTS` | Returns true if the operand is not null. | | `IN` | Returns true if the left operand is in the right operand list. | | `LIKE` |Returns true if the left operand matches the right operand pattern string.| | `NOT` | Negates the subsequent expression. | | `OR` | Returns true if any operand is true. Otherwise, returns false. | ### Bitwise operators Bitwise operators perform bitwise operations on bit patterns or binary numerals. |Operator| Meaning |Example |Result| |--------|-------------------|--------|------| | `&` | Bitwise and |`5 & 3` | `1` | | `|` | Bitwise or |`5 | 3` | `7` | | `^` | Bitwise xor |`5 ^ 3` | `6` | | `>>` |Bitwise shift right|`5 >> 3`| `0` | | `<<` |Bitwise shift left |`5 << 3`| `40` | ### Other operators | Operator | Meaning | Example | Result | |--------------|------------------------|-----------------------------------------------------------------------------------|-------------| | `||` | Concatenates strings | `'Hello' || ' world'` |`Hello world`| |`AT TIME ZONE`|Apply a time zone offset|*[View example](/influxdb3/enterprise/reference/sql/operators/other/#at-time-zone)*| | ## Keywords The following reserved keywords cannot be used as identifiers. ```sql AND ALL ANALYZE AS ASC AT TIME ZONE BETWEEN BOTTOM CASE DESC DISTINCT EXISTS EXPLAIN FROM GROUP BY HAVING IN INNER JOIN JOIN LEFT JOIN LIKE LIMIT NOT EXISTS NOT IN OR ORDER BY FULL OUTER JOIN RIGHT JOIN SELECT TOP TYPE UNION UNION ALL WHERE WITH ``` ## Conditional expressions Conditional expressions evaluate conditions based on input values. The following conditional expressions are supported: |Expression| Description | |----------|------------------------------------------------------------------| | CASE | Allows for use of WHEN-THEN-ELSE statements. | | COALESCE | Returns the first non-NULL expression in a specified list. | | NULLIF |Returns a NULL value if value1 = value2. Otherwise returns value1.| ## Statements and clauses InfluxDB SQL supports the following basic syntax for queries: ``` [ WITH with_query [, …] ] SELECT [ ALL | DISTINCT ] select_expr [, …] [ FROM from_item [, …] ] [ JOIN join_item [, …] ] [ WHERE condition ] [ GROUP BY grouping_element [, …] ] [ HAVING condition] [ UNION [ ALL ] ] [ ORDER BY expression [ ASC | DESC ][, …] ] [ LIMIT count ] ``` ### SELECT statement and FROM clause Use the SQL `SELECT` statement to query data from a specific measurement or measurements. The `FROM` clause always accompanies the `SELECT` statement. #### Examples ```sql SELECT * FROM "h2o_feet" ``` ### WHERE clause Use the `WHERE` clause to filter results based on `fields`, `tags`, and `timestamps`. Use predicates to evaluate each row. Rows that evaluate as `TRUE` are returned in the result set. Rows that evaluate as `FALSE` are omitted from the result set. #### Examples ```sql SELECT * FROM "h2o_feet" WHERE "water_level" <= 9 ``` ``` SELECT * FROM "h2o_feet" WHERE "location" = 'santa_monica' AND "level description" = 'below 3 feet' ``` ### JOIN clause Use the `JOIN` clause to join data from multiple measurements (tables). For more information about joins, see[JOIN clause](/influxdb3/enterprise/reference/sql/join/). The following join types are supported: [ **INNER JOIN** ](#inner-join) [ **LEFT [OUTER] JOIN** ](#left-outer-join) [ **RIGHT [OUTER] JOIN** ](#right-outer-join) [ **FULL [OUTER] JOIN** ](#full-outer-join) [](#inner-join) INNER JOIN Inner joins combine rows from tables on the left and right side of the join based on common column values defined in the `ON` clause. Rows that don’t have matching column values are not included in the output table. ```sql SELECT * FROM home INNER JOIN home_actions ON home.room = home_actions.room AND home.time = home_actions.time; ``` [](#left-outer-join) LEFT [OUTER] JOIN A left outer join returns all rows from the left side of the join and only returns data from the right side of the join in rows with matching column values defined in the `ON` clause. ```sql SELECT * FROM home LEFT OUTER JOIN home_actions ON home.room = home_actions.room AND home.time = home_actions.time; ``` [](#right-outer-join) RIGHT [OUTER] JOIN A right outer join returns all rows from the right side of the join and only returns data from the left side of the join in rows with matching column values defined in the `ON` clause. ```sql SELECT * FROM home RIGHT OUTER JOIN home_actions ON home.room = home_actions.room AND home.time = home_actions.time; ``` [](#full-outer-join) FULL [OUTER] JOIN A full outer join returns all data from the left and right sides of the join and combines rows with matching column values defined in the `ON` clause. ```sql SELECT * FROM home FULL OUTER JOIN home_actions ON home.room = home_actions.room AND home.time = home_actions.time; ``` ### GROUP BY clause Use the `GROUP BY` clause to group query results based on specified column values. `GROUP BY` **requires** an aggregate or selector function in the `SELECT` statement. #### Examples ``` SELECT MEAN("water_level"), "location" FROM "h2o_feet" GROUP BY "location" ``` ### HAVING clause Use the `HAVING` clause to filter query results based on a specified condition. The `HAVING` clause must *follow* the `GROUP BY` clause, but *precede* the `ORDER BY` clause. #### Examples ``` SELECT MEAN("water_level"), "location" FROM "h2o_feet" GROUP BY "location" HAVING MEAN("water_level") > 4 ORDER BY "location" ``` ### UNION clause The `UNION` clause combines the results of two or more `SELECT` statements without returning any duplicate rows. `UNION ALL` returns all results, including duplicates. #### Examples ``` SELECT 'pH' FROM "h2o_pH" UNION ALL SELECT "location" FROM "h2o_quality" ``` ### ORDER BY clause The `ORDER BY` clause orders results by specified columns and order. Sort data based on fields, tags, and timestamps. The following orders are supported: * `ASC`: ascending *(default)* * `DESC`: descending #### Examples ``` SELECT "water_level", "location" FROM "h2o_feet" ORDER BY "location", "time" DESC ``` ### LIMIT clause The `LIMIT` clause limits the number of rows to return. The defined limit should be a non-negative integer. #### Examples ``` SELECT "water_level", "location" FROM "h2o_feet" LIMIT 10 ``` ### WITH clause The `WITH` clause provides a way to write auxiliary statements for use in a larger query. It can help break down large, complicated queries into simpler forms. ``` WITH summary_data as (SELECT degrees, location, time FROM average_temperature) SELECT * FROM summary_data ``` ### OVER clause The `OVER` clause is used with SQL window functions. A **window function** performs a calculation across a set of table rows that are related in some way to the current row. While similar to aggregate functions, window functions output results into rows retaining their separate identities. ``` SELECT time, water_level FROM ( SELECT time, "water_level", row_number() OVER ( order by water_level desc ) as rn FROM h2o_feet ) WHERE rn <= 3; ``` ## Comments Use comments to describe and add detail or notes to your queries. * Single line comments use the double hyphen `--` symbol. Single line comments end with a line break. * Multi-line comments begin with `/*` and end with `*/`. ``` -- Single-line comment /* * Multi-line comment */ ``` ## Schema information InfluxDB 3 Enterprise supports the following metadata schema queries: ```sql SHOW tables SHOW columns FROM ``` ## Functions Following is a list of supported functions by type. ### Aggregate functions An aggregate function performs a calculation or computation on a set of data values in a column and returns a single value. |Function| Description | |--------|----------------------------------------------------------| |COUNT() |Returns returns the number of rows from a field or tag key| | AVG() | Returns the average value of a column | | SUM() | Returns the summed value of a column | | MEAN() | Returns the mean value of a column | | MIN() | Returns the smallest value of the selected column | | MAX() | Returns the largest value of the selected column | #### Examples ``` SELECT COUNT("water_level") FROM "h2o_feet" SELECT AVG("water_level"), "location" FROM "h2o_feet" GROUP BY "location" SELECT SUM("water_level"), "location" FROM "h2o_feet" GROUP BY "location" ``` ### Window aggregate functions Window functions let you calculate running totals, moving averages, or other aggregate-like results without collapsing rows into groups (unlike non-window aggregate functions). Window aggregate functions include **all [aggregate functions](#aggregate-functions)**and the [ranking functions](#ranking-functions). The SQL `OVER` clause syntactically distinguishes a window function from a non-window or aggregate function and defines how to group and order rows for the window operation. #### Examples: ```sql SELECT time, room, temp, avg(temp) OVER (PARTITION BY room) AS avg_room_temp FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T09:00:00Z' ORDER BY room, time ``` | time | room |temp|avg\_room\_temp| |-------------------|-----------|----|---------------| |2022-01-01T08:00:00| Kitchen |21.0| 22.0 | |2022-01-01T09:00:00| Kitchen |23.0| 22.0 | |2022-01-01T08:00:00|Living Room|21.1| 21.25 | |2022-01-01T09:00:00|Living Room|21.4| 21.25 | #### Ranking Functions | Function | Description | |---------------|---------------------------------------------------------------------------------| | CUME\_DIST() | Returns the cumulative distribution of a value within a group of values | | DENSE\_RANK() | Returns a rank for each row without gaps in the numbering | | NTILE() |Distributes the rows in an ordered partition into the specified number of groups | |PERCENT\_RANK()| Returns the percentage rank of the current row within its partition | | RANK() |Returns the rank of the current row in its partition, allowing gaps between ranks| | ROW\_NUMBER() | Returns the position of the current row in its partition | ### Selector functions Selector functions are unique to InfluxDB. They behave like aggregate functions in that they take a row of data and compute it down to a single value. However, selectors are unique in that they return a **time value** in addition to the computed value. In short, selectors return an aggregated value along with a timestamp. | Function | Description | |-----------------|--------------------------------------------------------------| |SELECTOR\_FIRST()| Returns the first value of a selected column and timestamp. | |SELECTOR\_LAST() | Returns the last value of a selected column and timestamp. | | SELECTOR\_MIN() |Returns the smallest value of a selected column and timestamp.| | SELECTOR\_MAX() |Returns the largest value of a selected column and timestamp. | #### Examples ``` SELECT SELECTOR_MAX("pH", time)['value'], SELECTOR_MAX("pH", time)['time'] FROM "h2o_pH" SELECT SELECTOR_LAST("water_level", time)['value'], SELECTOR_LAST("water_level", time)['time'] FROM "h2o_feet" WHERE time >= timestamp '2019-09-10T00:00:00Z' AND time <= timestamp '2019-09-19T00:00:00Z' ``` ### Date and time functions | Function | Description | |-------------|-----------------------------------------------------------------------------------------------| | DATE\_BIN() | Bins the input timestamp into a specified interval. | |DATE\_TRUNC()|Truncates a timestamp expression based on the date part specified, such as hour, day, or month.| |DATE\_PART() | Returns the specified part of a date. | | NOW() | Returns the current time (UTC). | #### Examples ```sql SELECT DATE_BIN(INTERVAL '1 hour', time, '2019-09-18T00:00:00Z') AS "_time", SUM(water_level) FROM "h2o_feet" GROUP BY "_time" ``` ```sql SELECT DATE_TRUNC('month',time) AS "date", SUM(water_level) FROM "h2o_feet" GROUP BY time ``` ### Approximate functions | Function | Description | |--------------------------------------|---------------------------------------------------------------------------------------------| | APPROX\_MEDIAN | Returns the approximate median of input values. | | APPROX\_DISTINCT |Returns the approximate count of the number of distinct values. Implemented only for strings.| | APPROX\_PERCENTILE\_CONT | Returns the approximate percentile of input values. | |APPROX\_PERCENTILE\_CONT\_WITH\_WEIGHT| Returns the approximate percentile of input values with weight. | ### Math functions |Function| Description | |--------|--------------------------------------------------------------------------------| | ABS() | Absolute value | | ACOS() | Inverse cosine | | ASIN() | Inverse sine | | ATAN() | Inverse tangent | |ATAN2() | Inverse tangent of y / x | | CEIL() |Returns the smallest integer value greater than or equal to the specified number| | COS() | Cosine | | EXP() | Exponential | |FLOOR() | Nearest integer less than or equal to the specified number | | LN() | Natural logarithm | |LOG10() | Base 10 logarithm | | LOG2() | Base 2 logarithm | |POWER() | Returns the value of a number raised to the power of the number | |ROUND() | Round to the nearest integer | |SIGNUM()| Sign of the argument (-1, 0, +1) | | SINE() | Sine | | SQRT() | Returns the square root of a number | | TAN() | Tangent | |TRUNC() | Truncates a number to the specified number of decimal places | ### Conditional functions |Function| Description | |--------|----------------------------------------------------------------------------------------------------------| |COALESCE|Returns the first argument that is not null. If all arguments are null, then `COALESCE` will return nulls.| | NULLIF | Returns a null value if value1 equals value2, otherwise returns value1. | ### Regular expression functions | Function | Description | |---------------|-----------------------------------------------------------------------------| | REGEXP\_MATCH |Matches a regular expression against a string and returns matched substrings.| |REGEXP\_REPLACE| Replaces substrings that match a regular expression by a new substring. | #### Related | Operator | Description | Example | Result | | --- | --- | --- | --- | | Operator | Description | Example | Result | | + | Addition | 2 + 2 | 4 | | - | Subtraction | 4 - 2 | 2 | | * | Multiplication | 2 * 3 | 6 | | / | Division | 6 / 3 | 2 | | % | Modulo | 7 % 2 | 1 | | Operator | Meaning | Example | | --- | --- | --- | | Operator | Meaning | Example | | = | Equal to | 123 = 123 | | <> | Not equal to | 123 <> 456 | | != | Not equal to | 123 != 456 | | > | Greater than | 3 > 2 | | >= | Greater than or equal to | 3 >= 2 | | < | Less than | 1 < 2 | | <= | Less than or equal to | 1 <= 2 | | ~ | Matches a regular expression | 'abc' ~ 'a.*' | | ~\* | Matches a regular expression (case-insensitive) | 'Abc' ~\* 'A.*' | | !~ | Does not match a regular expression | 'abc' !~ 'd.*' | | !~\* | Does not match a regular expression (case-insensitive) | 'Abc' !~\* 'a.*' | | Operator | Meaning | | --- | --- | | Operator | Meaning | | AND | Returns true if both operands are true. Otherwise, returns false. | | BETWEEN | Returns true if the left operand is within the range of the right operand. | | EXISTS | Returns true if the operand is not null. | | IN | Returns true if the left operand is in the right operand list. | | LIKE | Returns true if the left operand matches the right operand pattern string. | | NOT | Negates the subsequent expression. | | OR | Returns true if any operand is true. Otherwise, returns false. | | Operator | Meaning | Example | Result | | --- | --- | --- | --- | | Operator | Meaning | Example | Result | | & | Bitwise and | 5 & 3 | 1 | | | | Bitwise or | 5 | 3 | 7 | | ^ | Bitwise xor | 5 ^ 3 | 6 | | >> | Bitwise shift right | 5 >> 3 | 0 | | << | Bitwise shift left | 5 << 3 | 40 | | Operator | Meaning | Example | Result | | --- | --- | --- | --- | | Operator | Meaning | Example | Result | | || | Concatenates strings | 'Hello' || ' world' | Hello world | | AT TIME ZONE | Apply a time zone offset | View example | | | Expression | Description | | --- | --- | | Expression | Description | | CASE | Allows for use of WHEN-THEN-ELSE statements. | | COALESCE | Returns the first non-NULL expression in a specified list. | | NULLIF | Returns a NULL value if value1 = value2. Otherwise returns value1. | | Function | Description | | --- | --- | | Function | Description | | COUNT() | Returns returns the number of rows from a field or tag key | | AVG() | Returns the average value of a column | | SUM() | Returns the summed value of a column | | MEAN() | Returns the mean value of a column | | MIN() | Returns the smallest value of the selected column | | MAX() | Returns the largest value of the selected column | | time | room | temp | avg_room_temp | | --- | --- | --- | --- | | time | room | temp | avg_room_temp | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 22.0 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 22.0 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 21.25 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.25 | | Function | Description | | --- | --- | | Function | Description | | CUME_DIST() | Returns the cumulative distribution of a value within a group of values | | DENSE_RANK() | Returns a rank for each row without gaps in the numbering | | NTILE() | Distributes the rows in an ordered partition into the specified number of groups | | PERCENT_RANK() | Returns the percentage rank of the current row within its partition | | RANK() | Returns the rank of the current row in its partition, allowing gaps between ranks | | ROW_NUMBER() | Returns the position of the current row in its partition | | Function | Description | | --- | --- | | Function | Description | | SELECTOR_FIRST() | Returns the first value of a selected column and timestamp. | | SELECTOR_LAST() | Returns the last value of a selected column and timestamp. | | SELECTOR_MIN() | Returns the smallest value of a selected column and timestamp. | | SELECTOR_MAX() | Returns the largest value of a selected column and timestamp. | | Function | Description | | --- | --- | | Function | Description | | DATE_BIN() | Bins the input timestamp into a specified interval. | | DATE_TRUNC() | Truncates a timestamp expression based on the date part specified, such as hour, day, or month. | | DATE_PART() | Returns the specified part of a date. | | NOW() | Returns the current time (UTC). | | Function | Description | | --- | --- | | Function | Description | | APPROX_MEDIAN | Returns the approximate median of input values. | | APPROX_DISTINCT | Returns the approximate count of the number of distinct values. Implemented only for strings. | | APPROX_PERCENTILE_CONT | Returns the approximate percentile of input values. | | APPROX_PERCENTILE_CONT_WITH_WEIGHT | Returns the approximate percentile of input values with weight. | | Function | Description | | --- | --- | | Function | Description | | ABS() | Absolute value | | ACOS() | Inverse cosine | | ASIN() | Inverse sine | | ATAN() | Inverse tangent | | ATAN2() | Inverse tangent of y / x | | CEIL() | Returns the smallest integer value greater than or equal to the specified number | | COS() | Cosine | | EXP() | Exponential | | FLOOR() | Nearest integer less than or equal to the specified number | | LN() | Natural logarithm | | LOG10() | Base 10 logarithm | | LOG2() | Base 2 logarithm | | POWER() | Returns the value of a number raised to the power of the number | | ROUND() | Round to the nearest integer | | SIGNUM() | Sign of the argument (-1, 0, +1) | | SINE() | Sine | | SQRT() | Returns the square root of a number | | TAN() | Tangent | | TRUNC() | Truncates a number to the specified number of decimal places | | Function | Description | | --- | --- | | Function | Description | | COALESCE | Returns the first argument that is not null. If all arguments are null, then COALESCE will return nulls. | | NULLIF | Returns a null value if value1 equals value2, otherwise returns value1. | | Function | Description | | --- | --- | | Function | Description | | REGEXP_MATCH | Matches a regular expression against a string and returns matched substrings. | | REGEXP_REPLACE | Replaces substrings that match a regular expression by a new substring. | --- # SQL data types Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/data-types/ InfluxDB 3 Enterprise uses the [Apache Arrow DataFusion](https://arrow.apache.org/datafusion/)implementation of SQL. Data types define the type of values that can be stored in table columns. In InfluxDB’s SQL implementation, a **measurement** is structured as a table, and **tags**, **fields** and **timestamps** are exposed as columns. ## SQL and Arrow data types In SQL, each column, expression, and parameter has a data type. A data type is an attribute that specifies the type of data that the object can hold. DataFusion uses the [Arrow](https://arrow.apache.org/) type system for query execution. All SQL types are mapped to [Arrow data types](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html). Both SQL and Arrow data types play an important role in how data is operated on during query execution and returned in query results. > [!Note] > When performing casting operations, cast to the SQL data type unless you use[`arrow_cast()`](/influxdb3/enterprise/reference/sql/functions/misc/#arrow_cast)to cast to a specific Arrow type. > Names and identifiers in SQL are *case-insensitive* by default. For example: > > ``` > SELECT > '99'::BIGINT, > '2019-09-18T00:00:00Z'::timestamp > ``` ## String types |SQL data type|Arrow data type| Description | |-------------|---------------|---------------------------------| | STRING | UTF8 |Character string, variable-length| | CHAR | UTF8 | Character string, fixed-length | | VARCHAR | UTF8 |Character string, variable-length| | TEXT | UTF8 | Variable unlimited length | ##### Example string literals ```sql 'abcdefghijk' 'time' 'h2o_temperature' ``` ## Numeric types The following numeric types are supported: | SQL data type |Arrow data type| Description | |---------------|---------------|----------------------------------------------| | BIGINT | INT64 | 64-bit signed integer | |BIGINT UNSIGNED| UINT64 | 64-bit unsigned integer | | DOUBLE | FLOAT64 | 64-bit floating-point number (\~15 digits) | | FLOAT | FLOAT32 | 32-bit floating-point number (\~7 digits) | | REAL | FLOAT32 |32-bit floating-point number (alias for FLOAT)| ### Integers InfluxDB SQL supports the 64-bit signed integers: **Minimum signed integer**: `-9223372036854775808`**Maximum signed integer**: `9223372036854775807` ##### Example integer literals ```sql 234 -446 5 ``` ### Unsigned integers InfluxDB SQL supports the 64-bit unsigned integers: **Minimum unsigned integer**: `0`**Maximum unsigned integer**: `18446744073709551615` ##### Example unsigned integer literals Unsigned integer literals are comprised of an integer cast to the `BIGINT UNSIGNED` type: ```sql 234::BIGINT UNSIGNED 458374893::BIGINT UNSIGNED 5::BIGINT UNSIGNED ``` ### Floats InfluxDB SQL supports both 32-bit (single precision) and 64-bit (double precision) floating-point values. | Type |Precision|Significant Digits| Use Case | |------|---------|------------------|---------------------------------------------------------| |FLOAT | 32-bit | \~7 digits |Memory-efficient storage when full precision isn’t needed| |DOUBLE| 64-bit | \~15-16 digits | Default for most numeric operations | > [!Note] > InfluxDB stores float field values as 64-bit (FLOAT64) internally. > Casting to FLOAT (32-bit) may lose precision for values with more than \~7 significant digits. > Unlike PostgreSQL where FLOAT defaults to double precision, InfluxDB SQL treats FLOAT as single precision (32-bit). ##### Example float literals Float literals are stored as 64-bit double precision: ```sql 23.8 -446.89 5.00 0.033 ``` ##### Example float casting ```sql -- Cast to 32-bit float (may lose precision) SELECT 3.141592653589793::FLOAT; -- Returns: 3.1415927 (truncated to ~7 digits) -- Cast to 64-bit double (preserves precision) SELECT 3.141592653589793::DOUBLE; -- Returns: 3.141592653589793 ``` ## Date and time data types InfluxDB SQL supports the following DATE/TIME data types: |SQL data type| Arrow data type | Description | |-------------|------------------------------|---------------------------------------------| | TIMESTAMP | Timestamp(Nanosecond, None) |Nanosecond timestamp with no time zone offset| | INTERVAL |Interval(IntervalMonthDayNano)| Interval of time with a specified duration | ### Timestamp A time type is a single point in time using nanosecond precision. The following date and time formats are supported: ```sql YYYY-MM-DDT00:00:00.000Z YYYY-MM-DDT00:00:00.000-00:00 YYYY-MM-DD 00:00:00.000-00:00 YYYY-MM-DDT00:00:00Z YYYY-MM-DD 00:00:00.000 YYYY-MM-DD 00:00:00 ``` ##### Example timestamp literals ```sql '2023-01-02T03:04:06.000Z' '2023-01-02T03:04:06.000-00:00' '2023-01-02 03:04:06.000-00:00' '2023-01-02T03:04:06Z' '2023-01-02 03:04:06.000' '2023-01-02 03:04:06' ``` ### Interval The INTERVAL data type can be used with the following precision: * nanosecond * microsecond * millisecond * second * minute * hour * day * week * month * year * century ##### Example interval literals ```sql INTERVAL '10 minutes' INTERVAL '1 year' INTERVAL '2 days 1 hour 31 minutes' ``` ## Boolean types Booleans store TRUE or FALSE values. |SQL data type|Arrow data type| Description | |-------------|---------------|--------------------| | BOOLEAN | Boolean |True or false values| ##### Example boolean literals ```sql true TRUE false FALSE ``` ## Unsupported SQL types The following SQL types are not currently supported: * UUID * BLOB * CLOB * BINARY * VARBINARY * REGCLASS * NVARCHAR * CUSTOM * ARRAY * ENUM * SET * DATETIME * BYTEA ## Data types compatible with parameters For information about data types that can be substituted by parameters, see how to [use parameterized queries with SQL](/influxdb3/enterprise/query-data/sql/parameterized-queries/). #### Related * [Cast values to different types](/influxdb3/enterprise/query-data/sql/cast-types/) | SQL data type | Arrow data type | Description | | --- | --- | --- | | SQL data type | Arrow data type | Description | | STRING | UTF8 | Character string, variable-length | | CHAR | UTF8 | Character string, fixed-length | | VARCHAR | UTF8 | Character string, variable-length | | TEXT | UTF8 | Variable unlimited length | | SQL data type | Arrow data type | Description | | --- | --- | --- | | SQL data type | Arrow data type | Description | | BIGINT | INT64 | 64-bit signed integer | | BIGINT UNSIGNED | UINT64 | 64-bit unsigned integer | | DOUBLE | FLOAT64 | 64-bit floating-point number (~15 digits) | | FLOAT | FLOAT32 | 32-bit floating-point number (~7 digits) | | REAL | FLOAT32 | 32-bit floating-point number (alias for FLOAT) | | Type | Precision | Significant Digits | Use Case | | --- | --- | --- | --- | | Type | Precision | Significant Digits | Use Case | | FLOAT | 32-bit | ~7 digits | Memory-efficient storage when full precision isn’t needed | | DOUBLE | 64-bit | ~15-16 digits | Default for most numeric operations | | SQL data type | Arrow data type | Description | | --- | --- | --- | | SQL data type | Arrow data type | Description | | TIMESTAMP | Timestamp(Nanosecond, None) | Nanosecond timestamp with no time zone offset | | INTERVAL | Interval(IntervalMonthDayNano) | Interval of time with a specified duration | | SQL data type | Arrow data type | Description | | --- | --- | --- | | SQL data type | Arrow data type | Description | | BOOLEAN | Boolean | True or false values | --- # EXPLAIN command Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/explain/ The `EXPLAIN` command returns the [logical plan](/influxdb3/enterprise/reference/internals/query-plan/#logical-plan) and the [physical plan](/influxdb3/enterprise/reference/internals/query-plan/#physical-plan) for the specified SQL statement. ```sql EXPLAIN [ANALYZE] [VERBOSE] statement ``` ## `EXPLAIN` Returns the logical plan and physical (execution) plan of a statement. To output more details, use `EXPLAIN VERBOSE`. `EXPLAIN` doesn’t execute the statement. To execute the statement and view runtime metrics, use [`EXPLAIN ANALYZE`](#explain-analyze). ### Example `EXPLAIN` ```sql EXPLAIN SELECT room, avg(temp) AS temp FROM home GROUP BY room ``` [](#view-explain-example-output) View `EXPLAIN` example output | | plan\_type | plan | |---|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 0 |logical\_plan | Projection: home.room, AVG(home.temp) AS temp | | | | Aggregate: groupBy=[[home.room]], aggr=[[AVG(home.temp)]] | | | | TableScan: home projection=[room, temp] | | 1 |physical\_plan| ProjectionExec: expr=[room@0 as room, AVG(home.temp)@1 as temp] | | | | AggregateExec: mode=FinalPartitioned, gby=[room@0 as room], aggr=[AVG(home.temp)] | | | | CoalesceBatchesExec: target\_batch\_size=8192 | | | | RepartitionExec: partitioning=Hash([room@0], 8), input\_partitions=8 | | | | AggregateExec: mode=Partial, gby=[room@0 as room], aggr=[AVG(home.temp)] | | | |ParquetExec: file\_groups={8 groups: [[70434/116281/404d73cea0236530ea94f5470701eb814a8f0565c0e4bef5a2d2e33dfbfc3567/1be334e8-0af8-00da-2615-f67cd4be90f7.parquet, 70434/116281/b7a9e7c57fbfc3bba9427e4b3e35c89e001e2e618b0c7eb9feb4d50a3932f4db/d29370d4-262f-0d32-2459-fe7b099f682f.parquet], [70434/116281/c14418ba28a22a3abb693a1cb326a63b62dc611aec58c9bed438fdafd3bc5882/8b29ae98-761f-0550-2fe4-ee77503658e9.parquet], [70434/116281/fa677477eed622ae8123da1251aa7c351f801e2ee2f0bc28c0fe3002a30b3563/65bb4dc3-04e1-0e02-107a-90cee83c51b0.parquet], [70434/116281/db162bdd30261019960dd70da182e6ebd270284569ecfb5deffea7e65baa0df9/2505e079-67c5-06d9-3ede-89aca542dd18.parquet], [70434/116281/0c025dcccae8691f5fd70b0f131eea4ca6fafb95a02f90a3dc7bb015efd3ab4f/3f3e44c3-b71e-0ca4-3dc7-8b2f75b9ff86.parquet], …]}, projection=[room, temp]| ## `EXPLAIN ANALYZE` Executes a statement and returns the execution plan and runtime metrics of the statement. The report includes the [logical plan](/influxdb3/enterprise/reference/internals/query-plan/#logical-plan) and the [physical plan](/influxdb3/enterprise/reference/internals/query-plan/#physical-plan) annotated with execution counters, number of rows produced, and runtime metrics sampled during the query execution. If the plan requires reading lots of data files, `EXPLAIN` and `EXPLAIN ANALYZE` may truncate the list of files in the report. To output more information, including intermediate plans and paths for all scanned Parquet files, use [`EXPLAIN ANALYZE VERBOSE`](#explain-analyze-verbose). ### Example `EXPLAIN ANALYZE` ```sql EXPLAIN ANALYZE SELECT room, avg(temp) AS temp FROM home WHERE time >= '2023-01-01' AND time <= '2023-12-31' GROUP BY room ``` [](#view-explain-analyze-example-output) View `EXPLAIN ANALYZE` example output | | plan\_type | plan | |---|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 0 |Plan with Metrics| ProjectionExec: expr=[room@0 as room, AVG(home.temp)@1 as temp], metrics=[output\_rows=2, elapsed\_compute=4.768µs] | | | | AggregateExec: mode=FinalPartitioned, gby=[room@0 as room], aggr=[AVG(home.temp)], ordering\_mode=Sorted, metrics=[output\_rows=2, elapsed\_compute=140.405µs] | | | | CoalesceBatchesExec: target\_batch\_size=8192, metrics=[output\_rows=2, elapsed\_compute=6.821µs] | | | | RepartitionExec: partitioning=Hash([room@0], 8), input\_partitions=8, preserve\_order=true, sort\_exprs=room@0 ASC, metrics=[output\_rows=2, elapsed\_compute=18.408µs, repart\_time=59.698µs, fetch\_time=1.057882762s, send\_time=5.83µs] | | | | AggregateExec: mode=Partial, gby=[room@0 as room], aggr=[AVG(home.temp)], ordering\_mode=Sorted, metrics=[output\_rows=2, elapsed\_compute=137.577µs] | | | | RepartitionExec: partitioning=RoundRobinBatch(8), input\_partitions=6, preserve\_order=true, sort\_exprs=room@0 ASC, metrics=[output\_rows=46, elapsed\_compute=26.637µs, repart\_time=6ns, fetch\_time=399.971411ms, send\_time=6.658µs] | | | | ProjectionExec: expr=[room@0 as room, temp@2 as temp], metrics=[output\_rows=46, elapsed\_compute=3.102µs] | | | | CoalesceBatchesExec: target\_batch\_size=8192, metrics=[output\_rows=46, elapsed\_compute=25.585µs] | | | | FilterExec: time@1 \>= 1672531200000000000 AND time@1 \<= 1703980800000000000, metrics=[output\_rows=46, elapsed\_compute=26.51µs] | | | |ParquetExec: file\_groups={6 groups: [[70434/116281/404d73cea0236530ea94f5470701eb814a8f0565c0e4bef5a2d2e33dfbfc3567/1be334e8-0af8-00da-2615-f67cd4be90f7.parquet], [70434/116281/c14418ba28a22a3abb693a1cb326a63b62dc611aec58c9bed438fdafd3bc5882/8b29ae98-761f-0550-2fe4-ee77503658e9.parquet], [70434/116281/fa677477eed622ae8123da1251aa7c351f801e2ee2f0bc28c0fe3002a30b3563/65bb4dc3-04e1-0e02-107a-90cee83c51b0.parquet], [70434/116281/db162bdd30261019960dd70da182e6ebd270284569ecfb5deffea7e65baa0df9/2505e079-67c5-06d9-3ede-89aca542dd18.parquet], [70434/116281/0c025dcccae8691f5fd70b0f131eea4ca6fafb95a02f90a3dc7bb015efd3ab4f/3f3e44c3-b71e-0ca4-3dc7-8b2f75b9ff86.parquet], …]}, projection=[room, time, temp], output\_ordering=[room@0 ASC, time@1 ASC], predicate=time@6 \>= 1672531200000000000 AND time@6 \<= 1703980800000000000, pruning\_predicate=time\_max@0 \>= 1672531200000000000 AND time\_min@1 \<= 1703980800000000000, required\_guarantees=[], metrics=[output\_rows=46, elapsed\_compute=6ns, predicate\_evaluation\_errors=0, bytes\_scanned=3279, row\_groups\_pruned\_statistics=0, file\_open\_errors=0, file\_scan\_errors=0, pushdown\_rows\_filtered=0, num\_predicate\_creation\_errors=0, row\_groups\_pruned\_bloom\_filter=0, page\_index\_rows\_filtered=0, time\_elapsed\_opening=398.462968ms, time\_elapsed\_processing=1.626106ms, time\_elapsed\_scanning\_total=1.36822ms, page\_index\_eval\_time=33.474µs, pushdown\_eval\_time=14.267µs, time\_elapsed\_scanning\_until\_data=1.27694ms]| ## `EXPLAIN ANALYZE VERBOSE` Executes a statement and returns the execution plan, runtime metrics, and additional details helpful for debugging the statement. The report includes the following: * the [logical plan](/influxdb3/enterprise/reference/internals/query-plan/#logical-plan) * the [physical plan](/influxdb3/enterprise/reference/internals/query-plan/#physical-plan) annotated with execution counters, number of rows produced, and runtime metrics sampled during the query execution * Information truncated in the `EXPLAIN` report–for example, the paths for all [Parquet files retrieved for the query](/influxdb3/enterprise/reference/internals/query-plan/#file_groups). * All intermediate physical plans that DataFusion and the [Querier](/influxdb3/enterprise/reference/internals/storage-engine/#querier) generate before generating the final physical plan–helpful in debugging to see when an [`ExecutionPlan` node](/influxdb3/enterprise/reference/internals/query-plan/#executionplan-nodes) is added or removed, and how InfluxDB optimizes the query. ### Example `EXPLAIN ANALYZE VERBOSE` ```SQL EXPLAIN ANALYZE VERBOSE SELECT temp FROM home WHERE time >= now() - INTERVAL '7 days' AND room = 'Kitchen' ORDER BY time ``` #### Related | | plan_type | plan | | --- | --- | --- | | | plan_type | plan | | 0 | logical_plan | Projection: home.room, AVG(home.temp) AS temp | | | | Aggregate: groupBy=[[home.room]], aggr=[[AVG(home.temp)]] | | | | TableScan: home projection=[room, temp] | | 1 | physical_plan | ProjectionExec: expr=[room@0 as room, AVG(home.temp)@1 as temp] | | | | AggregateExec: mode=FinalPartitioned, gby=[room@0 as room], aggr=[AVG(home.temp)] | | | | CoalesceBatchesExec: target_batch_size=8192 | | | | RepartitionExec: partitioning=Hash([room@0], 8), input_partitions=8 | | | | AggregateExec: mode=Partial, gby=[room@0 as room], aggr=[AVG(home.temp)] | | | | ParquetExec: file_groups={8 groups: [[70434/116281/404d73cea0236530ea94f5470701eb814a8f0565c0e4bef5a2d2e33dfbfc3567/1be334e8-0af8-00da-2615-f67cd4be90f7.parquet, 70434/116281/b7a9e7c57fbfc3bba9427e4b3e35c89e001e2e618b0c7eb9feb4d50a3932f4db/d29370d4-262f-0d32-2459-fe7b099f682f.parquet], [70434/116281/c14418ba28a22a3abb693a1cb326a63b62dc611aec58c9bed438fdafd3bc5882/8b29ae98-761f-0550-2fe4-ee77503658e9.parquet], [70434/116281/fa677477eed622ae8123da1251aa7c351f801e2ee2f0bc28c0fe3002a30b3563/65bb4dc3-04e1-0e02-107a-90cee83c51b0.parquet], [70434/116281/db162bdd30261019960dd70da182e6ebd270284569ecfb5deffea7e65baa0df9/2505e079-67c5-06d9-3ede-89aca542dd18.parquet], [70434/116281/0c025dcccae8691f5fd70b0f131eea4ca6fafb95a02f90a3dc7bb015efd3ab4f/3f3e44c3-b71e-0ca4-3dc7-8b2f75b9ff86.parquet], …]}, projection=[room, temp] | | | plan_type | plan | | --- | --- | --- | | | plan_type | plan | | 0 | Plan with Metrics | ProjectionExec: expr=[room@0 as room, AVG(home.temp)@1 as temp], metrics=[output_rows=2, elapsed_compute=4.768µs] | | | | AggregateExec: mode=FinalPartitioned, gby=[room@0 as room], aggr=[AVG(home.temp)], ordering_mode=Sorted, metrics=[output_rows=2, elapsed_compute=140.405µs] | | | | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=2, elapsed_compute=6.821µs] | | | | RepartitionExec: partitioning=Hash([room@0], 8), input_partitions=8, preserve_order=true, sort_exprs=room@0 ASC, metrics=[output_rows=2, elapsed_compute=18.408µs, repart_time=59.698µs, fetch_time=1.057882762s, send_time=5.83µs] | | | | AggregateExec: mode=Partial, gby=[room@0 as room], aggr=[AVG(home.temp)], ordering_mode=Sorted, metrics=[output_rows=2, elapsed_compute=137.577µs] | | | | RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=6, preserve_order=true, sort_exprs=room@0 ASC, metrics=[output_rows=46, elapsed_compute=26.637µs, repart_time=6ns, fetch_time=399.971411ms, send_time=6.658µs] | | | | ProjectionExec: expr=[room@0 as room, temp@2 as temp], metrics=[output_rows=46, elapsed_compute=3.102µs] | | | | CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=46, elapsed_compute=25.585µs] | | | | FilterExec: time@1 >= 1672531200000000000 AND time@1 <= 1703980800000000000, metrics=[output_rows=46, elapsed_compute=26.51µs] | | | | ParquetExec: file_groups={6 groups: [[70434/116281/404d73cea0236530ea94f5470701eb814a8f0565c0e4bef5a2d2e33dfbfc3567/1be334e8-0af8-00da-2615-f67cd4be90f7.parquet], [70434/116281/c14418ba28a22a3abb693a1cb326a63b62dc611aec58c9bed438fdafd3bc5882/8b29ae98-761f-0550-2fe4-ee77503658e9.parquet], [70434/116281/fa677477eed622ae8123da1251aa7c351f801e2ee2f0bc28c0fe3002a30b3563/65bb4dc3-04e1-0e02-107a-90cee83c51b0.parquet], [70434/116281/db162bdd30261019960dd70da182e6ebd270284569ecfb5deffea7e65baa0df9/2505e079-67c5-06d9-3ede-89aca542dd18.parquet], [70434/116281/0c025dcccae8691f5fd70b0f131eea4ca6fafb95a02f90a3dc7bb015efd3ab4f/3f3e44c3-b71e-0ca4-3dc7-8b2f75b9ff86.parquet], …]}, projection=[room, time, temp], output_ordering=[room@0 ASC, time@1 ASC], predicate=time@6 >= 1672531200000000000 AND time@6 <= 1703980800000000000, pruning_predicate=time_max@0 >= 1672531200000000000 AND time_min@1 <= 1703980800000000000, required_guarantees=[], metrics=[output_rows=46, elapsed_compute=6ns, predicate_evaluation_errors=0, bytes_scanned=3279, row_groups_pruned_statistics=0, file_open_errors=0, file_scan_errors=0, pushdown_rows_filtered=0, num_predicate_creation_errors=0, row_groups_pruned_bloom_filter=0, page_index_rows_filtered=0, time_elapsed_opening=398.462968ms, time_elapsed_processing=1.626106ms, time_elapsed_scanning_total=1.36822ms, page_index_eval_time=33.474µs, pushdown_eval_time=14.267µs, time_elapsed_scanning_until_data=1.27694ms] | --- # SQL functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/ ### [Aggregate functions](/influxdb3/enterprise/reference/sql/functions/aggregate/) Aggregate data with SQL aggregate functions. ### [Selector functions](/influxdb3/enterprise/reference/sql/functions/selector/) Select data with SQL selector functions. ### [Time and date functions](/influxdb3/enterprise/reference/sql/functions/time-and-date/) Use time and date functions to work with time values and time series data. ### [Conditional functions](/influxdb3/enterprise/reference/sql/functions/conditional/) Use conditional functions to conditionally handle null values in SQL queries. ### [Math functions](/influxdb3/enterprise/reference/sql/functions/math/) Use math functions to perform mathematical operations in SQL queries. ### [String functions](/influxdb3/enterprise/reference/sql/functions/string/) Use string functions to operate on string values in SQL queries. ### [Binary string functions](/influxdb3/enterprise/reference/sql/functions/binary-string/) Use binary string functions to encode and decode binary string values in SQL queries. ### [Array functions](/influxdb3/enterprise/reference/sql/functions/array/) Use array functions to create and operate on Arrow arrays or lists in SQL queries. ### [Map functions](/influxdb3/enterprise/reference/sql/functions/map/) Use map functions to create and operate on Arrow maps in SQL queries. ### [Struct functions](/influxdb3/enterprise/reference/sql/functions/struct/) Use struct functions to create Arrow structs in SQL queries. ### [Regular expression functions](/influxdb3/enterprise/reference/sql/functions/regular-expression/) Use regular expression functions to operate on data in SQL queries. ### [Hashing functions](/influxdb3/enterprise/reference/sql/functions/hashing/) Use hashing functions to hash string values in SQL queries using established hashing algorithms. ### [Miscellaneous functions](/influxdb3/enterprise/reference/sql/functions/misc/) Use miscellaneous SQL functions to perform a variety of operations in SQL queries. ### [Cache functions](/influxdb3/enterprise/reference/sql/functions/cache/) Retrieve cached data from InfluxDB 3 Enterprise caches. ### [Window functions](/influxdb3/enterprise/reference/sql/functions/window/) SQL window functions perform an operation across a set of rows related to the current row. --- # SQL aggregate functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/aggregate/ SQL aggregate functions aggregate values in a specified column for each group or SQL partition and return a single row per group containing the aggregate value. ## General aggregate functions ### array\_agg Returns an array created from the expression elements. > [!Note] > `array_agg` returns a `LIST` Arrow type. Use bracket notation to reference the > index of an element in the returned array. Arrays are 1-indexed. ```sql array_agg(expression) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-array_agg-query-example) View `array_agg` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, array_agg(temp)[3] AS '3rd_temp' FROM home GROUP BY room ``` | room |3rd\_temp| |-----------|---------| | Kitchen | 22.7 | |Living Room| 21.8 | ### avg Returns the average of numeric values in the specified column. ```sql avg(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. ##### Aliases * `mean` [](#view-avg-query-example) View `avg` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, avg(precip) AS avg_precip FROM weather GROUP BY location ``` | location | avg\_precip | |-------------|--------------------| | Concord |0.027120658135283374| | Hayward |0.03708029197080292 | |San Francisco|0.03750912408759125 | ### bit\_and Computes the bitwise `AND` of all non-null input values. ```sql bit_and(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-bit_and-query-example) View `bit_and` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, bit_and(precip::BIGINT) AS precip_bit_and FROM weather GROUP BY location ``` | location |precip\_bit\_and| |-------------|----------------| | Concord | 0 | | Hayward | 0 | |San Francisco| 0 | ### bit\_or Computes the bitwise OR of all non-null input values. ```sql bit_or(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-bit_or-query-example) View `bit_or` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, bit_or(precip::BIGINT) AS precip_bit_or FROM weather GROUP BY location ``` | location |precip\_bit\_or| |-------------|---------------| | Concord | 7 | | Hayward | 7 | |San Francisco| 7 | ### bit\_xor Computes the bitwise exclusive OR of all non-null input values. ```sql bit_xor(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-bit_xor-query-example) View `bit_xor` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, bit_xor(precip::BIGINT) AS precip_bit_xor FROM weather GROUP BY location ``` | location |precip\_bit\_xor| |-------------|----------------| | Concord | 4 | | Hayward | 6 | |San Francisco| 4 | ### bool\_and Returns `true` if *all* non-null input values are `true`, otherwise returns `false`. ```sql bool_and(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-bool_and-query-example) View `bool_and` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, bool_and(precip > 0) AS precip_bool_and FROM weather GROUP BY location ``` | location |precip\_bool\_and| |-------------|-----------------| | Concord | false | | Hayward | false | |San Francisco| false | ### bool\_or Returns `true` if *any* non-null input value is `true`, otherwise returns `false`. ```sql bool_or(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-bool_or-query-example) View `bool_or` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, bool_or(precip > 0) AS precip_bool_or FROM weather GROUP BY location ``` | location |precip\_bool\_or| |-------------|----------------| | Concord | true | | Hayward | true | |San Francisco| true | ### count Returns the number of rows in the specified column. Count includes *null* values in the total count. To exclude *null* values from the total count, include ` IS NOT NULL`in the `WHERE` clause. ```sql count(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-count-query-example) View `count` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, count(precip) AS precip_count FROM weather GROUP BY location ``` | location |precip\_count| |-------------|-------------| | Concord | 1094 | | Hayward | 1096 | |San Francisco| 1096 | ### first\_value Returns the first element in an aggregation group according to the specified ordering. If no ordering is specified, returns an arbitrary element from the group. ```sql first_value(expression [ORDER BY expression]) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-first_value-query-example) View `first_value` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, first_value(temp_max ORDER BY time) AS temp_max_first_value FROM weather GROUP BY location ``` | location |temp\_max\_first\_value| |-------------|-----------------------| | Concord | 59 | | Hayward | 57 | |San Francisco| 66 | ### grouping Returns 1 if the data is aggregated across the specified column, or 0 if it is not aggregated in the result set. ```sql grouping(expression) ``` ##### Arguments * **expression**: Expression to evaluate whether data is aggregated across the specified column. Can be a constant, column, or function. [](#view-grouping-query-example) View `grouping` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, avg(temp_max) AS avg_max_temp, grouping(location) AS grouping FROM weather GROUP BY GROUPING SETS ((location), ()) ``` | location | avg\_max\_temp |grouping| |-------------|-----------------|--------| | Concord |75.54379562043796| 0 | | Hayward |69.12043795620438| 0 | |San Francisco|67.59945255474453| 0 | | |70.75456204379562| 1 | ### last\_value Returns the last element in an aggregation group according to the specified ordering. If no ordering is specified, returns an arbitrary element from the group. ```sql last_value(expression [ORDER BY expression]) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-last_value-query-example) View `last_value` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, last_value(temp_max ORDER BY time) AS temp_max_last_value FROM weather GROUP BY location ``` | location |temp\_max\_last\_value| |-------------|----------------------| | Concord | 59 | | Hayward | 58 | |San Francisco| 62 | ### max Returns the maximum value in the specified column. ```sql max(expression) ``` *To return both the maximum value and its associated timestamp, use[`selector_max`](/influxdb3/enterprise/reference/sql/functions/selector/#selector_max).* ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-max-query-example) View `max` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, max(precip) AS max_precip FROM weather GROUP BY location ``` | location |max\_precip| |-------------|-----------| | Concord | 4.53 | | Hayward | 4.34 | |San Francisco| 4.02 | ### mean *Alias of [`avg`](#avg).* ### median Returns the median value in the specified column. ``` median(expression) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-median-query-example) View `median` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, median(temp_avg) AS median_temp_avg FROM weather GROUP BY location ``` | location |median\_temp\_avg| |-------------|-----------------| | Concord | 61.0 | | Hayward | 59.0 | |San Francisco| 58.0 | ### min Returns the minimum value in the specified column. ```sql min(expression) ``` *To return both the minimum value and its associated timestamp, use[`selector_max`](/influxdb3/enterprise/reference/sql/functions/selector/#selector_min).* ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-min-query-example) View `min` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, min(temp_min) AS min_temp_min FROM weather GROUP BY location ``` | location |min\_temp\_min| |-------------|--------------| | Concord | 28.0 | | Hayward | 32.0 | |San Francisco| 35.0 | ### nth\_value Returns the nth value in a group of values. ```sql nth_value(expression, n [ORDER BY order_expression_1, ... order_expression_n]) ``` ##### arguments * **expression**: The column or expression to retrieve the nth value from. * **n**: The position (nth) of the value to retrieve, based on the ordering. * **order\_expression\_1, … order\_expression\_n**: Expressions to order by. Can be a column or function, and any combination of arithmetic operators. [](#view-nth_value-query-example) View `nth_value` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, nth_value(temp, 3 ORDER BY time) AS "3rd_temp" FROM home GROUP BY room ``` | room |3rd\_temp| |-----------|---------| |Living Room| 21.8 | | Kitchen | 22.7 | ### string\_agg Concatenates the values of string expressions and places separator values between them. ```sql string_agg(expression, delimiter) ``` ##### Arguments * **expression**: The string expression to concatenate. Can be a column or any valid string expression. * **delimiter**: A literal string to use as a separator between the concatenated values. [](#view-string_agg-query-example) View `string_agg` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, string_agg(temp_avg::STRING, ', ') AS string_agg FROM weather WHERE time > '2020-01-01T00:00:00Z' AND time < '2020-01-05T00:00:00Z' GROUP BY location ``` | location | string\_agg | |-------------|----------------| |San Francisco|54.0, 52.0, 54.0| | Hayward |51.0, 50.0, 51.0| | Concord |53.0, 49.0, 51.0| ### sum Returns the sum of all values in the specified column. ```sql sum(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-sum-query-example) View `sum` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, sum(precip) AS total_precip FROM weather GROUP BY location ``` | location | total\_precip | |-------------|------------------| | Concord |29.670000000000012| | Hayward | 40.64 | |San Francisco|41.110000000000014| ## Statistical aggregate functions ### corr Returns the coefficient of correlation between two numeric values. ```sql corr(expression1, expression2) ``` ##### Arguments * **expression1**: First column or literal value to operate on. * **expression2**: Second column or literal value to operate on. [](#view-corr-query-example) View `corr` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, corr(hum, temp) AS correlation FROM home GROUP BY room ``` | room | correlation | |-----------|-------------------| |Living Room|0.43665270457835725| | Kitchen |0.6741766954929539 | ### covar Returns the covariance of a set of number pairs. ```sql covar(expression1, expression2) ``` ##### Arguments * **expression1**: First column or literal value to operate on. * **expression2**: Second column or literal value to operate on. [](#view-covar-query-example) View `covar` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, covar(hum, temp) AS covar FROM home GROUP BY room ``` | room | covar | |-----------|-------------------| |Living Room|0.03346153846153959| | Kitchen |0.11134615384615432| ### covar\_pop Returns the population covariance of a set of number pairs. ```sql covar_pop(expression1, expression2) ``` ##### Arguments * **expression1**: First column or literal value to operate on. * **expression2**: Second column or literal value to operate on. [](#view-covar_pop-query-example) View `covar_pop` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, covar_pop(hum, temp) AS covar_pop FROM home GROUP BY room ``` | room | covar\_pop | |-----------|--------------------| | Kitchen |0.10278106508875783 | |Living Room|0.030887573964498087| ### covar\_samp Returns the sample covariance of a set of number pairs. ```sql covar_samp(expression1, expression2) ``` ##### Arguments * **expression1**: First column or literal value to operate on. * **expression2**: Second column or literal value to operate on. [](#view-covar_samp-query-example) View `covar_samp` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, covar_samp(hum, temp) AS covar_samp FROM home GROUP BY room ``` | room | covar\_samp | |-----------|-------------------| | Kitchen |0.11134615384615432| |Living Room|0.03346153846153959| ### regr\_avgx Computes the average of the independent variable (input), `expression_x`, for the non-null dependent variable, `expression_y`. ```sql regr_avgx(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_avgx-query-example) View `regr_avgx` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_avgx(temp_min, temp_max) AS temp_regr_avgx FROM weather GROUP BY location ``` | location |temp\_regr\_avgx | |-------------|-----------------| | Concord |75.54379562043796| | Hayward |69.14808043875686| |San Francisco|67.59945255474454| ### regr\_avgy Computes the average of the dependent variable (output), `expression_y`, for the non-null dependent variable, `expression_y`. ```sql regr_avgy(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_avgy-query-example) View `regr_avgy` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_avgy(temp_min, temp_max) AS temp_regr_avgy FROM weather GROUP BY location ``` | location | temp\_regr\_avgy | |-------------|------------------| | Concord |50.153284671532845| | Hayward |50.913162705667276| |San Francisco|51.52372262773722 | ### regr\_count Counts the number of non-null paired data points. ```sql regr_count(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_count-query-example) View `regr_count` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_count(temp_min, temp_max) AS temp_regr_count FROM weather GROUP BY location ``` | location |temp\_regr\_count| |-------------|-----------------| | Concord | 1096 | | Hayward | 1094 | |San Francisco| 1096 | ### regr\_intercept Computes the y-intercept of the linear regression line. For the equation `(y = kx + b)`, this function returns `b`. ```sql regr_intercept(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_intercept-query-example) View `regr_intercept` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_intercept(temp_min, temp_max) AS temp_regr_intercept FROM weather GROUP BY location ``` | location |temp\_regr\_intercept| |-------------|---------------------| | Concord | 11.636281392206769 | | Hayward | 12.876956842745152 | |San Francisco| 19.125237647086607 | ### regr\_r2 Computes the square of the correlation coefficient between the independent and dependent variables. ```sql regr_r2(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_r2-query-example) View `regr_r2` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_r2(temp_min, temp_max) AS temp_regr_r2 FROM weather GROUP BY location ``` | location | temp\_regr\_r2 | |-------------|------------------| | Concord |0.6474628308450441| | Hayward |0.5166296626320914| |San Francisco|0.5032317511200297| ### regr\_slope Returns the slope of the linear regression line for non-null pairs in aggregate columns. Given input column `Y` and `X`: `regr_slope(Y, X)` returns the slope (`k` in `Y = k*X + b`) using minimal RSS fitting. ```sql regr_slope(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Y expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: X expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_slope-query-example) View `regr_slope` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_slope(temp_min, temp_max) AS temp_regr_slope FROM weather GROUP BY location ``` | location |temp\_regr\_slope | |-------------|------------------| | Concord |0.5098632252058237| | Hayward |0.5500688612261629| |San Francisco|0.4792714105844738| ### regr\_sxx Computes the sum of squares of the independent variable. ```sql regr_sxx(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_sxx-query-example) View `regr_sxx` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_sxx(temp_min, temp_max) AS temp_regr_sxx FROM weather GROUP BY location ``` | location | temp\_regr\_sxx | |-------------|------------------| | Concord |210751.89781021897| | Hayward |99644.01096892142 | |San Francisco|77413.15967153282 | ### regr\_syy Computes the sum of squares of the dependent variable. ```sql regr_syy(expression_y, expression_x) ``` ##### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_syy-query-example) View `regr_syy` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_syy(temp_min, temp_max) AS temp_regr_syy FROM weather GROUP BY location ``` | location | temp\_regr\_syy | |-------------|------------------| | Concord |84618.24817518248 | | Hayward |58358.750457038404| |San Francisco|35335.38321167884 | ### regr\_sxy Computes the sum of products of paired data points. ```sql regr_sxy(expression_y, expression_x) ``` #### Arguments * **expression\_y**: Dependent variable. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Independent variable. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-regr_sxy-query-example) View `regr_sxy` query example *The following example uses the[NOAA Bay Area weather data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ``` SELECT location, regr_sxy(temp_min, temp_max) AS temp_regr_sxy FROM weather GROUP BY location ``` | location | temp\_regr\_sxy | |-------------|------------------| | Concord |107454.64233576645| | Hayward |54811.06764168191 | |San Francisco|37101.914233576645| ### stddev Returns the standard deviation of a set of numbers. ```sql stddev(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-stddev-query-example) View `stddev` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, stddev(co) AS stddev FROM home GROUP BY room ``` | room | stddev | |-----------|-----------------| |Living Room|5.885662718931967| | Kitchen |9.321879418735037| ### stddev\_pop Returns the population standard deviation of a set of numbers. ```sql stddev_pop(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-stddev_pop-query-example) View `stddev_pop` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, stddev_pop(co) AS stddev_pop FROM home GROUP BY room ``` | room | stddev\_pop | |-----------|-----------------| | Kitchen |8.956172047894082| |Living Room|5.654761830612032| ### stddev\_samp Returns the sample standard deviation of a set of numbers. ```sql stddev_samp(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-stddev_samp-query-example) View `stddev_samp` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, stddev_samp(co) AS stddev_samp FROM home GROUP BY room ``` | room | stddev\_samp | |-----------|-----------------| |Living Room|5.885662718931967| | Kitchen |9.321879418735037| ### var Returns the statistical variance of a set of numbers. ```sql var(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-var-query-example) View `var` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, var(co) AS var FROM home GROUP BY room ``` | room | var | |-----------|-----------------| |Living Room|34.64102564102564| | Kitchen |86.89743589743587| ### var\_pop Returns the statistical population variance of a set of numbers. ```sql var_pop(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. ##### Aliases * var\_population [](#view-var_pop-query-example) View `var_pop` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, var_pop(co) AS var_pop FROM home GROUP BY room ``` | room | var\_pop | |-----------|------------------| |Living Room|31.976331360946745| | Kitchen |80.21301775147927 | ### var\_population *Alias of [`var_pop`](#var_pop).* ### var\_samp Returns the statistical sample variance of a set of numbers. ```sql var_samp(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. ##### Aliases * var\_sample [](#view-var_samp-query-example) View `var_samp` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, var_samp(co) AS var_samp FROM home GROUP BY room ``` | room | var\_samp | |-----------|-----------------| | Kitchen |86.89743589743587| |Living Room|34.64102564102564| ### var\_sample *Alias of [var\_samp](#var_samp).* ## Approximate aggregate functions ### approx\_distinct Returns the approximate number of distinct input values calculated using the HyperLogLog algorithm. ```sql approx_distinct(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-approx_distinct-query-example) View `approx_distinct` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, approx_distinct(co::string) AS approx_distinct FROM home GROUP BY room ``` | room |approx\_distinct| |-----------|----------------| |Living Room| 7 | | Kitchen | 8 | ### approx\_median Returns the approximate median (50th percentile) of input values. It is an alias of `approx_percentile_cont(0.5) WITHIN GROUP (ORDER BY expression)`. ```sql approx_median(expression) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-approx_median-query-example) View `approx_median` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, approx_median(temp) AS approx_median FROM home GROUP BY room ``` | room |approx\_median| |-----------|--------------| | Kitchen | 22.7 | |Living Room| 22.3 | ### approx\_percentile\_cont Returns the approximate percentile of input values using the t-digest algorithm. ```sql approx_percentile_cont(percentile [, centroids]) WITHIN GROUP (ORDER BY expression) -- OR approx_percentile_cont(expression, percentile, centroids) ``` ##### Arguments * **percentile**: Percentile to compute. Must be a float value between 0 and 1 (inclusive). * **centroids**: Number of centroids to use in the t-digest algorithm. *Default is 100*. A higher number results in more accurate approximation but requires more memory. * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. If there are this number or fewer unique values, you can expect an exact result. A higher number of centroids results in a more accurate approximation, but requires more memory to compute. [](#view-approx_percentile_cont-query-example) View `approx_percentile_cont` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, approx_percentile_cont(0.99) WITHIN GROUP (ORDER BY temp) AS "99th_percentile" FROM home GROUP BY room ``` | room |99th\_percentile| |-----------|----------------| | Kitchen | 23.3 | |Living Room| 22.8 | ### approx\_percentile\_cont\_with\_weight Returns the weighted approximate percentile of input values using the t-digest algorithm. ```sql approx_percentile_cont_with_weight(weight, percentile [, centroids]) WITHIN GROUP (ORDER BY expression) -- OR approx_percentile_cont_with_weight(expression, weight, percentile) ``` ##### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **weight**: Expression to use as weight. Can be a constant, column, or function, and any combination of arithmetic operators. * **percentile**: Percentile to compute. Must be a float value between 0 and 1 (inclusive). * **centroids**: Number of centroids to use in the t-digest algorithm. *Default is 100.*A higher number results in more accurate approximation but requires more memory. [](#view-approx_percentile_cont_with_weight-query-example) View `approx_percentile_cont_with_weight` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, approx_percentile_cont_with_weight(co, 0.99) WITHIN GROUP (ORDER BY temp) AS "co_weighted_99th_percentile" FROM home GROUP BY room ``` | room |co\_weighted\_99th\_percentile| |-----------|------------------------------| | Kitchen | 23.3 | |Living Room| 22.8 | #### Related * [Aggregate data with SQL](/influxdb3/enterprise/query-data/sql/aggregate-select/) | room | 3rd_temp | | --- | --- | | room | 3rd_temp | | Kitchen | 22.7 | | Living Room | 21.8 | | location | avg_precip | | --- | --- | | location | avg_precip | | Concord | 0.027120658135283374 | | Hayward | 0.03708029197080292 | | San Francisco | 0.03750912408759125 | | location | precip_bit_and | | --- | --- | | location | precip_bit_and | | Concord | 0 | | Hayward | 0 | | San Francisco | 0 | | location | precip_bit_or | | --- | --- | | location | precip_bit_or | | Concord | 7 | | Hayward | 7 | | San Francisco | 7 | | location | precip_bit_xor | | --- | --- | | location | precip_bit_xor | | Concord | 4 | | Hayward | 6 | | San Francisco | 4 | | location | precip_bool_and | | --- | --- | | location | precip_bool_and | | Concord | false | | Hayward | false | | San Francisco | false | | location | precip_bool_or | | --- | --- | | location | precip_bool_or | | Concord | true | | Hayward | true | | San Francisco | true | | location | precip_count | | --- | --- | | location | precip_count | | Concord | 1094 | | Hayward | 1096 | | San Francisco | 1096 | | location | temp_max_first_value | | --- | --- | | location | temp_max_first_value | | Concord | 59 | | Hayward | 57 | | San Francisco | 66 | | location | avg_max_temp | grouping | | --- | --- | --- | | location | avg_max_temp | grouping | | Concord | 75.54379562043796 | 0 | | Hayward | 69.12043795620438 | 0 | | San Francisco | 67.59945255474453 | 0 | | | 70.75456204379562 | 1 | | location | temp_max_last_value | | --- | --- | | location | temp_max_last_value | | Concord | 59 | | Hayward | 58 | | San Francisco | 62 | | location | max_precip | | --- | --- | | location | max_precip | | Concord | 4.53 | | Hayward | 4.34 | | San Francisco | 4.02 | | location | median_temp_avg | | --- | --- | | location | median_temp_avg | | Concord | 61.0 | | Hayward | 59.0 | | San Francisco | 58.0 | | location | min_temp_min | | --- | --- | | location | min_temp_min | | Concord | 28.0 | | Hayward | 32.0 | | San Francisco | 35.0 | | room | 3rd_temp | | --- | --- | | room | 3rd_temp | | Living Room | 21.8 | | Kitchen | 22.7 | | location | string_agg | | --- | --- | | location | string_agg | | San Francisco | 54.0, 52.0, 54.0 | | Hayward | 51.0, 50.0, 51.0 | | Concord | 53.0, 49.0, 51.0 | | location | total_precip | | --- | --- | | location | total_precip | | Concord | 29.670000000000012 | | Hayward | 40.64 | | San Francisco | 41.110000000000014 | | room | correlation | | --- | --- | | room | correlation | | Living Room | 0.43665270457835725 | | Kitchen | 0.6741766954929539 | | room | covar | | --- | --- | | room | covar | | Living Room | 0.03346153846153959 | | Kitchen | 0.11134615384615432 | | room | covar_pop | | --- | --- | | room | covar_pop | | Kitchen | 0.10278106508875783 | | Living Room | 0.030887573964498087 | | room | covar_samp | | --- | --- | | room | covar_samp | | Kitchen | 0.11134615384615432 | | Living Room | 0.03346153846153959 | | location | temp_regr_avgx | | --- | --- | | location | temp_regr_avgx | | Concord | 75.54379562043796 | | Hayward | 69.14808043875686 | | San Francisco | 67.59945255474454 | | location | temp_regr_avgy | | --- | --- | | location | temp_regr_avgy | | Concord | 50.153284671532845 | | Hayward | 50.913162705667276 | | San Francisco | 51.52372262773722 | | location | temp_regr_count | | --- | --- | | location | temp_regr_count | | Concord | 1096 | | Hayward | 1094 | | San Francisco | 1096 | | location | temp_regr_intercept | | --- | --- | | location | temp_regr_intercept | | Concord | 11.636281392206769 | | Hayward | 12.876956842745152 | | San Francisco | 19.125237647086607 | | location | temp_regr_r2 | | --- | --- | | location | temp_regr_r2 | | Concord | 0.6474628308450441 | | Hayward | 0.5166296626320914 | | San Francisco | 0.5032317511200297 | | location | temp_regr_slope | | --- | --- | | location | temp_regr_slope | | Concord | 0.5098632252058237 | | Hayward | 0.5500688612261629 | | San Francisco | 0.4792714105844738 | | location | temp_regr_sxx | | --- | --- | | location | temp_regr_sxx | | Concord | 210751.89781021897 | | Hayward | 99644.01096892142 | | San Francisco | 77413.15967153282 | | location | temp_regr_syy | | --- | --- | | location | temp_regr_syy | | Concord | 84618.24817518248 | | Hayward | 58358.750457038404 | | San Francisco | 35335.38321167884 | | location | temp_regr_sxy | | --- | --- | | location | temp_regr_sxy | | Concord | 107454.64233576645 | | Hayward | 54811.06764168191 | | San Francisco | 37101.914233576645 | | room | stddev | | --- | --- | | room | stddev | | Living Room | 5.885662718931967 | | Kitchen | 9.321879418735037 | | room | stddev_pop | | --- | --- | | room | stddev_pop | | Kitchen | 8.956172047894082 | | Living Room | 5.654761830612032 | | room | stddev_samp | | --- | --- | | room | stddev_samp | | Living Room | 5.885662718931967 | | Kitchen | 9.321879418735037 | | room | var | | --- | --- | | room | var | | Living Room | 34.64102564102564 | | Kitchen | 86.89743589743587 | | room | var_pop | | --- | --- | | room | var_pop | | Living Room | 31.976331360946745 | | Kitchen | 80.21301775147927 | | room | var_samp | | --- | --- | | room | var_samp | | Kitchen | 86.89743589743587 | | Living Room | 34.64102564102564 | | room | approx_distinct | | --- | --- | | room | approx_distinct | | Living Room | 7 | | Kitchen | 8 | | room | approx_median | | --- | --- | | room | approx_median | | Kitchen | 22.7 | | Living Room | 22.3 | | room | 99th_percentile | | --- | --- | | room | 99th_percentile | | Kitchen | 23.3 | | Living Room | 22.8 | | room | co_weighted_99th_percentile | | --- | --- | | room | co_weighted_99th_percentile | | Kitchen | 23.3 | | Living Room | 22.8 | --- # SQL array functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/array/ Use array functions to create and operate on Arrow arrays or lists in SQL queries. ## array\_any\_value Returns the first non-null element in the array. ```sql array_any_value(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_any_value` [](#view-array_any_value-example) View `array_any_value` example ```sql SELECT array_any_value([NULL, 1, 2, 3]) AS array_any_value ``` |array\_any\_value| |-----------------| | 1 | ## array\_append Appends an element to the end of an array. ```sql array_append(array, element) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to append to the array. #### Aliases * `list_append` * `array_push_back` * `list_push_back` [](#view-array_append-example) View `array_append` example ```sql SELECT array_append([1, 2, 3], 4) AS array_append ``` |array\_append| |-------------| |[1, 2, 3, 4] | ## array\_cat *Alias of [array\_concat](#array_concat).* ## array\_concat Concatenates multiple arrays into a single array. ```sql array_concat(array[, ..., array_n]) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **array\_n**: Subsequent array column or literal array to concatenate. #### Aliases * `array_cat` * `list_concat` * `list_cat` [](#view-array_concat-example) View `array_concat` example ```sql SELECT array_concat([1, 2], [3, 4], [5, 6]) AS array_concat ``` | array\_concat | |------------------| |[1, 2, 3, 4, 5, 6]| ## array\_contains *Alias of [array\_has](#array_has).* ## array\_dims Returns an array of the array’s dimensions. ```sql array_dims(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_dims` [](#view-array_dims-example) View `array_dims` example ```sql SELECT array_dims([[1, 2, 3], [4, 5, 6]]) AS array_dims ``` |array\_dims(List([1,2,3,4,5,6]))| |--------------------------------| | [2, 3] | ## array\_distance Returns the Euclidean distance between two input arrays of equal length. ```sql array_distance(array1, array2) ``` ### Arguments * **array1**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **array2**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_distance` [](#view-array_distance-example) View `array_distance` example ```sql SELECT array_distance([1, 2], [1, 4]) AS array_distance ``` |array\_distance| |---------------| | 2.0 | ## array\_distinct Returns distinct values from the array after removing duplicates. ```sql array_distinct(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_distinct` [](#view-array_distinct-example) View `array_distinct` example ```sql SELECT array_distinct([1, 3, 2, 3, 1, 2, 4]) AS array_distinct ``` |array\_distinct(List([1,2,3,4]))| |--------------------------------| | [1, 2, 3, 4] | ## array\_element Extracts the element with the index n from the array. ```sql array_element(array, index) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **index**: Index to use to extract the element from the array. #### Aliases * `array_extract` * `list_element` * `list_extract` [](#view-array_element-example) View `array_element` example ```sql SELECT array_element([1, 2, 3, 4], 3) AS array_element ``` |array\_element| |--------------| | 3 | ## array\_empty *Alias of [empty](#empty).* ## array\_except Returns an array containing elements from the first array that are not present in the second array. ```sql array_except(array1, array2) ``` ### Arguments * **array1**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **array2**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_except` [](#view-array_except-example) View `array_except` example ```sql SELECT array_except([1, 2, 3, 4], [5, 6, 3, 4]) AS array_except ``` |array\_except| |-------------| | [1, 2] | ## array\_extract *Alias of [array\_element](#array_element).* ## array\_has Returns `true` if the array contains the element. ```sql array_has(array, element) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Scalar or Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_has` * `array_contains` * `list_contains` [](#view-array_has-example) View `array_has` example ```sql SELECT array_has([1, 2, 3], 2) AS array_has ``` |array\_has| |----------| | true | ## array\_has\_all Returns `true` if all elements of sub-array exist in array. ```sql array_has_all(array, sub-array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **sub-array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_has_all` [](#view-array_has_all-example) View `array_has_all` example ```sql SELECT array_has_all([1, 2, 3, 4], [2, 3]) AS array_has_all ``` |array\_has\_all| |---------------| | true | ## array\_has\_any Returns `true` if at least one element appears in both arrays. ```sql array_has_any(array, sub-array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **sub-array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_has_any` * `arrays_overlap` [](#view-array_has_any-example) View `array_has_any` example ```sql SELECT array_has_any([1, 2, 3], [3, 4]) AS array_has_any ``` |array\_has\_any| |---------------| | true | ## array\_indexof *Alias of [array\_position](#array_position).* ## array\_intersect Returns an array containing only the elements that appear in both **array1** and **array2**. ```sql array_intersect(array1, array2) ``` ### Arguments * **array1**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **array2**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_intersect` [](#view-array_intersect-example-with-intersecting-arrays) View `array_intersect` example with intersecting arrays ```sql SELECT array_intersect([1, 2, 3, 4], [5, 6, 3, 4]) AS array_intersect ``` |array\_intersect| |----------------| | [3, 4] | [](#view-array_intersect-example-with-non-intersecting-arrays) View `array_intersect` example with non-intersecting arrays ```sql SELECT array_intersect([1, 2, 3, 4], [5, 6, 7, 8]) AS array_intersect ``` |array\_intersect| |----------------| | [] | ## array\_join *Alias of [array\_to\_string](#array_to_string).* ## array\_length Returns the length of the array dimension. ```sql array_length(array, dimension) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **dimension**: Array dimension. Default is `1`. #### Aliases * `list_length` [](#view-array_length-example-with-single-dimension-array) View `array_length` example with single-dimension array ```sql SELECT array_length([1, 2, 3, 4, 5]) AS array_length ``` |array\_length| |-------------| | 5 | [](#view-array_length-example-with-multi-dimension-array) View `array_length` example with multi-dimension array ```sql WITH vars AS ( SELECT [ [1, 2, 3, 4, 5], [5, 6, 7, 8, 9] ] AS example_array ) SELECT array_length(example_array, 1) AS 'dim1_length', array_length(example_array, 2) AS 'dim2_length' FROM vars ``` |dim1\_length|dim2\_length| |------------|------------| | 2 | 5 | ## array\_max Returns the maximum value in the array. ```sql array_max(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_max` [](#view-array_max-example) View `array_max` example ```sql SELECT array_max([3,1,4,2]) AS array_max ``` |array\_max| |----------| | 4 | ## array\_min Returns the minimum value in the array. ```sql array_min(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. [](#view-array_min-example) View `array_min` example ```sql SELECT array_min([3,1,4,2]) AS array_min ``` |array\_min| |----------| | 1 | ## array\_ndims Returns the number of dimensions of the array. ```sql array_ndims(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_ndims` [](#view-array_ndims-example) View `array_ndims` example ```sql SELECT array_ndims([[1, 2, 3], [4, 5, 6]]) AS array_ndims ``` |array\_ndims| |------------| | 2 | ## array\_pop\_back Returns the array without the last element. ```sql array_pop_back(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_pop_back` [](#view-array_pop_back-example) View `array_pop_back` example ```sql SELECT array_pop_back([1, 2, 3]) AS array_pop_back ``` |array\_pop\_back| |----------------| | [1, 2] | ## array\_pop\_front Returns the array without the first element. ```sql array_pop_front(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_pop_front` [](#view-array_pop_front-example) View `array_pop_front` example ```sql SELECT array_pop_front([1, 2, 3]) AS array_pop_front ``` |array\_pop\_front| |-----------------| | [2, 3] | ## array\_position Returns the position of the first occurrence of the specified element in the array, or *NULL* if not found. ```sql array_position(array, element, index) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to search for position in the array. * **index**: Index at which to start searching (1-indexed). Default is `1`. #### Aliases * `list_position` * `array_indexof` * `list_indexof` [](#view-array_position-example) View `array_position` example ```sql SELECT array_position([1, 2, 2, 3, 1, 4], 2) AS array_position ``` |array\_position| |---------------| | 2 | [](#view-array_position-example-with-index-offset) View `array_position` example with index offset ```sql SELECT array_position([1, 2, 2, 3, 1, 4], 2, 3) AS array_position ``` |array\_position| |---------------| | 3 | ## array\_positions Searches for an element in the array and returns the position or index of each occurrence. ```sql array_positions(array, element) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to search for position in the array. #### Aliases * `list_positions` [](#view-array_positions-example) View `array_positions` example ```sql SELECT array_positions(['John', 'Jane', 'James', 'John'], 'John') AS array_positions ``` |array\_positions| |----------------| | [1, 4] | ## array\_prepend Prepends an element to the beginning of an array. ```sql array_prepend(element, array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to prepend to the array. #### Aliases * `list_prepend` * `array_push_front` * `list_push_front` [](#view-array_prepend-example) View `array_prepend` example ```sql SELECT array_prepend(1, [2, 3, 4]) AS array_prepend ``` |array\_prepend| |--------------| | [1, 2, 3, 4] | ## array\_push\_back *Alias of [array\_append](#array_append).* ## array\_push\_front *Alias of [array\_prepend](#array_prepend).* ## array\_remove Removes the first element from the array equal to the given value. ```sql array_remove(array, element) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to remove from the array. #### Aliases * `list_remove` [](#view-array_remove-example) View `array_remove` example ```sql SELECT array_remove([1, 2, 2, 3, 2, 1, 4], 2) AS array_remove ``` | array\_remove | |------------------| |[1, 2, 3, 2, 1, 4]| ## array\_remove\_all Removes all elements from the array equal to the specified value. ```sql array_remove_all(array, element) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to be removed from the array. #### Aliases * `list_remove_all` [](#view-array_remove_all-example) View `array_remove_all` example ```sql SELECT array_remove_all([1, 2, 2, 3, 2, 1, 4], 2) AS array_remove_all ``` |array\_remove\_all| |------------------| | [1, 3, 1, 4] | ## array\_remove\_n Removes the first `max` elements from the array equal to the specified value. ```sql array_remove_n(array, element, max) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **element**: Element to remove from the array. * **max**: Maximum number of occurrences to remove. #### Aliases * `list_remove_n` [](#view-array_remove_n-example) View `array_remove_n` example ```sql SELECT array_remove_n([1, 2, 2, 3, 2, 1, 4], 2, 2) AS array_remove_n ``` |array\_remove\_n| |----------------| |[1, 3, 2, 1, 4] | ## array\_repeat Returns an array containing element `count` times. ```sql array_repeat(element, count) ``` ### Arguments * **element**: Element expression. Can be a constant, column, or function, and any combination of array operators. * **count**: Number of times to repeat the element. #### Aliases * `list_repeat` [](#view-array_repeat-example-with-numeric-values) View `array_repeat` example with numeric values ```sql SELECT array_repeat(1, 3) AS array_repeat ``` |array\_repeat| |-------------| | [1, 1, 1] | [](#view-array_repeat-example-with-string-values) View `array_repeat` example with string values ```sql SELECT array_repeat('John', 3) AS array_repeat ``` | array\_repeat | |------------------| |[John, John, John]| [](#view-array_repeat-example-with-array-values) View `array_repeat` example with array values ```sql SELECT array_repeat([1, 2], 2) AS array_repeat ``` | array\_repeat | |----------------| |[[1, 2], [1, 2]]| ## array\_replace Replaces the first occurrence of the specified element with another specified element. ```sql array_replace(array, from, to) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **from**: Element to replace. * **to**: Replacement element. #### Aliases * `list_replace` [](#view-array_replace-example) View `array_replace` example ```sql SELECT array_replace(['John', 'Jane', 'James', 'John'], 'John', 'Joe') AS array_replace ``` | array\_replace | |------------------------| |[Joe, Jane, James, John]| ## array\_replace\_all Replaces all occurrences of the specified element with another specified element. ```sql array_replace_all(array, from, to) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **from**: Element to replace. * **to**: Replacement element. #### Aliases * `list_replace_all` [](#view-array_replace_all-example) View `array_replace_all` example ```sql SELECT array_replace_all(['John', 'Jane', 'James', 'John'], 'John', 'Joe') AS array_replace_all ``` | array\_replace\_all | |-----------------------| |[Joe, Jane, James, Joe]| ## array\_replace\_n Replaces the first `max` occurrences of the specified element with another specified element. ```sql array_replace_n(array, from, to, max) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **from**: Element to replace. * **to**: Replacement element. * **max**: Maximum number of occurrences to replace. #### Aliases * `list_replace_n` [](#view-array_replace_n-example) View `array_replace_n` example ```sql SELECT array_replace_n(['John', 'Jane', 'James', 'John', 'John'], 'John', 'Joe', 2) AS array_replace_n ``` | array\_replace\_n | |-----------------------------| |[Joe, Jane, James, Joe, John]| ## array\_resize Resizes the list to contain size elements. Initializes new elements with value Resizes the array to the specified size. If expanding, fills new elements with the specified value (or *NULL* if not provided). If shrinking, truncates excess elements. ```sql array_resize(array, size, value) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **size**: New size of the array. * **value**: Value to use for new elements. Default is *NULL*. #### Aliases * `list_resize` [](#view-array_resize-example) View `array_resize` example ```sql SELECT array_resize([1, 2, 3], 5, 0) AS array_resize ``` |array\_resize(List([1,2,3],5,0))| |--------------------------------| | [1, 2, 3, 0, 0] | ## array\_reverse Returns the array with the order of the elements reversed. ```sql array_reverse(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_reverse` [](#view-array_reverse-example) View `array_reverse` example ```sql SELECT array_reverse([1, 2, 3, 4]) AS array_reverse ``` |array\_reverse| |--------------| | [4, 3, 2, 1] | ## array\_slice Returns a slice of the array based on 1-indexed start and end positions. ```sql array_slice(array, begin, end) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **begin**: Index of the first element. If negative, it counts backward from the end of the array. * **end**: Index of the last element. If negative, it counts backward from the end of the array. * **stride**: Stride of the array slice. The default is `1`. #### Aliases * `list_slice` [](#view-array_slice-example) View `array_slice` example ```sql SELECT array_slice([1, 2, 3, 4, 5, 6, 7, 8], 3, 6) AS array_slice ``` |array\_slice| |------------| |[3, 4, 5, 6]| ## array\_sort Sorts elements in an array. If elements are numeric, it sorts elements in numerical order. If elements are strings, it sorts elements in lexicographical order. ```sql array_sort(array, sort_order, sort_nulls) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **sort\_order**: Sort order (`'ASC'` *(default)* or `'DESC'`). * **sort\_nulls**: Sort nulls first or last (`'NULLS FIRST'` *(default)* or `'NULLS LAST'`). #### Aliases * `list_sort` [](#view-array_sort-example-with-numeric-elements) View `array_sort` example with numeric elements ```sql SELECT array_sort([3, 1, 2]) AS array_sort ``` |array\_sort| |-----------| | [1, 2, 3] | [](#view-array_sort-example-with-string-elements) View `array_sort` example with string elements ```sql SELECT array_sort(['banana', 'apple', 'cherry'], 'DESC') AS array_sort ``` | array\_sort | |-----------------------| |[cherry, banana, apple]| [](#view-array_sort-example-with-_null_-elements) View `array_sort` example with *NULL* elements ```sql SELECT array_sort( ['banana', 'apple', NULL, 'cherry', NULL], 'ASC', 'NULLS LAST' ) AS array_sort ``` | array\_sort | |---------------------------| |[apple, banana, cherry, , ]| ## array\_to\_string Converts an array to a string by joining all elements with the specified delimiter. ```sql array_to_string(array, delimiter[, null_string]) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **delimiter**: Array element separator. * **null\_string**: Optional. String to replace *NULL* values in the array. If not provided, *NULL* elements are ignored. #### Aliases * `list_to_string` * `array_join` * `list_join` [](#view-array_to_string-example) View `array_to_string` example ```sql SELECT array_to_string([1,2,3,4,5,6,7,8], ',') AS array_to_string ``` |array\_to\_string| |-----------------| | 1,2,3,4,5,6,7,8 | [](#view-array_to_string-example-with-_null_-replacements) View `array_to_string` example with *NULL* replacements ```sql SELECT array_to_string([[1,2,3,4,5,NULL,7,8,NULL]], '-', '?') AS array_to_string ``` |array\_to\_string| |-----------------| |1-2-3-4-5-?-7-8-?| ## array\_union Returns an array of elements that are present in both arrays (all elements from Returns an array containing all unique elements from both input arrays, with duplicates removed. ```sql array_union(array1, array2) ``` ### Arguments * **array1**: Array expression. Can be a constant, column, or function, and any combination of array operators. * **array2**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `list_union` [](#view-array_union-example) View `array_union` example ```sql SELECT array_union([1, 2, 3, 4], [5, 6, 3, 4]) AS array_union ``` | array\_union | |------------------| |[1, 2, 3, 4, 5, 6]| ## arrays\_overlap *Alias of [array\_has\_any](#array_has_any).* ## cardinality Returns the total number of elements in the array. ```sql cardinality(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. [](#view-cardinality-example) View `cardinality` example ```sql SELECT cardinality([[1, 2, 3, 4], [5, 6, 7, 8]]) AS cardinality ``` |cardinality| |-----------| | 8 | ## empty Returns `true` for an empty array or `false` for a non-empty array. ```sql empty(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. #### Aliases * `array_empty` * `list_empty` [](#view-empty-example) View `empty` example ```sql SELECT empty(['apple']) AS empty ``` |empty| |-----| |false| ## flatten Flattens nested arrays into a single-level array. * Recursively flattens arrays at any depth of nesting * Returns unchanged if the array is already flat The result contains all elements from all nested arrays in a single flat array. ```sql flatten(array) ``` ### Arguments * **array**: Array expression. Can be a constant, column, or function, and any combination of array operators. [](#view-flatten-example) View `flatten` example ```sql SELECT flatten([[1, 2], [3, 4]]) AS flattened ``` | flattened | |------------| |[1, 2, 3, 4]| ## generate\_series Returns an array with values between the specified **start** and **stop** values generated at the specified **step**. The range `start..stop` contains all values greater than or equal to **start**and less than or equal to **stop** (`start <= x <= stop`). If **start** is greater than or equal to **stop** (`start >= stop`), the function returns an empty array. *`generate_series` is similar to [range](#range), but includes the upper bound (**stop**) in the output array.* ```sql generate_series(start, stop, step) ``` ### Arguments * **start**: Start of the series. Supports integers, timestamps, dates, or string types that can be coerced to `Date32`. * **stop**: Upper bound of the series. Supports integers, timestamps, dates, or string types that can be coerced to `Date32`. The type must be the same as **start**. * **step**: Increase by step (cannot be `0`). Steps less than a day are only supported for ranges with the `TIMESTAMP` type. ##### Related functions [range](#range) [](#view-generate_series-example) View `generate_series` example ```sql SELECT generate_series(1,5) AS generate_series ``` |generate\_series| |----------------| |[1, 2, 3, 4, 5] | [](#view-range-example-with-dates) View `range` example with dates ```sql SELECT generate_series( DATE '2025-03-01', DATE '2025-08-01', INTERVAL '1 month' ) AS generate_series ``` | generate\_series | |------------------------------------------------------------------------| |[2025-03-01, 2025-04-01, 2025-05-01, 2025-06-01, 2025-07-01, 2025-08-01]| [](#view-generate_series-example-using-timestamps) View `generate_series` example using timestamps ```sql SELECT generate_series( '2025-01-01T00:00:00Z'::timestamp, '2025-01-01T06:00:00Z'::timestamp, INTERVAL '2 hours' ) AS generate_series ``` | generate\_series | |------------------------------------------------------------------------------------| |[2025-01-01T00:00:00, 2025-01-01T02:00:00, 2025-01-01T04:00:00, 2025-01-01T06:00:00]| ## list\_any\_value *Alias of [array\_any\_value](#array_any_value).* ## list\_append *Alias of [array\_append](#array_append).* ## list\_cat *Alias of [array\_concat](#array_concat).* ## list\_concat *Alias of [array\_concat](#array_concat).* ## list\_contains *Alias of [array\_has](#array_has).* ## list\_dims *Alias of [array\_dims](#array_dims).* ## list\_distance *Alias of [array\_distance](#array_distance).* ## list\_distinct *Alias of [array\_distinct](#array_distinct).* ## list\_element *Alias of [array\_element](#array_element).* ## list\_empty *Alias of [empty](#empty).* ## list\_except *Alias of [array\_except](#array_except).* ## list\_extract *Alias of [array\_element](#array_element).* ## list\_has *Alias of [array\_has](#array_has).* ## list\_has\_all *Alias of [array\_has\_all](#array_has_all).* ## list\_has\_any *Alias of [array\_has\_any](#array_has_any).* ## list\_indexof *Alias of [array\_position](#array_position).* ## list\_intersect *Alias of [array\_intersect](#array_intersect).* ## list\_join *Alias of [array\_to\_string](#array_to_string).* ## list\_length *Alias of [array\_length](#array_length).* ## list\_max *Alias of [array\_max](#array_max).* ## list\_ndims *Alias of [array\_ndims](#array_ndims).* ## list\_pop\_back *Alias of [array\_pop\_back](#array_pop_back).* ## list\_pop\_front *Alias of [array\_pop\_front](#array_pop_front).* ## list\_position *Alias of [array\_position](#array_position).* ## list\_positions *Alias of [array\_positions](#array_positions).* ## list\_prepend *Alias of [array\_prepend](#array_prepend).* ## list\_push\_back *Alias of [array\_append](#array_append).* ## list\_push\_front *Alias of [array\_prepend](#array_prepend).* ## list\_remove *Alias of [array\_remove](#array_remove).* ## list\_remove\_all *Alias of [array\_remove\_all](#array_remove_all).* ## list\_remove\_n *Alias of [array\_remove\_n](#array_remove_n).* ## list\_repeat *Alias of [array\_repeat](#array_repeat).* ## list\_replace *Alias of [array\_replace](#array_replace).* ## list\_replace\_all *Alias of [array\_replace\_all](#array_replace_all).* ## list\_replace\_n *Alias of [array\_replace\_n](#array_replace_n).* ## list\_resize *Alias of [array\_resize](#array_resize).* ## list\_reverse *Alias of [array\_reverse](#array_reverse).* ## list\_slice *Alias of [array\_slice](#array_slice).* ## list\_sort *Alias of [array\_sort](#array_sort).* ## list\_to\_string *Alias of [array\_to\_string](#array_to_string).* ## list\_union *Alias of [array\_union](#array_union).* ## make\_array Returns an array using the specified input expressions. ```sql make_array(expression1[, ..., expression_n]) ``` ### Arguments * **expression\_n**: Expression to include in the output array. Can be a constant, column, or function, and any combination of arithmetic or string operators. #### Aliases * `make_list` [](#view-make_array-example) View `make_array` example ```sql SELECT make_array(1, 2, 3, 4, 5) AS make_array ``` | make\_array | |---------------| |[1, 2, 3, 4, 5]| ## make\_list *Alias of [make\_array](#make_array).* ## range Returns an array with values between the specified **start** and **stop** values generated at the specified **step**. The range `start..stop` contains all values greater than or equal to **start**and less than **stop** (`start <= x < stop`). If **start** is greater than or equal to **stop** (`start >= stop`), the function returns an empty array. *`range` is similar to [generate\_series](#generate_series), but does not include the upper bound (**stop**) in the output array.* ```sql range(start, stop, step) ``` ### Arguments * **start**: Start of the series. Supports integers, timestamps, dates, or string types that can be coerced to `Date32`. * **stop**: Upper bound of the series. Supports integers, timestamps, dates, or string types that can be coerced to `Date32`. The type must be the same as **start**. * **step**: Increase by step (cannot be `0`). Steps less than a day are only supported for ranges with the `TIMESTAMP` type. ##### Related functions [generate\_series](#generate_series) [](#view-range-example) View `range` example ```sql SELECT range(1, 5, 1) AS range ``` | range | |------------| |[1, 2, 3, 4]| [](#view-range-example-with-dates) View `range` example with dates ```sql SELECT range( DATE '2025-03-01', DATE '2025-08-01', INTERVAL '1 month' ) AS range ``` | range | |------------------------------------------------------------| |[2025-03-01, 2025-04-01, 2025-05-01, 2025-06-01, 2025-07-01]| [](#view-range-example-with-timestamps) View range example with timestamps ```sql SELECT range( '2025-01-01T00:00:00Z'::timestamp, '2025-01-01T06:00:00Z'::timestamp, INTERVAL '2 hours' ) AS range ``` | range | |---------------------------------------------------------------| |[2025-01-01T00:00:00, 2025-01-01T02:00:00, 2025-01-01T04:00:00]| ## string\_to\_array Splits a string into an array of substrings based on a delimiter. Any substrings matching the optional `null_str` argument are replaced with `NULL`. ```sql string_to_array(str, delimiter[, null_str]) ``` ### Arguments * **str**: String expression to split. * **delimiter**: Delimiter string to split on. * **null\_str**: *(Optional)* Substring values to replace with `NULL`. #### Aliases * `string_to_list` [](#view-string_to_array-example-with-comma-delimited-list) View `string_to_array` example with comma-delimited list ```sql SELECT string_to_array('abc, def, ghi', ', ') AS string_to_array ``` |string\_to\_array| |-----------------| | [abc, def, ghi] | [](#view-string_to_array-example-with-a-non-standard-delimiter) View `string_to_array` example with a non-standard delimiter ```sql SELECT string_to_array('abc##def', '##') AS string_to_array ``` |string\_to\_array| |-----------------| | [‘abc’, ‘def’] | [](#view-string_to_array-example-with-_null_-replacements) View `string_to_array` example with *NULL* replacements ```sql SELECT string_to_array('abc def', ' ', 'def') AS string_to_array ``` |string\_to\_array| |-----------------| | [‘abc’, NULL] | ## string\_to\_list *Alias of [string\_to\_array](#string_to_array).* | array_any_value | | --- | | array_any_value | | 1 | | array_append | | --- | | array_append | | [1, 2, 3, 4] | | array_concat | | --- | | array_concat | | [1, 2, 3, 4, 5, 6] | | array_dims(List([1,2,3,4,5,6])) | | --- | | array_dims(List([1,2,3,4,5,6])) | | [2, 3] | | array_distance | | --- | | array_distance | | 2.0 | | array_distinct(List([1,2,3,4])) | | --- | | array_distinct(List([1,2,3,4])) | | [1, 2, 3, 4] | | array_element | | --- | | array_element | | 3 | | array_except | | --- | | array_except | | [1, 2] | | array_has | | --- | | array_has | | true | | array_has_all | | --- | | array_has_all | | true | | array_has_any | | --- | | array_has_any | | true | | array_intersect | | --- | | array_intersect | | [3, 4] | | array_intersect | | --- | | array_intersect | | [] | | array_length | | --- | | array_length | | 5 | | dim1_length | dim2_length | | --- | --- | | dim1_length | dim2_length | | 2 | 5 | | array_max | | --- | | array_max | | 4 | | array_min | | --- | | array_min | | 1 | | array_ndims | | --- | | array_ndims | | 2 | | array_pop_back | | --- | | array_pop_back | | [1, 2] | | array_pop_front | | --- | | array_pop_front | | [2, 3] | | array_position | | --- | | array_position | | 2 | | array_position | | --- | | array_position | | 3 | | array_positions | | --- | | array_positions | | [1, 4] | | array_prepend | | --- | | array_prepend | | [1, 2, 3, 4] | | array_remove | | --- | | array_remove | | [1, 2, 3, 2, 1, 4] | | array_remove_all | | --- | | array_remove_all | | [1, 3, 1, 4] | | array_remove_n | | --- | | array_remove_n | | [1, 3, 2, 1, 4] | | array_repeat | | --- | | array_repeat | | [1, 1, 1] | | array_repeat | | --- | | array_repeat | | [John, John, John] | | array_repeat | | --- | | array_repeat | | [[1, 2], [1, 2]] | | array_replace | | --- | | array_replace | | [Joe, Jane, James, John] | | array_replace_all | | --- | | array_replace_all | | [Joe, Jane, James, Joe] | | array_replace_n | | --- | | array_replace_n | | [Joe, Jane, James, Joe, John] | | array_resize(List([1,2,3],5,0)) | | --- | | array_resize(List([1,2,3],5,0)) | | [1, 2, 3, 0, 0] | | array_reverse | | --- | | array_reverse | | [4, 3, 2, 1] | | array_slice | | --- | | array_slice | | [3, 4, 5, 6] | | array_sort | | --- | | array_sort | | [1, 2, 3] | | array_sort | | --- | | array_sort | | [cherry, banana, apple] | | array_sort | | --- | | array_sort | | [apple, banana, cherry, , ] | | array_to_string | | --- | | array_to_string | | 1,2,3,4,5,6,7,8 | | array_to_string | | --- | | array_to_string | | 1-2-3-4-5-?-7-8-? | | array_union | | --- | | array_union | | [1, 2, 3, 4, 5, 6] | | cardinality | | --- | | cardinality | | 8 | | empty | | --- | | empty | | false | | flattened | | --- | | flattened | | [1, 2, 3, 4] | | generate_series | | --- | | generate_series | | [1, 2, 3, 4, 5] | | generate_series | | --- | | generate_series | | [2025-03-01, 2025-04-01, 2025-05-01, 2025-06-01, 2025-07-01, 2025-08-01] | | generate_series | | --- | | generate_series | | [2025-01-01T00:00:00, 2025-01-01T02:00:00, 2025-01-01T04:00:00, 2025-01-01T06:00:00] | | make_array | | --- | | make_array | | [1, 2, 3, 4, 5] | | range | | --- | | range | | [1, 2, 3, 4] | | range | | --- | | range | | [2025-03-01, 2025-04-01, 2025-05-01, 2025-06-01, 2025-07-01] | | range | | --- | | range | | [2025-01-01T00:00:00, 2025-01-01T02:00:00, 2025-01-01T04:00:00] | | string_to_array | | --- | | string_to_array | | [abc, def, ghi] | | string_to_array | | --- | | string_to_array | | [‘abc’, ‘def’] | | string_to_array | | --- | | string_to_array | | [‘abc’, NULL] | --- # SQL binary string functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/binary-string/ Use binary string functions to encode and decode binary string values in SQL queries. ## decode Decode binary data from textual representation in string. ```sql decode(expression, format) ``` ### Arguments * **expression**: Expression containing encoded string data. Can be a constant, column, or function, and any combination of string operators. * **format**: Encoding format of the encoded string. Supported formats are: * base64 * hex ##### Related functions [encode](#encode) ## encode Encode binary data into a textual representation. ```sql encode(expression, format) ``` ### Arguments * **expression**: Expression containing string or binary data. Can be a constant, column, or function, and any combination of string operators. * **format**: Encoding format to use. Supported formats are: * base64 * hex ##### Related functions [decode](#decode) [](#view-encode-query-example) View `encode` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, encode(location::string, 'hex') AS location_encoded FROM weather ``` | location | location\_encoded | |-------------|--------------------------| | Concord | 436f6e636f7264 | | Hayward | 48617977617264 | |San Francisco|53616e204672616e636973636f| | location | location_encoded | | --- | --- | | location | location_encoded | | Concord | 436f6e636f7264 | | Hayward | 48617977617264 | | San Francisco | 53616e204672616e636973636f | --- # SQL cache functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/cache/ The InfluxDB 3 Enterprise SQL implementation supports the following functions that return data from InfluxDB 3 Enterprise caches. ## distinct\_cache Returns data from an InfluxDB 3 Enterprise distinct value cache. ```sql distinct_cache(table_name, cache_name) ``` ### Arguments * **table\_name**: Name of the table associated with the distinct value cache*(formatted as a string literal)*. * **cache\_name**: Name of the distinct value cache to query*(formatted as a string literal)*. [](#view-distinct_cache-query-example) View `distinct_cache` query example ```sql SELECT * FROM distinct_cache('example_table', 'exampleCacheName') ``` ## last\_cache Returns data from an InfluxDB 3 Enterprise last value cache. ```sql last_cache(table_name, cache_name) ``` ### Arguments * **table\_name**: Name of the table associated with the last value cache*(formatted as a string literal)*. * **cache\_name**: Name of the last value cache to query*(formatted as a string literal)*. [](#view-last_cache-query-example) View `last_cache` query example ```sql SELECT * FROM last_cache('example_table', 'exampleCacheName') ``` --- # SQL conditional functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/conditional/ The InfluxDB 3 Enterprise SQL implementation supports the following conditional functions for conditionally handling *null* values: ## coalesce Returns the first of its arguments that is not *null*. Returns *null* if all arguments are *null*. This function is often used to substitute a default value for *null* values. ```sql coalesce(expression1[, ..., expression_n]) ``` ##### Arguments * **expression1, expression\_n**: Expression to use if previous expressions are *null*. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary. [](#view-coalesce-query-example) View `coalesce` query example ```sql SELECT val1, val2, val3, coalesce(val1, val2, val3, 'quz') AS coalesce FROM (values ('foo', 'bar', 'baz'), (NULL, 'bar', 'baz'), (NULL, NULL, 'baz'), (NULL, NULL, NULL) ) data(val1, val2, val3) ``` |val1|val2|val3|coalesce| |----|----|----|--------| |foo |bar |baz | foo | | |bar |baz | bar | | | |baz | baz | | | | | quz | ## greatest Returns the greatest value in a list of expressions. Returns *null* if all expressions are *null*. ```sql greatest(expression1[, ..., expression_n]) ``` ##### Arguments * **expression1, expression\_n**: Expressions to compare and return the greatest value. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary. [](#view-greatest-query-example) View `greatest` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT time, greatest(a, b) AS greatest FROM numbers LIMIT 4 ``` | time | greatest | |-------------------|-------------------| |2023-01-01T01:20:00|\-0.515490223280789| |2023-01-01T01:21:00| 0.803201312042983 | |2023-01-01T01:22:00| 0.970938142399892 | |2023-01-01T01:23:00|0.0493748366311344 | ## ifnull *Alias of [nvl](#nvl).* ## least Returns the least value in a list of expressions. Returns *null* if all expressions are *null*. ```sql least(expression1[, ..., expression_n]) ``` ##### Arguments * **expression1, expression\_n**: Expressions to compare and return the least value. Can be a constant, column, or function, and any combination of arithmetic operators. Pass as many expression arguments as necessary. [](#view-least-query-example) View `least` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT time, least(a, b) AS least FROM numbers LIMIT 4 ``` | time | least | |-------------------|-------------------| |2023-01-01T00:10:00|\-1.08759833527982 | |2023-01-01T00:11:00|\-0.187620086586211| |2023-01-01T00:12:00|\-0.81371037157091 | |2023-01-01T00:13:00|\-0.339781659874945| ## nullif Returns *null* if *expression1* equals *expression2*; otherwise it returns *expression1*. This can be used to perform the inverse operation of [`coalesce`](#coalesce). ```sql nullif(expression1, expression2) ``` ##### Arguments * **expression1**: Expression to compare and return if equal to expression2. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression2**: Expression to compare to expression1. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-nullif-query-example) View `nullif` query example ```sql SELECT value, nullif(value, 'baz') AS nullif FROM (values ('foo'), ('bar'), ('baz') ) data(value) ``` |value|nullif| |-----|------| | foo | foo | | bar | bar | | baz | | ## nvl Returns *expression2* if *expression1* is *null*; otherwise it returns *expression1*. ```sql nvl(expression1, expression2) ``` ##### Arguments * **expression1**: Return this expression if not *null*. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression2**: Return this expression if *expression1* is *null*. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-nvl-query-example) View `nvl` query example ```sql SELECT value, nvl(value, 'baz') AS nvl FROM (values ('foo'), ('bar'), (NULL) ) data(value) ``` |value|nvl| |-----|---| | foo |foo| | bar |bar| | |baz| ## nvl2 Returns *expression2* if *expression1* is **not** *null*; otherwise it returns *expression3*. ```sql nvl2(expression1, expression2, expression3) ``` ##### Arguments * **expression1**: First expression to test for *null*. Can be a constant, column, or function, and any combination of operators. * **expression2**: Second expression to return if *expression1* is not *null*. Can be a constant, column, or function, and any combination of operators. * **expression3**: Expression to return if *expression1* is *null*. Can be a constant, column, or function, and any combination of operators. [](#view-nvl2-query-example) View `nvl2` query example ```sql SELECT val1, val2, val3, nvl2(val1, val2, val3) AS nvl2 FROM (values ('foo', 'bar', 'baz'), (NULL, 'bar', 'baz'), (NULL, NULL, 'baz'), ) data(val1, val2, val3) ``` |val1|val2|val3|nvl2| |----|----|----|----| |foo |bar |baz |bar | | |bar |baz |baz | | | |baz |baz | | val1 | val2 | val3 | coalesce | | --- | --- | --- | --- | | val1 | val2 | val3 | coalesce | | foo | bar | baz | foo | | | bar | baz | bar | | | | baz | baz | | | | | quz | | time | greatest | | --- | --- | | time | greatest | | 2023-01-01T01:20:00 | -0.515490223280789 | | 2023-01-01T01:21:00 | 0.803201312042983 | | 2023-01-01T01:22:00 | 0.970938142399892 | | 2023-01-01T01:23:00 | 0.0493748366311344 | | time | least | | --- | --- | | time | least | | 2023-01-01T00:10:00 | -1.08759833527982 | | 2023-01-01T00:11:00 | -0.187620086586211 | | 2023-01-01T00:12:00 | -0.81371037157091 | | 2023-01-01T00:13:00 | -0.339781659874945 | | value | nullif | | --- | --- | | value | nullif | | foo | foo | | bar | bar | | baz | | | value | nvl | | --- | --- | | value | nvl | | foo | foo | | bar | bar | | | baz | | val1 | val2 | val3 | nvl2 | | --- | --- | --- | --- | | val1 | val2 | val3 | nvl2 | | foo | bar | baz | bar | | | bar | baz | baz | | | | baz | baz | --- # SQL hashing functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/hashing/ Use hashing functions to hash string values in SQL queries using established hashing algorithms. ## digest Computes the binary hash of an expression using the specified algorithm. ```sql digest(expression, algorithm) ``` ### Arguments * **expression**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. * **algorithm**: String expression specifying algorithm to use. Must be one of the following: * md5 * sha224 * sha256 * sha384 * sha512 * blake2s * blake2b * blake3 [](#view-digest-query-example) View `digest` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, digest(location, 'sha256') AS location_digest FROM weather ``` | location | location\_digest | |-------------|----------------------------------------------------------------| | Concord |21e60acfb20fca2e38c5e51191d44235949de045912dca77e978333c1ec965a2| | Hayward |5d5b651a71084f3117cca3381ff7102efbc318c2026e1d09d5d4707354883102| |San Francisco|5aa34886f7f3741de8460690b636f4c8b7c2044df88e2e8adbb4f7e6f8534931| ## md5 Computes an MD5 128-bit checksum for a string expression. ```sql md5(expression) ``` ### Arguments * **expression**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. [](#view-md5-query-example) View `md5` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, md5(location) AS location_md5 FROM weather ``` | location | location\_md5 | |-------------|--------------------------------| | Concord |7c6a8e5769c20331e6a1b93a44021dbc| | Hayward |5823ccc25b256955c4e97cb4a23865ee| |San Francisco|f4334fdfa1c728eae375fe781e2e2d9d| ## sha224 Computes the SHA-224 hash of a binary string. ```sql sha224(expression) ``` ### Arguments * **expression**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. [](#view-sha224-query-example) View `sha224` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, sha224(location) AS location_sha224 FROM weather ``` | location | location\_sha224 | |-------------|--------------------------------------------------------| | Concord |3f6c0456de97d3d752672161db8e6f0df115979460a2276d64386114| | Hayward |767cfab6a1562e54f6906c8b5c9ee64583caa3f4b6c37a4298a7639f| |San Francisco|f723898f95bd53cdaf9c057b6f2cf0537b29e0bff077dabb39059b76| ## sha256 Computes the SHA-256 hash of a binary string. ```sql sha256(expression) ``` ### Arguments * **expression**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. [](#view-sha256-query-example) View `sha256` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, sha256(location) AS location_sha256 FROM weather ``` | location | location\_sha256 | |-------------|----------------------------------------------------------------| | Concord |21e60acfb20fca2e38c5e51191d44235949de045912dca77e978333c1ec965a2| | Hayward |5d5b651a71084f3117cca3381ff7102efbc318c2026e1d09d5d4707354883102| |San Francisco|5aa34886f7f3741de8460690b636f4c8b7c2044df88e2e8adbb4f7e6f8534931| ## sha384 Computes the SHA-384 hash of a binary string. ```sql sha384(expression) ``` ### Arguments * **expression**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. [](#view-sha384-query-example) View `sha384` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, sha384(location) AS location_sha384 FROM weather ``` | location | location\_sha384 | |-------------|------------------------------------------------------------------------------------------------| | Concord |19bf5ba95a6c4a28d2a997fa4e03e542d24b111593395b587a3b357af50cc687364f27b4e96935c234e2ae69236c7883| | Hayward |8b034ee0f7c8d191e145002722bf296a7f310f455458b9ffdbafcb2da2e2300d0f1e7868c6b1d512c81ae0c055979abd| |San Francisco|15c0c4a0e80f198e922235ee30a5c2b1e288c88f61bfc55ef9723903a0e948a7de04b65aa35b4ba46251b90460bab625| ## sha512 Computes the SHA-512 hash of a binary string. ```sql sha512(expression) ``` ### Arguments * **expression**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. [](#view-sha512-query-example) View `sha512` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT DISTINCT location, sha512(location) AS location_sha512 FROM weather ``` | location | location\_sha512 | |-------------|--------------------------------------------------------------------------------------------------------------------------------| | Concord |227e28fd661eebb39b10f49728d17b03e3a1f0cc2d2a413a9f124292d77249a2c0d7cf99941fd3d2eabea1b2894698bbd12c2cf3a7137ec67fca221ee57e0ca7| | Hayward |2db79fba2fc2888f9826d95ce6bb2605d95b841da61f049e8a92467de0d6ec3161f3919275208a3c385f6412d7848bfc957e81cde550f4f28cd834a332381142| |San Francisco|4c84f24d166978fb59d77779bf9a54ac112cba8e5f826c50440b2d3063fa0cbb0e2ccd30c93261a0f96805f71d15c3e726c9e00426161880e6a5dad267b2d883| | location | location_digest | | --- | --- | | location | location_digest | | Concord | 21e60acfb20fca2e38c5e51191d44235949de045912dca77e978333c1ec965a2 | | Hayward | 5d5b651a71084f3117cca3381ff7102efbc318c2026e1d09d5d4707354883102 | | San Francisco | 5aa34886f7f3741de8460690b636f4c8b7c2044df88e2e8adbb4f7e6f8534931 | | location | location_md5 | | --- | --- | | location | location_md5 | | Concord | 7c6a8e5769c20331e6a1b93a44021dbc | | Hayward | 5823ccc25b256955c4e97cb4a23865ee | | San Francisco | f4334fdfa1c728eae375fe781e2e2d9d | | location | location_sha224 | | --- | --- | | location | location_sha224 | | Concord | 3f6c0456de97d3d752672161db8e6f0df115979460a2276d64386114 | | Hayward | 767cfab6a1562e54f6906c8b5c9ee64583caa3f4b6c37a4298a7639f | | San Francisco | f723898f95bd53cdaf9c057b6f2cf0537b29e0bff077dabb39059b76 | | location | location_sha256 | | --- | --- | | location | location_sha256 | | Concord | 21e60acfb20fca2e38c5e51191d44235949de045912dca77e978333c1ec965a2 | | Hayward | 5d5b651a71084f3117cca3381ff7102efbc318c2026e1d09d5d4707354883102 | | San Francisco | 5aa34886f7f3741de8460690b636f4c8b7c2044df88e2e8adbb4f7e6f8534931 | | location | location_sha384 | | --- | --- | | location | location_sha384 | | Concord | 19bf5ba95a6c4a28d2a997fa4e03e542d24b111593395b587a3b357af50cc687364f27b4e96935c234e2ae69236c7883 | | Hayward | 8b034ee0f7c8d191e145002722bf296a7f310f455458b9ffdbafcb2da2e2300d0f1e7868c6b1d512c81ae0c055979abd | | San Francisco | 15c0c4a0e80f198e922235ee30a5c2b1e288c88f61bfc55ef9723903a0e948a7de04b65aa35b4ba46251b90460bab625 | | location | location_sha512 | | --- | --- | | location | location_sha512 | | Concord | 227e28fd661eebb39b10f49728d17b03e3a1f0cc2d2a413a9f124292d77249a2c0d7cf99941fd3d2eabea1b2894698bbd12c2cf3a7137ec67fca221ee57e0ca7 | | Hayward | 2db79fba2fc2888f9826d95ce6bb2605d95b841da61f049e8a92467de0d6ec3161f3919275208a3c385f6412d7848bfc957e81cde550f4f28cd834a332381142 | | San Francisco | 4c84f24d166978fb59d77779bf9a54ac112cba8e5f826c50440b2d3063fa0cbb0e2ccd30c93261a0f96805f71d15c3e726c9e00426161880e6a5dad267b2d883 | --- # SQL map functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/map/ Use map functions to create and operate on Arrow maps in SQL queries. ## element\_at *Alias of [map\_extract](#map_extract).* ## make\_map Returns an Arrow map with the specified key and value. ```sql make_map(key, value) ``` ### Arguments * **key**: Expression to use for the key. Can be a constant, column, function, or any combination of arithmetic or string operators. * **value**: Expression to use for the value. Can be a constant, column, function, or any combination of arithmetic or string operators. [](#view-make_map-query-example) View `make_map` query example *The following example uses the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT make_map(room, temp) AS make_map FROM home LIMIT 4 ``` | make\_map | |-------------------| | {Kitchen: 22.4} | |{Living Room: 22.2}| | {Kitchen: 22.7} | |{Living Room: 22.2}| ## map Returns an Arrow map with the specified key-value pairs. Keys are mapped to values by their positions in each respective list. Each *key* must be unique and non-null. ```sql map(key_list, value_list) -- or map { key: value, ... } ``` ### Arguments * **key\_list**: List of keys to use in the map. Each key must be unique and non-null. * **value\_list**: List of values to map to the corresponding keys. [](#view-map-query-example) View `map` query example ```sql SELECT map( [400, 401, 402, 403, 404], ['Bad Request', 'Unauthorized', 'Payment Required', 'Forbidden', 'Not Found'] ) AS map ``` | map | |--------------------------------------------------------------------------------------------| |{400: Bad Request, 401: Unauthorized, 402: Payment Required, 403: Forbidden, 404: Not Found}| [](#view-map-query-example-with-alternate-syntax) View `map` query example with alternate syntax ```sql SELECT map { 400: 'Bad Request', 401: 'Unauthorized', 402: 'Payment Required', 403: 'Forbidden', 404: 'Not Found' } AS map ``` | map | |--------------------------------------------------------------------------------------------| |{400: Bad Request, 401: Unauthorized, 402: Payment Required, 403: Forbidden, 404: Not Found}| ## map\_extract Returns a list containing the value for the given key or an empty list if the Returns a list containing the value for the given key, or an empty list if the key is not present in the map. The returned list will contain exactly one element (the value) when the key is found. ```sql map_extract(map, key) ``` ### Arguments * **map**: Map expression. Can be a constant, column, or function, and any combination of map operators. * **key**: Key to extract from the map. Can be a constant, column, or function, any combination of arithmetic or string operators, or a named expression of the previously listed. #### Aliases * `element_at` ##### Related functions [get\_field](/influxdb3/enterprise/reference/sql/functions/misc/#get_field) [](#view-map_extract-query-example) View `map_extract` query example The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data)to perform the a query that: * Defines a set of constants that includes a map that assigns integers to days of the week. * Queries the weather sample data and use `date_part` to extract an integer representing the day of the week of the row’s `time` value. * Uses `map_extract` and the output of `date_part` to return an array containing the name of the day of the week. * Uses bracket notation (`[i]`) to reference an element by index in the returned list (SQL arrays are 1-indexed, so `[1]` retrieves the first element). ``` WITH constants AS ( SELECT map( [0, 1, 2, 3, 4, 5, 6], ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'] ) AS days_of_week ) SELECT weather.time, map_extract(c.days_of_week, date_part('dow', time))[1] AS day_of_week FROM weather, constants AS c ORDER BY weather.time LIMIT 6 ``` | time |day\_of\_week| |-------------------|-------------| |2020-01-01T00:00:00| Wednesday | |2020-01-01T00:00:00| Wednesday | |2020-01-01T00:00:00| Wednesday | |2020-01-02T00:00:00| Thursday | |2020-01-02T00:00:00| Thursday | |2020-01-02T00:00:00| Thursday | ## map\_keys Returns a list of all keys in the map. ```sql map_keys(map) ``` ### Arguments * **map**: Map expression. Can be a constant, column, or function, and any combination of map operators. ##### Related functions [get\_field](/influxdb3/enterprise/reference/sql/functions/misc/#get_field) [](#view-map_keys-query-example) View `map_keys` query example ```sql SELECT map_keys(map {'a': 1, 'b': NULL, 'c': 3}) AS map_keys ``` |map\_keys| |---------| |[a, b, c]| ## map\_values Returns a list of all values in the map. ```sql map_values(map) ``` ### Arguments * **map**: Map expression. Can be a constant, column, or function, and any combination of map operators. [](#view-map_values-query-example) View `map_values` query example ```sql SELECT map_values(map {'a': 1, 'b': NULL, 'c': 3}) AS map_values ``` |map\_values| |-----------| | [1, , 3] | | make_map | | --- | | make_map | | {Kitchen: 22.4} | | {Living Room: 22.2} | | {Kitchen: 22.7} | | {Living Room: 22.2} | | map | | --- | | map | | {400: Bad Request, 401: Unauthorized, 402: Payment Required, 403: Forbidden, 404: Not Found} | | map | | --- | | map | | {400: Bad Request, 401: Unauthorized, 402: Payment Required, 403: Forbidden, 404: Not Found} | | time | day_of_week | | --- | --- | | time | day_of_week | | 2020-01-01T00:00:00 | Wednesday | | 2020-01-01T00:00:00 | Wednesday | | 2020-01-01T00:00:00 | Wednesday | | 2020-01-02T00:00:00 | Thursday | | 2020-01-02T00:00:00 | Thursday | | 2020-01-02T00:00:00 | Thursday | | map_keys | | --- | | map_keys | | [a, b, c] | | map_values | | --- | | map_values | | [1, , 3] | --- # SQL math functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/math/ The InfluxDB 3 Enterprise SQL implementation supports the following math functions for performing mathematic operations: ## abs Returns the absolute value of a number. ```sql abs(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-abs-query-example) View `abs` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT abs(temp) AS abs FROM home LIMIT 3 ``` |abs | |----| | 21 | | 23 | |22.7| ## acos Returns the arc cosine or inverse cosine of a number. ```sql acos(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-acos-query-example) View `acos` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT acos(temp * .01) AS acos FROM home LIMIT 3 ``` | acos | |------------------| |1.359221367036801 | |1.3387186439321834| |1.3418001704498232| ## acosh Returns the area hyperbolic cosine or inverse hyperbolic cosine of a number. ```sql acosh(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-acosh-query-example) View `acosh` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT acosh(temp) AS acosh FROM home LIMIT 3 ``` | acosh | |------------------| |3.737102242198924 | |3.8281684713331012| |3.8150265878962055| ## asin Returns the arc sine or inverse sine of a number. ```sql asin(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-asin-query-example) View `asin` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT asin(temp * .01) AS asin FROM home LIMIT 3 ``` | asin | |-------------------| |0.2115749597580956 | |0.23207768286271319| |0.22899615634507337| ## asinh Returns the area hyperbolic sine or inverse hyperbolic sine of a number. ```sql asinh(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-asinh-query-example) View `asinh` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT asinh(temp) AS asinh FROM home LIMIT 3 ``` | asinh | |------------------| |3.7382360302615427| |3.8291136516208812| |3.8159969160459988| ## atan Returns the arc tangent or inverse tangent of a number. ```sql atan(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-atan-query-example) View `atan` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT atan(temp * .01) AS atan FROM home LIMIT 3 ``` | atan | |-------------------| | 0.206992194219821 | |0.22606838799388393| |0.22321725383717603| ## atanh Returns the area hyperbolic tangent or inverse hyperbolic tangent of a number. ```sql atanh(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-atanh-query-example) View `atanh` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT atanh(temp * .01) AS atanh FROM home LIMIT 3 ``` | atanh | |-------------------| |0.21317134656485978| |0.2341894667593668 | |0.23102419806174476| ## atan2 Returns the arc tangent or inverse tangent of `expression_y / expression_x`. ```sql atan2(expression_y, expression_x) ``` ### Arguments * **expression\_y**: First numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_x**: Second numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-atan2-query-example) View `atan2` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT atan2(temp, hum) AS atan2 FROM home LIMIT 3 ``` | atan2 | |------------------| |0.5292859396993504| |0.5660139100632452| |0.5613335864315844| ## cbrt Returns the cube root of a number. ```sql cbrt(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-cbrt-query-example) View `cbrt` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT cbrt(temp) AS cbrt FROM home LIMIT 3 ``` | cbrt | |------------------| |2.7589241763811208| |2.843866979851566 | |2.831448188528187 | ## ceil Returns the nearest integer greater than or equal to a number. ```sql ceil(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-ceil-query-example) View `ceil` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT ceil(temp) AS ceil FROM home LIMIT 3 ``` |ceil| |----| | 21 | | 23 | | 23 | ## cos Returns the cosine of a number. ```sql cos(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-cos-query-example) View `cos` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT cos(temp) AS cos FROM home LIMIT 3 ``` | cos | |--------------------| |\-0.5477292602242684| |\-0.5328330203333975| |\-0.7591100556583898| ## cosh Returns the hyperbolic cosine of a number. ```sql cosh(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-cosh-query-example) View `cosh` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT cosh(temp) AS cosh FROM home LIMIT 3 ``` | cosh | |------------------| |659407867.2416073 | |4872401723.124452 | |3609563974.9715896| ## cot Returns the cotangent of a number. ```sql cot(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of operators. [](#view-cot-query-example) View `cot` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT cot(a) AS cot FROM numbers LIMIT 3 ``` | cot | |-------------------| |2.9293528483724423 | | 3.705570308335524 | |\-3.314652247498361| ## degrees Converts radians to degrees. ```sql degrees(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-degrees-query-example) View `degrees` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT degrees(a) AS degrees FROM numbers LIMIT 3 ``` | degrees | |--------------------| | 19.428488139031185 | |\-44.403317464348774| |\-52.771542485064785| ## exp Returns the base-e exponential of a number. ```sql exp(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to use as the exponent. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-exp-query-example) View `exp` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT exp(temp) AS exp FROM home LIMIT 3 ``` | exp | |------------------| |1318815734.4832146| |9744803446.248903 | |7219127949.943179 | ## factorial Returns 1 if value is less than 2. ```sql factorial(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Must be an integer (`BIGINT`). Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-factorial-query-example) View `factorial` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT factorial((b + 2)::BIGINT) AS factorial FROM numbers LIMIT 3 ``` |factorial| |---------| | 1 | | 2 | | 1 | ## floor Returns the nearest integer less than or equal to a number. ```sql floor(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-floor-query-example) View `floor` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT floor(temp) AS floor FROM home LIMIT 3 ``` |floor| |-----| | 21 | | 23 | | 22 | ## gcd Returns the greatest common divisor of `expression_x` and `expression_y`. Returns `0` if both inputs are zero. ```sql gcd(expression_x, expression_y) ``` ### Arguments * **expression\_x**: First numeric expression to operate on. Must be an integer (`BIGINT`). Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_y**: Second numeric expression to operate on. Must be an integer (`BIGINT`). Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-gcd-query-example) View `gcd` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT gcd((a * 5)::BIGINT, (b * 5)::BIGINT) AS gcd FROM numbers LIMIT 3 ``` |gcd| |---| | 1 | | 3 | | 2 | ## isnan Returns `true` if a given number is ±NaN, otherwise returns `false`. ```sql isnan(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Must be a float (`DOUBLE`). Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-isnan-query-example) View `isnan` query example *The following example uses the[Table value constructor](/influxdb3/enterprise/reference/sql/table-value-constructor/)to simulate sample data.* ```sql SELECT isnan(a) FROM (VALUES (4.56), ('NaN'::DOUBLE), (16.2) ) AS data(a) ``` |isnan| |-----| |false| |true | |false| ## iszero Returns `true` if the given number is ±0.0, otherwise returns `false`. ```sql iszero(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-iszero-query-example) View `iszero` query example *The following example uses the[Table value constructor](/influxdb3/enterprise/reference/sql/table-value-constructor/)to simulate sample data.* ```sql SELECT iszero(a) FROM (VALUES (0), (1), (2) ) AS data(a) ``` |iszero| |------| | true | |false | |false | ## lcm Returns the least common multiple of `expression_x` and `expression_y`. Returns `0` if either input is zero. ```sql lcm(expression_x, expression_y) ``` ### Arguments * **expression\_x**: First numeric expression to operate on. Must be an integer (`BIGINT`). Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_y**: Second numeric expression to operate on. Must be an integer (`BIGINT`). Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-lcm-query-example) View `lcm` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT lcm((a * 10)::BIGINT, (b * 10)::BIGINT) AS lcm FROM numbers LIMIT 3 ``` |lcm| |---| | 3 | | 7 | |36 | ## ln Returns the natural logarithm of a number. ```sql ln(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-ln-query-example) View `ln` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT ln(temp) AS ln FROM home LIMIT 3 ``` | ln | |------------------| |3.044522437723423 | |3.1354942159291497| |3.122364924487357 | ## log Returns the base-x logarithm of a number. ``` log([base, ]numeric_expression) ``` #### Arguments * **base**: Base numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. Default is `10`. * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-log-query-example) View `log` query example *The following example uses the sample data set provided in the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT temp, log(2, temp) AS temp_log2, log(4, temp) AS temp_log4, log(temp) AS temp_log10 FROM home LIMIT 3 ``` |temp| temp\_log2 | temp\_log4 | temp\_log10 | |----|-----------------|------------------|------------------| | 21 |4.392317422778761|2.1961587113893803|1.322219294733919 | | 23 |4.523561956057013|2.2617809780285065|1.3617278360175928| |22.7|4.504620392403553|2.2523101962017766|1.3560258571931225| ## log10 Returns the base-10 logarithm of a number. ```sql log10(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-log10-query-example) View `log10` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT log10(temp) AS log10 FROM home LIMIT 3 ``` | log10 | |------------------| |1.3222192947339193| |1.3617278360175928| |1.3560258571931227| ## log2 Returns the base-2 logarithm of a number. ```sql log2(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-log2-query-example) View `log2` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT log2(temp) AS log2 FROM home LIMIT 3 ``` | log2 | |-----------------| |4.392317422778761| |4.523561956057013| |4.504620392403552| ## nanvl Returns the first argument if it’s not `±NaN`. Otherwise returns the second argument. ```sql nanvl(expression_x, expression_y) ``` ### Arguments * **expression\_x**: Numeric expression to return if it’s not `NaN`. Can be a constant, column, or function, and any combination of arithmetic operators. * **expression\_y**: Numeric expression to return if the first expression is `NaN`. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-nanvl-query-example) View `nanvl` query example *The following example uses the[Table value constructor](/influxdb3/enterprise/reference/sql/table-value-constructor/)to simulate sample data.* ```sql SELECT nanvl(a, 0.0) AS nanvl FROM (VALUES (4.56), ('NaN'::DOUBLE), (16.2) ) AS data(a) ``` |nanvl| |-----| |4.56 | | 0 | |16.2 | ## pi Returns an approximate value of π. ```sql pi() ``` [](#view-pi-query-example) View `pi` query example ```sql SELECT pi() AS pi ``` | pi | |-----------------| |3.141592653589793| ## power Returns a base expression raised to the power of an exponent. ```sql power(base, exponent) ``` ##### Aliases * `pow` ### Arguments * **base**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **exponent**: Exponent numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-power-query-example) View `power` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT power(temp, hum * .1) AS power FROM home LIMIT 3 ``` | power | |------------------| |55817.099910217476| |85007.01501569824 | |78569.38332452129 | ## `pow` *Alias of [power](#power).* ## radians Converts degrees to radians. ```sql radians(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-radians-query-example) View `radians` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT radians(b) AS radians FROM numbers LIMIT 3 ``` | radians | |----------------------| | \-0.0028561101762876 | |0.0023917008411179744 | |\-0.008428949313343818| ## random Returns a random float value between 0 and 1. The random seed is unique to each row. ```sql random() ``` [](#view-random-query-example) View `random` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT temp * random() AS random FROM home LIMIT 3 ``` > [!Note] > Due to the nature of the function, your results will not match the results below. | random | |------------------| |0.5030770374815072| |12.938847036567514| |2.8204596545385385| ## round Rounds a number to the nearest integer. ```sql round(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-round-query-example) View `round` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT round(temp) AS round FROM home LIMIT 3 ``` |round| |-----| | 21 | | 23 | | 23 | ## signum Returns the sign of a number. Negative numbers return `-1`. Zero and positive numbers return `1`. ```sql signum(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-signum-query-example) View `signum` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT signum(temp - 23) AS signum FROM home LIMIT 3 ``` |signum| |------| | \-1 | | 1 | | \-1 | ## sin Returns the sine of a number. ```sql sin(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-sin-query-example) View `sin` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT sin(temp) AS sin FROM home LIMIT 3 ``` | sin | |--------------------| | 0.8366556385360561 | |\-0.8462204041751706| |\-0.6509623056662469| ## sinh Returns the hyperbolic sine of a number. ```sql sinh(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-sinh--query-example) View `sinh `query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT sinh(temp) AS sinh FROM home LIMIT 3 ``` | sinh | |------------------| |659407867.2416073 | |4872401723.124452 | |3609563974.9715896| ## sqrt Returns the square root of a number. ```sql sqrt(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-sqrt-query-example) View `sqrt` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT sqrt(temp) AS sqrt FROM home LIMIT 3 ``` | sqrt | |-----------------| |4.58257569495584 | |4.795831523312719| |4.764451699828638| ## tan Returns the tangent of a number. ```sql tan(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-tan-query-example) View `tan` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT tan(temp) AS tan FROM home LIMIT 3 ``` | tan | |--------------------| |\-1.5274985276366035| | 1.5881530833912738 | | 0.8575335036257101 | ## tanh Returns the hyperbolic tangent of a number. ```sql tanh(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-tanh-query-example) View `tanh` query example *The following example uses the[Random numbers sample dataset](/influxdb3/enterprise/reference/sample-data/#random-numbers-sample-data).* ```sql SELECT tanh(a) AS tanh FROM numbers LIMIT 3 ``` | tanh | |--------------------| |0.32666571332086836 | |\-0.6498182711525403| |\-0.7263877015335474| ## trunc Truncates a number toward zero (at the decimal point). ```sql trunc(numeric_expression) ``` ### Arguments * **numeric\_expression**: Numeric expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-trunc-query-example) View `trunc` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT trunc(temp) AS trunc FROM home LIMIT 3 ``` |trunc| |-----| | 21 | | 23 | | 22 | | abs | | --- | | abs | | 21 | | 23 | | 22.7 | | acos | | --- | | acos | | 1.359221367036801 | | 1.3387186439321834 | | 1.3418001704498232 | | acosh | | --- | | acosh | | 3.737102242198924 | | 3.8281684713331012 | | 3.8150265878962055 | | asin | | --- | | asin | | 0.2115749597580956 | | 0.23207768286271319 | | 0.22899615634507337 | | asinh | | --- | | asinh | | 3.7382360302615427 | | 3.8291136516208812 | | 3.8159969160459988 | | atan | | --- | | atan | | 0.206992194219821 | | 0.22606838799388393 | | 0.22321725383717603 | | atanh | | --- | | atanh | | 0.21317134656485978 | | 0.2341894667593668 | | 0.23102419806174476 | | atan2 | | --- | | atan2 | | 0.5292859396993504 | | 0.5660139100632452 | | 0.5613335864315844 | | cbrt | | --- | | cbrt | | 2.7589241763811208 | | 2.843866979851566 | | 2.831448188528187 | | ceil | | --- | | ceil | | 21 | | 23 | | 23 | | cos | | --- | | cos | | -0.5477292602242684 | | -0.5328330203333975 | | -0.7591100556583898 | | cosh | | --- | | cosh | | 659407867.2416073 | | 4872401723.124452 | | 3609563974.9715896 | | cot | | --- | | cot | | 2.9293528483724423 | | 3.705570308335524 | | -3.314652247498361 | | degrees | | --- | | degrees | | 19.428488139031185 | | -44.403317464348774 | | -52.771542485064785 | | exp | | --- | | exp | | 1318815734.4832146 | | 9744803446.248903 | | 7219127949.943179 | | factorial | | --- | | factorial | | 1 | | 2 | | 1 | | floor | | --- | | floor | | 21 | | 23 | | 22 | | gcd | | --- | | gcd | | 1 | | 3 | | 2 | | isnan | | --- | | isnan | | false | | true | | false | | iszero | | --- | | iszero | | true | | false | | false | | lcm | | --- | | lcm | | 3 | | 7 | | 36 | | ln | | --- | | ln | | 3.044522437723423 | | 3.1354942159291497 | | 3.122364924487357 | | temp | temp_log2 | temp_log4 | temp_log10 | | --- | --- | --- | --- | | temp | temp_log2 | temp_log4 | temp_log10 | | 21 | 4.392317422778761 | 2.1961587113893803 | 1.322219294733919 | | 23 | 4.523561956057013 | 2.2617809780285065 | 1.3617278360175928 | | 22.7 | 4.504620392403553 | 2.2523101962017766 | 1.3560258571931225 | | log10 | | --- | | log10 | | 1.3222192947339193 | | 1.3617278360175928 | | 1.3560258571931227 | | log2 | | --- | | log2 | | 4.392317422778761 | | 4.523561956057013 | | 4.504620392403552 | | nanvl | | --- | | nanvl | | 4.56 | | 0 | | 16.2 | | pi | | --- | | pi | | 3.141592653589793 | | power | | --- | | power | | 55817.099910217476 | | 85007.01501569824 | | 78569.38332452129 | | radians | | --- | | radians | | -0.0028561101762876 | | 0.0023917008411179744 | | -0.008428949313343818 | | random | | --- | | random | | 0.5030770374815072 | | 12.938847036567514 | | 2.8204596545385385 | | round | | --- | | round | | 21 | | 23 | | 23 | | signum | | --- | | signum | | -1 | | 1 | | -1 | | sin | | --- | | sin | | 0.8366556385360561 | | -0.8462204041751706 | | -0.6509623056662469 | | sinh | | --- | | sinh | | 659407867.2416073 | | 4872401723.124452 | | 3609563974.9715896 | | sqrt | | --- | | sqrt | | 4.58257569495584 | | 4.795831523312719 | | 4.764451699828638 | | tan | | --- | | tan | | -1.5274985276366035 | | 1.5881530833912738 | | 0.8575335036257101 | | tanh | | --- | | tanh | | 0.32666571332086836 | | -0.6498182711525403 | | -0.7263877015335474 | | trunc | | --- | | trunc | | 21 | | 23 | | 22 | --- # Miscellaneous SQL functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/misc/ The InfluxDB 3 Enterprise SQL implementation supports the following miscellaneous functions for performing a variety of operations: ## arrow\_cast Casts a value to a specific Arrow data type. ```sql arrow_cast(expression, datatype) ``` ### Arguments * **expression**: Expression to cast. Can be a constant, column, or function, and any combination of arithmetic or string operators. * **datatype**: [Arrow data type](/influxdb3/enterprise/reference/sql/data-types/#sql-and-arrow-data-types)to cast to. [](#view-arrow_cast-query-example) View `arrow_cast` query example *The following example uses the sample data set provided in[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT arrow_cast(time, 'Int64') AS time, arrow_cast(temp, 'Utf8') AS temp, arrow_cast(co, 'Float64')AS co FROM home LIMIT 1 ``` | time |temp|co | |-------------------|----|---| |1641024000000000000|21.0| 0 | ## arrow\_typeof Returns the underlying [Arrow data type](https://arrow.apache.org/datafusion/user-guide/sql/data_types.html)of the expression: ```sql arrow_typeof(expression) ``` ### Arguments * **expression**: Expression to evaluate. Can be a constant, column, or function, and any combination of arithmetic or string operators. [](#view-arrow_typeof-query-example) View `arrow_typeof` query example *The following example uses the sample data set provided in[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT arrow_typeof(time) AS time, arrow_typeof(room) AS room, arrow_typeof(temp) AS temp, arrow_typeof(co) AS co FROM home LIMIT 1 ``` | time | room | temp | co | |---------------------------|-----------------------|-------|-----| |Timestamp(Nanosecond, None)|Dictionary(Int32, Utf8)|Float64|Int64| ## get\_field Returns a field from a map or a struct with the specified key. > [!Note] > Typically, `get_field` is indirectly invoked via field access syntax such as`my_struct['field_name']` which results in the call:`get_field(my_struct, 'field_name')`. ```sql get_field(collection, field) ``` ### Arguments * **collection**: The map or struct to retrieve a field from. * **field**: The name of field the field to retrieve from the map or struct. Must evaluate to a string. [](#view-get_field-example-with-a-struct-column) View `get_field` example with a struct column ```sql SELECT get_field(influxdb_struct, 'version') AS influxdb_version FROM (VALUES (struct('influxdb' AS product, 'v1' AS version)), (struct('influxdb' AS product, 'v2' AS version)), (struct('influxdb' AS product, 'v3' AS version)) ) AS data(influxdb_struct) ``` |influxdb\_version| |-----------------| | v1 | | v2 | | v3 | [](#view-get_field-example-with-a-map-column) View `get_field` example with a map column ```sql SELECT get_field(influxdb_map, 'version') AS influxdb_version FROM (VALUES (map {'product': 'influxdb', 'version': 'v1'}), (map {'product': 'influxdb', 'version': 'v2'}), (map {'product': 'influxdb', 'version': 'v3'}) ) AS data(influxdb_map) ``` |influxdb\_version| |-----------------| | v1 | | v2 | | v3 | ## interpolate Fills null values in a specified aggregated column by interpolating values from existing values. Must be used with [`date_bin_gapfill`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill). ```sql interpolate(aggregate_expression) ``` ### Arguments * **aggregate\_expression**: Aggregate operation on a specified expression. The operation can use any [aggregate function](/influxdb3/enterprise/reference/sql/functions/aggregate/). The expression can be a constant, column, or function, and any combination of arithmetic operators supported by the aggregate function. ##### Related functions [date\_bin\_gapfill](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill),[locf](#locf) [](#view-interpolate-query-example) View `interpolate` query example *The following example uses the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as _time, room, interpolate(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY _time, room ``` | \_time | room |AVG(home.temp)| |--------------------|-----------|--------------| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | 22 | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | 22.85 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.1 | |2022-01-01T08:30:00Z|Living Room| 21.25 | |2022-01-01T09:00:00Z|Living Room| 21.4 | |2022-01-01T09:30:00Z|Living Room| 21.6 | |2022-01-01T10:00:00Z|Living Room| 21.8 | ## locf Fills null values in a specified aggregated column by carrying the last observed value forward. Must be used with [`date_bin_gapfill`](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill). *LOCF is an initialism of “last observation carried forward.”* ```sql locf(aggregate_expression) ``` ### Arguments * **aggregate\_expression**: Aggregate operation on a specified expression. The operation can use any [aggregate function](/influxdb3/enterprise/reference/sql/functions/aggregate/). The expression can be a constant, column, or function, and any combination of arithmetic operators supported by the aggregate function. ##### Related functions [date\_bin\_gapfill](/influxdb3/enterprise/reference/sql/functions/time-and-date/#date_bin_gapfill),[interpolate](#interpolate) [](#view-locf-query-example) View `locf` query example *The following example uses the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as _time, room, locf(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY _time, room ``` | \_time | room |AVG(home.temp)| |--------------------|-----------|--------------| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | 21 | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | 23 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.1 | |2022-01-01T08:30:00Z|Living Room| 21.1 | |2022-01-01T09:00:00Z|Living Room| 21.4 | |2022-01-01T09:30:00Z|Living Room| 21.4 | |2022-01-01T10:00:00Z|Living Room| 21.8 | ## version Returns the version of [DataFusion](https://datafusion.apache.org/) used by the query engine. > [!Note] > The `version()` function returns the DataFusion query engine version, not the > InfluxDB product version. > To identify your InfluxDB version, see [Identify version](/influxdb3/enterprise/admin/identify-version/). ```sql version() ``` [](#view-version-query-example) View `version` query example ```sql SELECT version() ``` | version() | |------------------------------------------| |Apache DataFusion 49.0.2, aarch64 on linux| The output includes the DataFusion version, CPU architecture, and operating system. | time | temp | co | | --- | --- | --- | | time | temp | co | | 1641024000000000000 | 21.0 | 0 | | time | room | temp | co | | --- | --- | --- | --- | | time | room | temp | co | | Timestamp(Nanosecond, None) | Dictionary(Int32, Utf8) | Float64 | Int64 | | influxdb_version | | --- | | influxdb_version | | v1 | | v2 | | v3 | | influxdb_version | | --- | | influxdb_version | | v1 | | v2 | | v3 | | _time | room | AVG(home.temp) | | --- | --- | --- | | _time | room | AVG(home.temp) | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | 22 | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | 22.85 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | 21.25 | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | 21.6 | | 2022-01-01T10:00:00Z | Living Room | 21.8 | | _time | room | AVG(home.temp) | | --- | --- | --- | | _time | room | AVG(home.temp) | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | 21 | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | 23 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | 21.1 | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | 21.4 | | 2022-01-01T10:00:00Z | Living Room | 21.8 | | version() | | --- | | version() | | Apache DataFusion 49.0.2, aarch64 on linux | --- # SQL regular expression functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/regular-expression/ The InfluxDB 3 Enterprise SQL implementation uses the[PCRE-like](https://en.wikibooks.org/wiki/Regular_Expressions/Perl-Compatible_Regular_Expressions)regular expression [syntax](https://docs.rs/regex/latest/regex/#syntax)(excluding some features such as look-around and back-references) and supports the following regular expression functions: ## regexp\_count Returns the number of matches that a regular expression has in a string. ```sql regexp_count(str, regexp[, start, flags]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. * **regexp**: Regular expression to operate on. Can be a constant, column, or function, and any combination of operators. * **start**: Optional start position (the first position is 1) to search for the regular expression. Can be a constant, column, or function. * **flags**: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: * **i**: (insensitive) Ignore case when matching. * **m**: (multi-line) `^` and `$` match the beginning and end of a line, respectively. * **s**: (single-line) `.` matches newline (`\n`). * **R**: (CRLF) When multi-line mode is enabled, `\r\n` is used to delimit lines. * **U**: (ungreedy) Swap the meaning of `x*` and `x*?`. [](#view-regexp_count-query-example) View `regexp_count` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, regexp_count(room::STRING, '[Ro]', 1, 'i') AS regexp_count FROM home ``` | room |regexp\_count| |-----------|-------------| | Kitchen | 0 | |Living Room| 3 | ## regexp\_like True if a regular expression has at least one match in a string; false otherwise. ```sql regexp_like(str, regexp[, flags]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **regexp**: Regular expression to test against the string expression. Can be a constant, column, or function. * **flags**: Optional regular expression flags that control the behavior of the regular expression. The following flags are supported: * **i**: (insensitive) Ignore case when matching. * **m**: (multi-line) `^` and `$` match the beginning and end of a line, respectively. * **s**: (single-line) `.` matches newline (`\n`). * **R**: (CRLF) When multi-line mode is enabled, `\r\n` is used to delimit lines. * **U**: (ungreedy) Swap the meaning of `x*` and `x*?`. [](#view-regexp_like-query-example) View `regexp_like` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, regexp_like(room::STRING, 'R', 'i') AS regexp_like FROM home ``` | room |regexp\_like| |-----------|------------| | Kitchen | false | |Living Room| true | ## regexp\_match Returns a list of regular expression matches in a string. ```sql regexp_match(str, regexp, flags) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **regexp**: Regular expression to match against. Can be a constant, column, or function. * **flags**: Regular expression flags that control the behavior of the regular expression. The following flags are supported. * **i**: (insensitive) Ignore case when matching. [](#view-regexp_match-query-example) View `regexp_match` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* > [!Note] > `regexp_match` returns a *list* Arrow type. > Use *bracket notation* to reference a value in the list. > Lists use 1-based indexing. ```sql SELECT DISTINCT room, regexp_match(room::STRING, '.{3}')[1] AS regexp_match FROM home ``` | room |regexp\_match| |-----------|-------------| | Kitchen | Kit | |Living Room| Liv | ## regexp\_replace Replaces substrings in a string that match a regular expression. ```sql regexp_replace(str, regexp, replacement, flags) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **regexp**: Regular expression to match against. Can be a constant, column, or function. * **replacement**: Replacement string expression. Can be a constant, column, or function, and any combination of string operators. * **flags**: Regular expression flags that control the behavior of the regular expression. The following flags are supported. * **g**: (global) Search globally and don’t return after the first match. * **i**: (insensitive) Ignore case when matching. [](#view-regexp_replace-query-example) View `regexp_replace` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, regexp_replace(room::STRING, '\sRoom', '', 'gi') AS regexp_replace FROM home ``` | room |regexp\_replace| |-----------|---------------| | Kitchen | Kitchen | |Living Room| Living | [regular expressions](/influxdb3/enterprise/tags/regular-expressions/)[sql](/influxdb3/enterprise/tags/sql/) | room | regexp_count | | --- | --- | | room | regexp_count | | Kitchen | 0 | | Living Room | 3 | | room | regexp_like | | --- | --- | | room | regexp_like | | Kitchen | false | | Living Room | true | | room | regexp_match | | --- | --- | | room | regexp_match | | Kitchen | Kit | | Living Room | Liv | | room | regexp_replace | | --- | --- | | room | regexp_replace | | Kitchen | Kitchen | | Living Room | Living | --- # SQL selector functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/selector/ SQL selector functions are designed to work with time series data. They behave similarly to aggregate functions in that they take a collection of data and return a single value. However, selectors are unique in that they return a *struct* that contains a **time value** in addition to the computed value. ## How do selector functions work? Each selector function returns an [Arrow *struct*](https://arrow.apache.org/docs/format/Columnar.html#struct-layout)(similar to a JSON object) representing a single time and value from the specified column in the each group. What time and value get returned depend on the logic in the selector function. For example, `selector_first` returns the value of specified column in the first row of the group.`selector_max` returns the maximum value of the specified column in the group. ### Selector struct schema The struct returned from a selector function has two properties: * **time**: `time` value in the selected row * **value**: value of the specified column in the selected row ```js {time: 2023-01-01T00:00:00Z, value: 72.1} ``` ### Selector functions in use In your `SELECT` statement, execute a selector function and use bracket notation to reference properties of the [returned struct](#selector-struct-schema) to populate the column value: ```sql SELECT selector_first(temp, time)['time'] AS time, selector_first(temp, time)['value'] AS temp, room FROM home GROUP BY room ``` ## Selector functions ### selector\_min Returns the smallest value of a selected column and a timestamp. ```sql selector_min(expression, timestamp) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of string or arithmetic operators. * **timestamp**: Time expression. Can be a constant, column, or function. [](#view-selector_min-query-example) View `selector_min` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT location, selector_min(temp_min, time)['time'] AS time, selector_min(temp_min, time)['value'] AS min_temp FROM weather GROUP BY location ``` | location | time | min\_temp | +—————+———————+———-+ | Concord | 2022-01-02T00:00:00 | 28.0 | | Hayward | 2021-01-26T00:00:00 | 32.0 | | San Francisco | 2022-01-02T00:00:00 | 35.0 | ### selector\_max Returns the largest value of a selected column and a timestamp. ```sql selector_max(expression, timestamp) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of string or arithmetic operators. * **timestamp**: Time expression. Can be a constant, column, or function. [](#view-selector_max-query-example) View `selector_max` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT location, selector_max(temp_max, time)['time'] AS time, selector_max(temp_max, time)['value'] AS max_temp FROM weather GROUP BY location ``` | location | time |max\_temp| |-------------|-------------------|---------| | Concord |2020-09-07T00:00:00| 112.0 | | Hayward |2022-09-06T00:00:00| 107.0 | |San Francisco|2020-09-06T00:00:00| 102.0 | ### selector\_first Returns the first value ordered by time ascending. ```sql selector_first(expression, timestamp) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of string or arithmetic operators. * **timestamp**: Time expression. Can be a constant, column, or function. [](#view-selector_first-query-example) View `selector_first` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT location, selector_first(precip, time)['time'] AS time, selector_first(precip, time)['value'] AS first_precip FROM (SELECT * FROM weather WHERE precip > 0) GROUP BY location ``` | location | time |first\_precip| |-------------|-------------------|-------------| | Concord |2020-01-08T00:00:00| 0.01 | | Hayward |2020-01-09T00:00:00| 0.17 | |San Francisco|2020-01-07T00:00:00| 0.03 | ### selector\_last Returns the last value ordered by time ascending. ```sql selector_last(expression, timestamp) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of string or arithmetic operators. * **timestamp**: Time expression. Can be a constant, column, or function. [](#view-selector_last-query-example) View `selector_last` query example *The following example uses the[NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data).* ```sql SELECT location, selector_last(precip, time)['time'] AS time, selector_last(precip, time)['value'] AS last_precip FROM (SELECT * FROM weather WHERE precip > 0) GROUP BY location ``` | location | time |last\_precip| |-------------|-------------------|------------| | Concord |2022-12-31T00:00:00| 3.04 | | Hayward |2022-12-31T00:00:00| 4.34 | |San Francisco|2022-12-31T00:00:00| 3.67 | #### Related * [Aggregate data with SQL](/influxdb3/enterprise/query-data/sql/aggregate-select/) | location | time | max_temp | | --- | --- | --- | | location | time | max_temp | | Concord | 2020-09-07T00:00:00 | 112.0 | | Hayward | 2022-09-06T00:00:00 | 107.0 | | San Francisco | 2020-09-06T00:00:00 | 102.0 | | location | time | first_precip | | --- | --- | --- | | location | time | first_precip | | Concord | 2020-01-08T00:00:00 | 0.01 | | Hayward | 2020-01-09T00:00:00 | 0.17 | | San Francisco | 2020-01-07T00:00:00 | 0.03 | | location | time | last_precip | | --- | --- | --- | | location | time | last_precip | | Concord | 2022-12-31T00:00:00 | 3.04 | | Hayward | 2022-12-31T00:00:00 | 4.34 | | San Francisco | 2022-12-31T00:00:00 | 3.67 | --- # SQL string functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/string/ The InfluxDB 3 Enterprise SQL implementation supports the following string functions for operating on string values: ## ascii Returns the ASCII value of the first character in a string. > [!Note] > `ascii` returns a 32-bit integer. > To use with InfluxDB, [cast the return value to 64-bit integer](/influxdb3/enterprise/query-data/sql/cast-types/#cast-to-an-integer). ```sql ascii(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [chr](#chr) [](#view-ascii-query-example) View `ascii` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, ascii(room)::BIGINT AS ascii FROM home ``` | room |ascii| |-----------|-----| | Kitchen | 75 | |Living Room| 76 | ## bit\_length Returns the bit length of a string. > [!Note] > `bit_length` returns a 32-bit integer. > To use with InfluxDB, [cast the return value to 64-bit integer](/influxdb3/enterprise/query-data/sql/cast-types/#cast-to-an-integer). ```sql bit_length(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [length](#length), [octet\_length](#octet_length) [](#view-bit_length-query-example) View `bit_length` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, bit_length(room)::BIGINT AS bit_length FROM home ``` | room |bit\_length| |-----------|-----------| |Living Room| 88 | | Kitchen | 56 | ## btrim Trims the specified trim string from the start and end of a string. If no trim string is provided, all whitespace is removed from the start and end of the input string. ```sql btrim(str[, trim_str]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **trim\_str**: String expression to trim from the beginning and end of the input string. Can be a constant, column, or function, and any combination of arithmetic operators.*Default is whitespace characters*. ##### Related functions [ltrim](#ltrim),[rtrim](#rtrim),[trim](#trim) [](#view-btrim-query-example) View `btrim` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, btrim(room::STRING, ' Room') AS btrim FROM home ``` | room | btrim | |-----------|-------| |Living Room|Living | | Kitchen |Kitchen| ## char\_length *Alias of [length](#length).* ## character\_length *Alias of [length](#length).* ## chr Returns the character with the specified ASCII or Unicode code value. ``` chr(expression) ``` #### Arguments * **expression**: Expression containing the ASCII or Unicode code value to operate on. Can be a constant, column, or function, and any combination of arithmetic or string operators. ##### Related functions [ascii](#ascii) [](#view-chr-query-example) View `chr` query example ```sql SELECT ascii, chr(ascii) AS chr FROM (values (112), (75), (214) ) data(ascii) ``` |ascii|chr| |-----|---| | 112 | p | | 75 | K | | 214 | Ö | ## concat Concatenates multiple strings together. ```sql concat(str[, ..., str_n]) ``` ### Arguments * **str**: String expression to concatenate. Can be a constant, column, or function, and any combination of string operators. * **str\_n**: Subsequent string expression to concatenate. ##### Related functions [concat\_ws](#concat_ws) [](#view-concat-query-example) View `concat` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT concat('At ', time::STRING, ', the ', room, ' was ', temp::STRING, '°C.') AS concat FROM home LIMIT 3 ``` | concat | |-----------------------------------------------| |At 2022-01-01T08:00:00, the Kitchen was 21.0°C.| |At 2022-01-01T09:00:00, the Kitchen was 23.0°C.| |At 2022-01-01T10:00:00, the Kitchen was 22.7°C.| ## concat\_ws Concatenates multiple strings together with a specified separator. ```sql concat_ws(separator, str[, ..., str_n]) ``` ### Arguments * **separator**: Separator to insert between concatenated strings. * **str**: String expression to concatenate. Can be a constant, column, or function, and any combination of string operators. * **str\_n**: Subsequent string expression to concatenate. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [concat](#concat) [](#view-concat_ws-query-example) View `concat_ws` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT concat_ws(' -- ', time::STRING, room, temp::STRING) AS concat_ws FROM home LIMIT 3 ``` | concat\_ws | |--------------------------------------| |2022-01-01T08:00:00 -- Kitchen -- 21.0| |2022-01-01T09:00:00 -- Kitchen -- 23.0| |2022-01-01T10:00:00 -- Kitchen -- 22.7| ## contains Returns true if a string contains a search string (case-sensitive). ```sql contains(str, search_str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of operators. * **search\_str**: The string to search for in *str*. ## ends\_with Tests if a string ends with a substring. ```sql ends_with(str, substr) ``` ### Arguments * **str**: String expression to test. Can be a constant, column, or function, and any combination of string operators. * **substr**: Substring to test for. [](#view-ends_with-query-example) View `ends_with` query example ```sql SELECT string, ends_with(string, 'USA') AS ends_with FROM (values ('New York, USA'), ('London, UK'), ('San Francisco, USA') ) data(string) ``` | string |ends\_with| |------------------|----------| | New York, USA | true | | London, UK | false | |San Francisco, USA| true | ## find\_in\_set Returns the position of a string in a comma-delimited list of substrings. Returns 0 if the string is not in the list of substrings. ```sql find_in_set(str, strlist) ``` ### Arguments * **str**: String expression to find in `strlist`. * **strlist**: A string containing a comma-delimited list of substrings. [](#view-find_in_set-query-example) View `find_in_set` query example ```sql SELECT string, find_in_set(string, 'Isaac,John,Sara') AS find_in_set FROM (values ('John'), ('Sarah'), ('Isaac') ) data(string) ``` |string|find\_in\_set| |------|-------------| | John | 2 | |Sarah | 0 | |Isaac | 1 | ## initcap Capitalizes the first character in each word in the input string. Words are delimited by non-alphanumeric characters. ```sql initcap(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [lower](#lower),[upper](#upper) [](#view-initcap-query-example) View `initcap` query example ```sql SELECT string, initcap(string) AS initcap FROM (values ('hello world'), ('hello-world'), ('hello_world') ) data(string) ``` | string | initcap | |------------|------------| |hello world |Hello World | |hello-world |Hello-World | |hello\_world|Hello\_World| ## instr Returns the location where a substring first appears in a string (starting at 1). If the substring is not in the string, the function returns 0. ```sql instr(str, substr) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **substr**: Substring expression to search for. Can be a constant, column, or function, and any combination of string operators. [](#view-instr-query-example) View `instr` query example ```sql SELECT string, instr(string, 'neighbor') AS instr FROM (values ('good neighbor'), ('bad neighbor'), ('next-door neighbor'), ('friend') ) data(string) ``` | string |instr| |------------------|-----| | good neighbor | 6 | | bad neighbor | 5 | |next-door neighbor| 11 | | friend | 0 | ## left Returns a specified number of characters from the left side of a string. ```sql left(str, n) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **n**: Number of characters to return. ##### Related functions [right](#right) [](#view-left-query-example) View `left` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, left(room::STRING, 3) AS left FROM home ``` | room |left| |-----------|----| | Kitchen |Kit | |Living Room|Liv | ## length Returns the number of characters in a string. > [!Note] > `char_length` returns a 32-bit integer. > To use with InfluxDB, [cast the return value to 64-bit integer](/influxdb3/enterprise/query-data/sql/cast-types/#cast-to-an-integer). ```sql length(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Aliases * char\_length * character\_length ##### Related functions [bit\_length](#bit_length),[octet\_length](#octet_length) [](#view-length-query-example) View `length` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, length(room)::BIGINT AS length FROM home ``` | room |length| |-----------|------| | Kitchen | 7 | |Living Room| 11 | ## levenshtein Returns the [Levenshtein distance](https://en.wikipedia.org/wiki/Levenshtein_distance)between two strings. ```sql levenshtein(str1, str2) ``` ### Arguments * **str1**: First string expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **str2**: Second string expression to operate on. Can be a constant, column, or function, and any combination of string operators. [](#view-levenshtein-query-example) View `levenshtein` query example ```sql SELECT string1, string2, levenshtein(string1, string2) AS levenshtein FROM (values ('kitten', 'sitting'), ('puppy', 'jumping'), ('cow', 'lowing') ) data(string1, string2) ``` |string1|string2|levenshtein| |-------|-------|-----------| |kitten |sitting| 3 | | puppy |jumping| 5 | | cow |lowing | 4 | ## lower Converts a string to lower-case. ```sql lower(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [initcap](#initcap),[upper](#upper) [](#view-lower-query-example) View `lower` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, lower(room::STRING) AS lower FROM home ``` | room | lower | |-----------|-----------| | Kitchen | kitchen | |Living Room|living room| ## lpad Pads the left side of a string with another string to a specified string length. ```sql lpad(str, n[, padding_str]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **n**: String length to pad to. * **padding\_str**: String expression to pad with. Can be a constant, column, or function, and any combination of string operators.*Default is a space.* ##### Related functions [rpad](#rpad) [](#view-lpad-query-example) View `lpad` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, lpad(room::STRING, 14, '-') AS lpad FROM home ``` | room | lpad | |-----------|---------------| | Kitchen |\-------Kitchen| |Living Room|\---Living Room| ## ltrim Removes leading spaces from a string. ```sql ltrim(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [btrim](#btrim),[rtrim](#rtrim),[trim](#trim) [](#view-ltrim-query-example) View `ltrim` query example ```sql SELECT string, ltrim(string) AS ltrim FROM (values (' Leading spaces'), ('Trailing spaces '), (' Leading and trailing spaces ') ) data(string) ``` | string | ltrim | |---------------------------|---------------------------| | Leading spaces | Leading spaces | | Trailing spaces | Trailing spaces | |Leading and trailing spaces|Leading and trailing spaces| ## octet\_length Returns the length of a string in bytes. > [!Note] > `length` returns a 32-bit integer. > To use with InfluxDB, [cast the return value to 64-bit integer](/influxdb3/enterprise/query-data/sql/cast-types/#cast-to-an-integer). ```sql octet_length(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [bit\_length](#bit_length),[length](#length) [](#view-octet_length-query-example) View `octet_length` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, octet_length(room)::BIGINT AS octet_length FROM home ``` | room |octet\_length| |-----------|-------------| |Living Room| 11 | | Kitchen | 7 | ## overlay Replaces part of a string with another substring using a specified starting position and number of characters to replace. ```sql overlay(str PLACING substr FROM pos [FOR count]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **substr**: Substring to use to replace part of the specified string (`str`). Can be a constant, column, or function, and any combination of string operators. * **pos**: Start position of the substring replacement (`substr`). * **count**: Number of characters in the string (`str`) to replace with the substring (`substr`) beginning from the start position (`pos`). If not specified, the function uses the length of the substring. [](#view-overlay-query-example) View `overlay` query example ```sql SELECT string, overlay(string PLACING '****' FROM 1 FOR 12) AS overlay FROM (values ('2223000048410010'), ('2222420000001113'), ('4917484589897107') ) data(string) ``` | string | overlay | |----------------|------------| |2223000048410010|\*\*\*\*0010| |2222420000001113|\*\*\*\*1113| |4917484589897107|\*\*\*\*7107| ## position Returns the position of a substring in a string. ```sql position(substr IN str) ``` ### Arguments * **substr**: Substring expression to search for. Can be a constant, column, or function, and any combination of string operators. * **str**: String expression to search. Can be a constant, column, or function, and any combination of string operators. [](#view-position-query-example) View `position` query example ```sql SELECT string, position('oo' IN string) AS position FROM (values ('cool'), ('scoop'), ('ice cream') ) data(string) ``` | string |position| |---------|--------| | cool | 2 | | scoop | 3 | |ice cream| 0 | ## repeat Returns a string with an input string repeated a specified number of times. ```sql repeat(str, n) ``` ### Arguments * **str**: String expression to repeat. Can be a constant, column, or function, and any combination of string operators. * **n**: Number of times to repeat the input string. [](#view-repeat-query-example) View `repeat` query example ```sql SELECT string, repeat(string, 3) AS repeat FROM (values ('foo '), ('bar '), ('baz ') ) data(string) ``` |string| repeat | |------|-----------| | foo |foo foo foo| | bar |bar bar bar| | baz |baz baz baz| ## replace Replaces all occurrences of a specified substring in a string with a new substring. ```sql replace(str, substr, replacement) ``` ### Arguments * **str**: String expression to repeat. Can be a constant, column, or function, and any combination of string operators. * **substr**: Substring expression to replace in the input string. Can be a constant, column, or function, and any combination of string operators. * **replacement**: Replacement substring expression. Can be a constant, column, or function, and any combination of string operators. [](#view-replace-query-example) View `replace` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, replace(room::STRING, ' ', '_') AS replace FROM home ``` | room | replace | |-----------|------------| | Kitchen | Kitchen | |Living Room|Living\_Room| ## reverse Reverses the character order of a string. ```sql reverse(str) ``` ### Arguments * **str**: String expression to repeat. Can be a constant, column, or function, and any combination of string operators. [](#view-reverse-query-example) View `reverse` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, reverse(room::STRING) AS reverse FROM home ``` | room | reverse | |-----------|-----------| | Kitchen | nehctiK | |Living Room|mooR gniviL| ## right Returns a specified number of characters from the right side of a string. ```sql right(str, n) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **n**: Number of characters to return. ##### Related functions [left](#left) [](#view-right-query-example) View `right` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, right(room::STRING, 3) AS right FROM home ``` | room |right| |-----------|-----| |Living Room| oom | | Kitchen | hen | ## rpad Pads the right side of a string with another string to a specified string length. ```sql rpad(str, n[, padding_str]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **n**: String length to pad to. * **padding\_str**: String expression to pad with. Can be a constant, column, or function, and any combination of string operators.*Default is a space.* ##### Related functions [lpad](#lpad) [](#view-rpad-query-example) View `rpad` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, rpad(room::STRING, 14, '-') AS rpad FROM home ``` | room | rpad | |-----------|--------------| | Kitchen |Kitchen-------| |Living Room|Living Room---| ## rtrim Removes trailing spaces from a string. ```sql rtrim(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [btrim](#btrim),[ltrim](#ltrim),[trim](#trim) [](#view-rtrim-query-example) View `rtrim` query example ```sql SELECT string, rtrim(string) AS rtrim FROM (values (' Leading spaces'), ('Trailing spaces '), (' Leading and trailing spaces ') ) data(string) ``` | string | rtrim | |---------------------------|----------------------------| | Leading spaces | Leading spaces | | Trailing spaces | Trailing spaces | |Leading and trailing spaces| Leading and trailing spaces| ## split\_part Splits a string based on a specified delimiter and returns the substring in the specified position. ```sql split_part(str, delimiter, pos) ``` ### Arguments * **str**: String expression to spit. Can be a constant, column, or function, and any combination of string operators. * **delimiter**: String or character to split on. * **pos**: Position of the part to return. [](#view-split_part-query-example) View `split_part` query example ```sql SELECT url, split_part(url, '.', 1) AS split_part FROM (values ('www.influxdata.com'), ('docs.influxdata.com'), ('community.influxdata.com') ) data(url) ``` | url |split\_part| |------------------------|-----------| | www.influxdata.com | www | | docs.influxdata.com | docs | |community.influxdata.com| community | ## starts\_with Tests if a string starts with a substring. ```sql starts_with(str, substr) ``` ### Arguments * **str**: String expression to test. Can be a constant, column, or function, and any combination of string operators. * **substr**: Substring to test for. [](#view-starts_with-query-example) View `starts_with` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, starts_with(room::STRING, 'Kit') AS starts_with FROM home ``` | room |starts\_with| |-----------|------------| | Kitchen | true | |Living Room| false | ## strpos Returns the starting position of a specified substring in a string. Positions begin at 1. If the substring does not exist in the string, the function returns 0. > [!Note] > `strpos` returns a 32-bit integer. > To use with InfluxDB, [cast the return value to 64-bit integer](/influxdb3/enterprise/query-data/sql/cast-types/#cast-to-an-integer). ```sql strpos(str, substr) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **substr**: Substring expression to search for. Can be a constant, column, or function, and any combination of string operators. [](#view-strpos-query-example) View `strpos` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, strpos(room::STRING, 'Room')::BIGINT AS strpos FROM home ``` | room |strpos| |-----------|------| | Kitchen | 0 | |Living Room| 8 | ## substr Extracts a substring of a specified number of characters from a specific starting position in a string. ```sql substr(str, start_pos[, length]) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **start\_pos**: Character position to start the substring at. The first character in the string has a position of 1. * **length**: Number of characters to extract. If not specified, returns the rest of the string after the start position. [](#view-substr-query-example) View `substr` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, substr(room::STRING, 3, 5) AS substr FROM home ``` | room |substr| |-----------|------| |Living Room| ving | | Kitchen |tchen | ## substr\_index Returns the substring that occurs before or after the specified number (`count`) of delimiter (`delimiter`) occurrences in a string (`str`). If the count is positive, the function returns everything to the left of the final delimiter (counting from the left). If the count is negative, the function returns everything to the right of the final delimiter (counting from the right). ```sql substr_index(str, delimiter, count) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **delimiter**: String expression to use to delimit substrings in the string (`str`). Can be a constant, column, or function, and any combination of string operators. * **count**: The Nth occurrence of the delimiter (`delimiter`) to split on. Can be a constant, column, or function, and any combination of arithmetic operators. Supports positive and negative numbers. [](#view-substr_index-query-example) View `substr_index` query example ```sql SELECT url, substr_index(url, '.', 1) AS subdomain, substr_index(url, '.', -1) AS tld FROM (values ('docs.influxdata.com'), ('community.influxdata.com'), ('cloud2.influxdata.com') ) data(url) ``` | url |subdomain|tld| |------------------------|---------|---| | docs.influxdata.com | docs |com| |community.influxdata.com|community|com| | arrow.apache.org | arrow |org| ## substring *Alias of [substr](#substr).* ## substring\_index *Alias of [substr\_index](#substr_index).* ## translate Translates characters in a string to specified translation characters. ```sql translate(str, chars, translation) ``` * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. * **chars**: Characters to translate. * **translation**: Translation characters. Translation characters replace only characters at the same position in the **chars** string. [](#view-translate-query-example) View `translate` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, translate(room::STRING, 'Rom', 'sOn') AS translate FROM home ``` | room | translate | |-----------|-----------| |Living Room|Living sOOn| | Kitchen | Kitchen | ## to\_hex Converts an integer to a hexadecimal string. ```sql to_hex(int) ``` ### Arguments * **int**: Integer expression to convert. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-to_hex-query-example) View `to_hex` query example ```sql SELECT int, to_hex(int) AS to_hex FROM (values (123), (345), (678) ) data(int) ``` |int|to\_hex| |---|-------| |123| 7b | |345| 159 | |678| 2a6 | ## trim Removes leading and trailing spaces from a string. ```sql trim(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [btrim](#btrim),[ltrim](#ltrim),[rtrim](#rtrim) [](#view-trim-query-example) View `trim` query example ```sql SELECT string, trim(string) AS trim FROM (values (' Leading spaces'), ('Trailing spaces '), (' Leading and trailing spaces ') ) data(string) ``` | string | trim | |---------------------------|---------------------------| | Leading spaces | Leading spaces | | Trailing spaces | Trailing spaces | |Leading and trailing spaces|Leading and trailing spaces| ## upper Converts a string to upper-case. ```sql upper(str) ``` ### Arguments * **str**: String expression to operate on. Can be a constant, column, or function, and any combination of string operators. ##### Related functions [initcap](#initcap),[lower](#lower) [](#view-upper-query-example) View `upper` query example *The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT DISTINCT room, upper(room::STRING) AS upper FROM home ``` | room | upper | |-----------|-----------| |Living Room|LIVING ROOM| | Kitchen | KITCHEN | ## uuid Returns a UUID v4 string value that is unique per row. ```sql uuid() ``` [](#view-uuid-query-example) View `uuid` query example *The following example uses the sample data set provided in the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT room, uuid() AS uuid FROM (SELECT DISTINCT room FROM home) ``` | room | uuid | |-----------|------------------------------------| | Kitchen |f0b41da9-e334-4b7d-b925-a54ca6b082f3| |Living Room|c31be90e-c4ed-4304-b633-47b969ef3ab6| | room | ascii | | --- | --- | | room | ascii | | Kitchen | 75 | | Living Room | 76 | | room | bit_length | | --- | --- | | room | bit_length | | Living Room | 88 | | Kitchen | 56 | | room | btrim | | --- | --- | | room | btrim | | Living Room | Living | | Kitchen | Kitchen | | ascii | chr | | --- | --- | | ascii | chr | | 112 | p | | 75 | K | | 214 | Ö | | concat | | --- | | concat | | At 2022-01-01T08:00:00, the Kitchen was 21.0°C. | | At 2022-01-01T09:00:00, the Kitchen was 23.0°C. | | At 2022-01-01T10:00:00, the Kitchen was 22.7°C. | | concat_ws | | --- | | concat_ws | | 2022-01-01T08:00:00 -- Kitchen -- 21.0 | | 2022-01-01T09:00:00 -- Kitchen -- 23.0 | | 2022-01-01T10:00:00 -- Kitchen -- 22.7 | | string | ends_with | | --- | --- | | string | ends_with | | New York, USA | true | | London, UK | false | | San Francisco, USA | true | | string | find_in_set | | --- | --- | | string | find_in_set | | John | 2 | | Sarah | 0 | | Isaac | 1 | | string | initcap | | --- | --- | | string | initcap | | hello world | Hello World | | hello-world | Hello-World | | hello_world | Hello_World | | string | instr | | --- | --- | | string | instr | | good neighbor | 6 | | bad neighbor | 5 | | next-door neighbor | 11 | | friend | 0 | | room | left | | --- | --- | | room | left | | Kitchen | Kit | | Living Room | Liv | | room | length | | --- | --- | | room | length | | Kitchen | 7 | | Living Room | 11 | | string1 | string2 | levenshtein | | --- | --- | --- | | string1 | string2 | levenshtein | | kitten | sitting | 3 | | puppy | jumping | 5 | | cow | lowing | 4 | | room | lower | | --- | --- | | room | lower | | Kitchen | kitchen | | Living Room | living room | | room | lpad | | --- | --- | | room | lpad | | Kitchen | -------Kitchen | | Living Room | ---Living Room | | string | ltrim | | --- | --- | | string | ltrim | | Leading spaces | Leading spaces | | Trailing spaces | Trailing spaces | | Leading and trailing spaces | Leading and trailing spaces | | room | octet_length | | --- | --- | | room | octet_length | | Living Room | 11 | | Kitchen | 7 | | string | overlay | | --- | --- | | string | overlay | | 2223000048410010 | ****0010 | | 2222420000001113 | ****1113 | | 4917484589897107 | ****7107 | | string | position | | --- | --- | | string | position | | cool | 2 | | scoop | 3 | | ice cream | 0 | | string | repeat | | --- | --- | | string | repeat | | foo | foo foo foo | | bar | bar bar bar | | baz | baz baz baz | | room | replace | | --- | --- | | room | replace | | Kitchen | Kitchen | | Living Room | Living_Room | | room | reverse | | --- | --- | | room | reverse | | Kitchen | nehctiK | | Living Room | mooR gniviL | | room | right | | --- | --- | | room | right | | Living Room | oom | | Kitchen | hen | | room | rpad | | --- | --- | | room | rpad | | Kitchen | Kitchen------- | | Living Room | Living Room--- | | string | rtrim | | --- | --- | | string | rtrim | | Leading spaces | Leading spaces | | Trailing spaces | Trailing spaces | | Leading and trailing spaces | Leading and trailing spaces | | url | split_part | | --- | --- | | url | split_part | | www.influxdata.com | www | | docs.influxdata.com | docs | | community.influxdata.com | community | | room | starts_with | | --- | --- | | room | starts_with | | Kitchen | true | | Living Room | false | | room | strpos | | --- | --- | | room | strpos | | Kitchen | 0 | | Living Room | 8 | | room | substr | | --- | --- | | room | substr | | Living Room | ving | | Kitchen | tchen | | url | subdomain | tld | | --- | --- | --- | | url | subdomain | tld | | docs.influxdata.com | docs | com | | community.influxdata.com | community | com | | arrow.apache.org | arrow | org | | room | translate | | --- | --- | | room | translate | | Living Room | Living sOOn | | Kitchen | Kitchen | | int | to_hex | | --- | --- | | int | to_hex | | 123 | 7b | | 345 | 159 | | 678 | 2a6 | | string | trim | | --- | --- | | string | trim | | Leading spaces | Leading spaces | | Trailing spaces | Trailing spaces | | Leading and trailing spaces | Leading and trailing spaces | | room | upper | | --- | --- | | room | upper | | Living Room | LIVING ROOM | | Kitchen | KITCHEN | | room | uuid | | --- | --- | | room | uuid | | Kitchen | f0b41da9-e334-4b7d-b925-a54ca6b082f3 | | Living Room | c31be90e-c4ed-4304-b633-47b969ef3ab6 | --- # SQL struct functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/struct/ Use struct functions to create Arrow structs in SQL queries. ## named\_struct Returns an *Arrow struct* using the specified name and input expressions pairs. ```sql named_struct(expression1_name, expression1_input[, ..., expression_n_name, expression_n_input]) ``` ### Arguments * **expression\_n\_name**: Name of the column field. Must be a constant string. * **expression\_n\_input**: Expression to include in the output struct. Can be a constant, column, or function, and any combination of arithmetic or string operators. ##### Related functions [get\_field](/influxdb3/enterprise/reference/sql/functions/misc/#get_field) [](#view-named_struct-query-example) View `named_struct` query example *The following example uses the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT named_struct('time', time, 'temperature', temp, 'humidity', hum) AS named_struct FROM home WHERE room = 'Kitchen' LIMIT 4 ``` | named\_struct | |--------------------------------------------------------------| |{time: 2022-01-01T13:00:00, temperature: 22.8, humidity: 36.5}| |{time: 2022-01-01T12:00:00, temperature: 22.5, humidity: 36.0}| |{time: 2022-01-01T15:00:00, temperature: 22.7, humidity: 36.2}| |{time: 2022-01-01T18:00:00, temperature: 23.3, humidity: 36.9}| ## row *Alias of [`struct`](#struct).* ## struct Returns an *Arrow struct* using the specified input expressions optionally named. Fields in the returned struct use the optional name or the `cN` naming convention. Fields in the returned struct use the `cN` naming convention (for example: `c0`, `c1`, `c2`, etc.) unless you specify custom names using the `AS` operator within individual expressions. ```sql struct(expression1[, ..., expression_n]) ``` ### Arguments * **expression1, expression\_n**: Expression to include in the output struct. Can be a constant, column, or function, and any combination of arithmetic or string operators. ### Aliases * row ##### Related functions [get\_field](/influxdb3/enterprise/reference/sql/functions/misc/#get_field) [](#view-struct-query-example) View `struct` query example *The following example uses the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT struct(time, temp, hum) AS struct FROM home WHERE room = 'Kitchen' LIMIT 4 ``` | struct | |---------------------------------------------| |{c0: 2022-01-01T13:00:00, c1: 22.8, c2: 36.5}| |{c0: 2022-01-01T12:00:00, c1: 22.5, c2: 36.0}| |{c0: 2022-01-01T15:00:00, c1: 22.7, c2: 36.2}| |{c0: 2022-01-01T18:00:00, c1: 23.3, c2: 36.9}| [](#view-struct-query-example-with-named-fields) View `struct` query example with named fields Use the `AS` operator in a `struct` expression argument to assign a name to the struct field. *The following example uses the[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data).* ```sql SELECT struct(time AS 'time', temp AS 'temperature', hum) AS struct FROM home WHERE room = 'Kitchen' LIMIT 4 ``` | struct | |--------------------------------------------------------| |{time: 2022-01-01T13:00:00, temperature: 22.8, c2: 36.5}| |{time: 2022-01-01T12:00:00, temperature: 22.5, c2: 36.0}| |{time: 2022-01-01T15:00:00, temperature: 22.7, c2: 36.2}| |{time: 2022-01-01T18:00:00, temperature: 23.3, c2: 36.9}| | named_struct | | --- | | named_struct | | {time: 2022-01-01T13:00:00, temperature: 22.8, humidity: 36.5} | | {time: 2022-01-01T12:00:00, temperature: 22.5, humidity: 36.0} | | {time: 2022-01-01T15:00:00, temperature: 22.7, humidity: 36.2} | | {time: 2022-01-01T18:00:00, temperature: 23.3, humidity: 36.9} | | struct | | --- | | struct | | {c0: 2022-01-01T13:00:00, c1: 22.8, c2: 36.5} | | {c0: 2022-01-01T12:00:00, c1: 22.5, c2: 36.0} | | {c0: 2022-01-01T15:00:00, c1: 22.7, c2: 36.2} | | {c0: 2022-01-01T18:00:00, c1: 23.3, c2: 36.9} | | struct | | --- | | struct | | {time: 2022-01-01T13:00:00, temperature: 22.8, c2: 36.5} | | {time: 2022-01-01T12:00:00, temperature: 22.5, c2: 36.0} | | {time: 2022-01-01T15:00:00, temperature: 22.7, c2: 36.2} | | {time: 2022-01-01T18:00:00, temperature: 23.3, c2: 36.9} | --- # SQL time and date functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/time-and-date/ The InfluxDB 3 Enterprise SQL implementation supports time and date functions that are useful when working with time series data. ## current\_date Returns the current UTC date. > [!Note] > `current_date` returns a `DATE32` Arrow type, which isn’t supported by InfluxDB. > To use with InfluxDB, [cast the return value to a timestamp or string](/influxdb3/enterprise/query-data/sql/cast-types/). The `current_date()` return value is determined at query time and returns the same date, no matter when in the query plan the function executes. ``` current_date() ``` [](#view-current_date-query-example) View `current_date` query example *The following example uses the sample data set provided in[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT time, temp, current_date()::TIMESTAMP AS current_date FROM home WHERE time > current_date()::TIMESTAMP - INTERVAL '5 years' LIMIT 3 ``` | time |temp| current\_date | |--------------------|----|--------------------| |2022-01-01T08:00:00Z| 21 |2026-11-17T00:00:00Z| |2022-01-01T09:00:00Z| 23 |2026-11-17T00:00:00Z| |2022-01-01T10:00:00Z|22.7|2026-11-17T00:00:00Z| ## current\_time Returns the current UTC time. The `current_time()` return value is determined at query time and returns the same time, no matter when in the query plan the function executes. ``` current_time() ``` [](#view-current_time-query-example) View `current_time` query example *The following example uses the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT time, temp, current_time() AS current_time FROM home LIMIT 3 ``` | time |temp| current\_time | |--------------------|----|--------------------| |2022-01-01T08:00:00Z| 21 |2021-01-01T00:00:00Z| |2022-01-01T09:00:00Z| 23 |2021-01-01T00:00:00Z| |2022-01-01T10:00:00Z|22.7|2021-01-01T00:00:00Z| ## current\_timestamp *Alias of [now](#now).* ## date\_bin Calculates time intervals and returns the start of the interval nearest to the specified timestamp. Use `date_bin` to downsample time series data by grouping rows into time-based “bins” or “windows” and applying an aggregate or selector function to each window. For example, if you “bin” or “window” data into 15-minute intervals, an input timestamp of `2023-01-01T18:18:18Z` will be updated to the start time of the 15-minute bin it is in: `2023-01-01T18:15:00Z`. ```sql date_bin(interval, expression[, origin_timestamp]) ``` ### Arguments * **interval**: Bin interval. Supports the following interval units: * nanoseconds * microseconds * milliseconds * seconds * minutes * hours * days * weeks * months * years * century * **expression**: Time expression to operate on. Can be a constant, column, or function. * **origin\_timestamp**: Starting point used to determine bin boundaries.*Default is the Unix epoch.* [](#view-date_bin-query-example) View `date_bin` query example The following query returns the daily average of water levels in the queried time range. ```sql SELECT date_bin(INTERVAL '1 day', time, TIMESTAMP '1970-01-01 00:00:00Z') AS time, avg("water_level") AS water_level_avg FROM "h2o_feet" WHERE time >= timestamp '2019-09-10T00:00:00Z' AND time <= timestamp '2019-09-20T00:00:00Z' GROUP BY 1 ORDER BY time DESC ``` | time |water\_level\_avg | |------------------------|------------------| |2019-09-17T00:00:00.000Z|4.370175687443861 | |2019-09-16T00:00:00.000Z|4.6034785848437485| |2019-09-15T00:00:00.000Z|4.680651501506248 | |2019-09-14T00:00:00.000Z|4.857683652395836 | |2019-09-13T00:00:00.000Z|4.911051520291668 | |2019-09-12T00:00:00.000Z|4.763990784533338 | |2019-09-11T00:00:00.000Z|4.6582452515041695| |2019-09-10T00:00:00.000Z|4.608425018785421 | ## date\_bin\_gapfill Calculates time intervals and returns the start of the interval nearest to the specified timestamp. If no rows exist in a time interval, a new row is inserted with a `time` value set to the interval start time, all columns in the `GROUP BY` clause populated, and null values in aggregate columns. Use `date_bin_gapfill` with [`interpolate`](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate)or [`locf`](/influxdb3/enterprise/reference/sql/functions/misc/#locf) to[fill gaps in data](/influxdb3/enterprise/query-data/sql/fill-gaps/)at specified time intervals. ```sql date_bin_gapfill(interval, expression[, origin_timestamp]) ``` > [!Note] > `date_bin_gapfill` requires [time bounds](/influxdb3/enterprise/query-data/sql/basic-query/#query-data-within-time-boundaries)in the `WHERE` clause. ### Arguments * **interval**: Bin interval. Supports the following interval units: * nanoseconds * microseconds * milliseconds * seconds * minutes * hours * days * weeks The following intervals are not currently supported: * months * years * century * **expression**: Time expression to operate on. Can be a constant, column, or function. * **origin\_timestamp**: Starting point used to determine bin boundaries.*Default is the Unix epoch.* ##### Related functions [interpolate](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate),[locf](/influxdb3/enterprise/reference/sql/functions/misc/#locf) [](#use-date_bin_gapfill-to-insert-rows-when-no-rows-exists) Use `date_bin_gapfill` to insert rows when no rows exists *The following example uses the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as _time, room, avg(temp) as temp FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY _time, room ``` | \_time | room |temp| |--------------------|-----------|----| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | | |2022-01-01T10:00:00Z| Kitchen |22.7| |2022-01-01T08:00:00Z|Living Room|21.1| |2022-01-01T08:30:00Z|Living Room| | |2022-01-01T09:00:00Z|Living Room|21.4| |2022-01-01T09:30:00Z|Living Room| | |2022-01-01T10:00:00Z|Living Room|21.8| [](#use-date_bin_gapfill-to-fill-gaps-in-data) Use `date_bin_gapfill` to fill gaps in data Use `interpolate` and `locf` to fill the null values in rows inserted by`date_bin_gapfill`. *The following examples use the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* [interpolate](#interpolate)[locf](#locf) The example below uses [`interpolate`](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate)to fill null values by interpolating values between non-null values. ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as _time, room, interpolate(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY _time, room ``` | \_time | room |AVG(home.temp)| |--------------------|-----------|--------------| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | 22 | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | 22.85 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.1 | |2022-01-01T08:30:00Z|Living Room| 21.25 | |2022-01-01T09:00:00Z|Living Room| 21.4 | |2022-01-01T09:30:00Z|Living Room| 21.6 | |2022-01-01T10:00:00Z|Living Room| 21.8 | The example below uses [`locf`](/influxdb3/enterprise/reference/sql/functions/misc/#locf)to fill null values by carrying the last observed value forward. ```sql SELECT date_bin_gapfill(INTERVAL '30 minutes', time) as _time, room, locf(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY _time, room ``` | \_time | room |AVG(home.temp)| |--------------------|-----------|--------------| |2022-01-01T08:00:00Z| Kitchen | 21 | |2022-01-01T08:30:00Z| Kitchen | 21 | |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T09:30:00Z| Kitchen | 23 | |2022-01-01T10:00:00Z| Kitchen | 22.7 | |2022-01-01T08:00:00Z|Living Room| 21.1 | |2022-01-01T08:30:00Z|Living Room| 21.1 | |2022-01-01T09:00:00Z|Living Room| 21.4 | |2022-01-01T09:30:00Z|Living Room| 21.4 | |2022-01-01T10:00:00Z|Living Room| 21.8 | ## date\_bin\_wallclock Calculates time intervals using the timezone of a specified time value and returns the start of the interval nearest to the specified timestamp. Use `date_bin_wallclock` to downsample time series data by grouping rows into time-based “bins” or “windows” that are based off “wall clock” times in a specific timezone and applying an aggregate or selector function to each window. ### Time zone shifts Many regions use time zone shifts (such as daylight saving time (DST)). If a wall clock time bin starts at a time that does not exist in the specified time zone, the timestamp is adjusted to the time that is the same offset from the start of the day in that time zone. If a wall clock time represents an ambiguous time in the region then the behavior depends on the size of the specified interval. If the interval is larger than the difference between the two possible timestamps, then the earlier timestamp is used. Otherwise, the function uses the timestamp that matches the UTC offset of the input timestamp. ```sql date_bin_wallclock(interval, expression[, origin_timestamp]) ``` ### Arguments * **interval**: Bin interval. Supports the following interval units: * nanoseconds * microseconds * milliseconds * seconds * minutes * hours * days * weeks > [!Note] > `date_bin_wallclock` does *not* support month-, year-, or century-based intervals. * **expression**: Time expression to operate on. Can be a constant, column, or function. The output timestamp uses the time zone from this time expression. * **origin\_timestamp**: Starting point used to determine bin boundaries. This must be a “wall clock” timestamp (no time zone).*Default is the Unix epoch.* #### Avoid bins in time zone discontinuities #### [Time zone shifts](#time-zone-shifts) result in *discontinuities*–breaks in the continuity of time intervals (losing an hour or gaining an hour)–that can result in unexpected timestamps when using `date_bin_wallclock`. Avoid using an `interval` and `origin_timestamp` combination that results in a bin falling inside a time discontinuity. As a general rule, use either the default `origin_timestamp` or an origin timestamp with an offset relative to the Unix epoch that is equal to your specified `interval`. [](#view-time-zone-discontinuity-example) View time zone discontinuity example The following query illustrates how two timestamps, only one minute apart, result in timestamps two hours apart when binned across a daylight saving boundary: ``` SELECT tz('2020-10-25T02:29:00+01:00', 'Europe/Paris') AS original_time, date_bin_wallclock( INTERVAL '1 hour', tz('2020-10-25T02:29:00+01:00', 'Europe/Paris'), '1970-01-01T00:30:00' ) AT TIME ZONE 'UTC' AS utc_bin_time UNION SELECT tz('2020-10-25T02:30:00+01:00', 'Europe/Paris') AS original_time, date_bin_wallclock( INTERVAL '1 hour', tz('2020-10-25T02:30:00+01:00', 'Europe/Paris'), '1970-01-01T00:30:00' ) AT TIME ZONE 'UTC' AS utc_bin_time ORDER BY original_time; ``` | original\_time | utc\_bin\_time | |-------------------------|--------------------| |2020-10-25T02:29:00+01:00|2020-10-24T23:30:00Z| |2020-10-25T02:30:00+01:00|2020-10-25T01:30:00Z| [](#view-date_bin_wallclock-query-example) View `date_bin_wallclock` query example The following query uses the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol)and returns the 12-hour average temperature for each room using times in the`America/Los_Angeles` time zone. ```sql SELECT date_bin_wallclock(INTERVAL '12 hours', tz(time, 'America/Los_Angeles')) AS time, room, avg(temp) AS avg_temp FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T20:00:00Z' GROUP BY 1, room ``` | time | room | avg\_temp | |-------------------------|-----------|------------------| |2022-01-01T00:00:00-08:00| Kitchen |22.61666666666667 | |2022-01-01T12:00:00-08:00| Kitchen | 22.7 | |2022-01-01T00:00:00-08:00|Living Room|22.166666666666668| |2022-01-01T12:00:00-08:00|Living Room| 22.2 | ## date\_bin\_wallclock\_gapfill Calculates time intervals using the timezone of a specified time value and returns the start of the interval nearest to the specified timestamp. If no rows exist in a time interval, a new row is inserted with a `time` value set to the interval start time, all columns in the `GROUP BY` clause populated, and null values in aggregate columns. Use `date_bin_wallclock_gapfill` with [`interpolate`](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate)or [`locf`](/influxdb3/enterprise/reference/sql/functions/misc/#locf) to[fill gaps in data](/influxdb3/enterprise/query-data/sql/fill-gaps/)at specified time intervals in a specified time zone. ### Time zone shifts Many regions use time zone shifts (such as daylight saving time (DST)). If a wall clock time bin starts at a time that does not exist in the specified time zone, the timestamp is adjusted to the time that is the same offset from the start of the day in that time zone. If a wall clock time represents an ambiguous time in the region then the behavior depends on the size of the specified interval. If the interval is larger than the difference between the two possible timestamps, then the earlier timestamp is used. Otherwise, the function uses the timestamp that matches the UTC offset of the input timestamp. ```sql date_bin_wallclock_gapfill(interval, expression[, origin_timestamp]) ``` > [!Note] > `date_bin_wallclock_gapfill` requires [time bounds](/influxdb3/enterprise/query-data/sql/basic-query/#query-data-within-time-boundaries)in the `WHERE` clause. ### Arguments * **interval**: Bin interval. Supports the following interval units: * nanoseconds * microseconds * milliseconds * seconds * minutes * hours * days * weeks > [!Note] > `date_bin_wallclock_gapfill` does *not* support month-, year-, or century-based intervals. * **expression**: Time expression to operate on. Can be a constant, column, or function. The output timestamp uses the time zone from this time expression. * **origin\_timestamp**: Starting point used to determine bin boundaries. This must be a “wall clock” timestamp (no time zone).*Default is the Unix epoch.* #### Avoid bins in time zone discontinuities #### [Time zone shifts](#time-zone-shifts) result in *discontinuities*–breaks in the continuity of time intervals (losing an hour or gaining an hour)–that can result in unexpected timestamps when using `date_bin_wallclock_gapfill`. Avoid using an `interval` and `origin_timestamp` combination that results in a bin falling inside a time discontinuity. As a general rule, use either the default `origin_timestamp` or an origin timestamp with an offset relative to the Unix epoch that is equal to your specified `interval`. [View time zone discontinuity example](#view-time-zone-discontinuity-example) ##### Related functions [interpolate](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate),[locf](/influxdb3/enterprise/reference/sql/functions/misc/#locf) [](#use-date_bin_wallclock_gapfill-to-insert-rows-when-no-rows-exists) Use `date_bin_wallclock_gapfill` to insert rows when no rows exists *The following example uses the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ```sql SELECT date_bin_wallclock_gapfill(INTERVAL '30 minutes', tz(time, 'America/Los_Angeles')) as time, room, avg(temp) as temp FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY 1, room ``` | time | room |temp| |-------------------------|-----------|----| |2022-01-01T00:00:00-08:00| Kitchen | 21 | |2022-01-01T00:30:00-08:00| Kitchen | | |2022-01-01T01:00:00-08:00| Kitchen | 23 | |2022-01-01T01:30:00-08:00| Kitchen | | |2022-01-01T02:00:00-08:00| Kitchen |22.7| |2022-01-01T00:00:00-08:00|Living Room|21.1| |2022-01-01T00:30:00-08:00|Living Room| | |2022-01-01T01:00:00-08:00|Living Room|21.4| |2022-01-01T01:30:00-08:00|Living Room| | |2022-01-01T02:00:00-08:00|Living Room|21.8| [](#use-date_bin_wallclock_gapfill-to-fill-gaps-in-data) Use `date_bin_wallclock_gapfill` to fill gaps in data Use `interpolate` and `locf` to fill the null values in rows inserted by`date_bin_wallclock_gapfill`. *The following examples use the sample data set provided in the[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* [interpolate](#interpolate-2)[locf](#locf-2) The example below uses [`interpolate`](/influxdb3/enterprise/reference/sql/functions/misc/#interpolate)to fill null values by interpolating values between non-null values. ```sql SELECT date_bin_wallclock_gapfill(INTERVAL '30 minutes', tz(time, 'America/Los_Angeles')) as time, room, interpolate(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY 1, room ``` | time | room |interpolate(avg(home.temp))| |-------------------------|-----------|---------------------------| |2022-01-01T00:00:00-08:00| Kitchen | 21 | |2022-01-01T00:30:00-08:00| Kitchen | 22 | |2022-01-01T01:00:00-08:00| Kitchen | 23 | |2022-01-01T01:30:00-08:00| Kitchen | 22.85 | |2022-01-01T02:00:00-08:00| Kitchen | 22.7 | |2022-01-01T00:00:00-08:00|Living Room| 21.1 | |2022-01-01T00:30:00-08:00|Living Room| 21.25 | |2022-01-01T01:00:00-08:00|Living Room| 21.4 | |2022-01-01T01:30:00-08:00|Living Room| 21.6 | |2022-01-01T02:00:00-08:00|Living Room| 21.8 | The example below uses [`locf`](/influxdb3/enterprise/reference/sql/functions/misc/#locf)to fill null values by carrying the last observed value forward. ```sql SELECT date_bin_wallclock_gapfill(INTERVAL '30 minutes', tz(time, 'America/Los_Angeles')) as time, room, locf(avg(temp)) FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T10:00:00Z' GROUP BY 1, room ``` | time | room |locf(avg(home.temp))| |-------------------------|-----------|--------------------| |2022-01-01T00:00:00-08:00| Kitchen | 21 | |2022-01-01T00:30:00-08:00| Kitchen | 21 | |2022-01-01T01:00:00-08:00| Kitchen | 23 | |2022-01-01T01:30:00-08:00| Kitchen | 23 | |2022-01-01T02:00:00-08:00| Kitchen | 22.7 | |2022-01-01T00:00:00-08:00|Living Room| 21.1 | |2022-01-01T00:30:00-08:00|Living Room| 21.1 | |2022-01-01T01:00:00-08:00|Living Room| 21.4 | |2022-01-01T01:30:00-08:00|Living Room| 21.4 | |2022-01-01T02:00:00-08:00|Living Room| 21.8 | ## date\_trunc Truncates a timestamp value to a specified precision. ```sql date_trunc(precision, expression) ``` ### Arguments * **precision**: Time precision to truncate to. The following precisions are supported: * year * month * week * day * hour * minute * second * **expression**: Time expression to operate on. Can be a constant, column, or function. ##### Aliases * `datetrunc` [](#view-date_trunc-query-examples) View `date_trunc` query examples #### Use date\_trunc to return hourly averages ```sql SELECT avg(water_level) AS level, date_trunc('hour', time) AS hour FROM h2o_feet WHERE time >= timestamp '2019-09-10T00:00:00Z' AND time <= timestamp '2019-09-12T00:00:00Z' GROUP BY hour ORDER BY hour ``` | hour | level | |------------------------|------------------| |2019-09-10T00:00:00.000Z|3.7248000000000006| |2019-09-10T01:00:00.000Z|3.8561499999999995| |2019-09-10T02:00:00.000Z|4.5405999999999995| |2019-09-10T03:00:00.000Z|5.5548072072500005| |2019-09-10T04:00:00.000Z|6.433900000000001 | |2019-09-10T05:00:00.000Z|6.810949999999998 | #### Use date\_trunc to return weekly averages ```sql SELECT mean(water_level) as level, date_trunc('week',time) AS week FROM h2o_feet WHERE time >= timestamp '2019-08-01T00:00:00Z' AND time <= timestamp '2019-10-31T00:00:00Z' GROUP BY week ORDER BY week ``` | level | week | |------------------|------------------------| |4.3314415259020835|2019-08-12T00:00:00.000Z| |4.234838403584523 |2019-08-19T00:00:00.000Z| |4.4184818559633925|2019-08-26T00:00:00.000Z| |4.405153386766021 |2019-09-02T00:00:00.000Z| |4.725866897257734 |2019-09-09T00:00:00.000Z| |4.499938596774042 |2019-09-16T00:00:00.000Z| ## datetrunc *Alias of [date\_trunc](#date_trunc).* ## date\_format *Alias of [to\_char](#to_char).* ## date\_part Returns the specified part of the date as an integer. ```sql date_part(part, expression) ``` ### Arguments * **part**: Part of the date to return. The following date parts are supported: * year * month * week *(week of the year)* * day *(day of the month)* * hour * minute * second * millisecond * microsecond * nanosecond * dow *(day of the week)* * doy *(day of the year)* * **expression**: Time expression to operate on. Can be a constant, column, or function. ##### Aliases * `datepart` [](#view-date_part-query-examples) View `date_part` query examples ``` SELECT date_part('hour', time) AS hour, time, "level description", location FROM h2o_feet WHERE time >= timestamp '2019-08-17T02:54:00Z' AND time <= timestamp '2019-08-17T03:06:00Z' ORDER BY time ``` |hour| time | level description | location | |----|--------------------|--------------------|-------------| | 2 |2019-08-17T02:54:00Z|between 3 and 6 feet|coyote\_creek| | 2 |2019-08-17T02:54:00Z|between 3 and 6 feet|santa\_monica| | 3 |2019-08-17T03:00:00Z|between 3 and 6 feet|coyote\_creek| | 3 |2019-08-17T03:00:00Z|between 3 and 6 feet|santa\_monica| | 3 |2019-08-17T03:06:00Z|between 3 and 6 feet|coyote\_creek| | 3 |2019-08-17T03:06:00Z|between 3 and 6 feet|santa\_monica| ## datepart *Alias of [date\_part](#date_part).* ## extract Returns a sub-field from a time value as an integer. Similar to `date_part`, but with different arguments. ```sql extract(field FROM source) ``` ### Arguments * **field**: Part or field of the date to return. The following date fields are supported: * year * month * week *(week of the year)* * day *(day of the month)* * hour * minute * second * millisecond * microsecond * nanosecond * dow *(day of the week)* * doy *(day of the year)* * **source**: Source time expression to operate on. Can be a constant, column, or function. [](#view-extract-query-example) View `extract` query example ``` SELECT extract(day from time) AS day FROM h2o_feet LIMIT 1 ``` |day| |---| |25 | ## from\_unixtime Converts an integer (Unix timestamp in seconds) to a timestamp value. The underlying result is a timestamp (`Timestamp(TimeUnit::Second, None)`). If you output query results as JSON (default for the API), CSV, or pretty (default for the CLI), the timestamp is formatted as an ISO 8601 string (`YYYY-MM-DDTHH:MM:SS`, without a timezone indicator). When output to Parquet, the raw integer value (for example, `1641042000`) is preserved. ```sql from_unixtime(expression) ``` ### Arguments * **expression**: Integer expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. ##### Related functions [to\_unixtime](#to_unixtime) [](#view-from_unixtime-query-example) View `from_unixtime` query example ```sql SELECT from_unixtime(1672531200000000000) AS RFC3339 ``` | RFC3339 | |--------------------| |2023-01-01T00:00:00Z| ## make\_date Returns a date using the component parts (year, month, day). > [!Note] > `make_date` returns a `DATE32` Arrow type, which isn’t supported by InfluxDB. > To use with InfluxDB, [cast the return value to a timestamp or string](/influxdb3/enterprise/query-data/sql/cast-types/). ```sql make_date(year, month, day) ``` ### Arguments * **year**: Year to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators. * **month**: Month to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators. * **day**: Day to use when making the date. Can be a constant, column or function, and any combination of arithmetic operators [](#view-make_date-query-example) View `make_date` query example ```sql SELECT make_date(2024, 01, 01)::STRING AS date ``` | date | |----------| |2023-01-01| ## now Returns the current UTC timestamp. The `now()` return value is determined at query time and returns the same timestamp, no matter when in the query plan the function executes. ```sql now() ``` ##### Aliases * [current\_timestamp](#current_timestamp) [](#view-now-query-example) View `now` query example ```sql SELECT "water_level", "time" FROM h2o_feet WHERE time <= now() - interval '12 minutes' ``` ## today *Alias of [current\_date](#current_date).* ## to\_char Returns the string representation of a date, time, timestamp, or duration based on a [Rust Chrono format string](https://docs.rs/chrono/latest/chrono/format/strftime/index.html). > [!Note] > Unlike the PostgreSQL `TO_CHAR()` function, this function does not support > numeric formatting. ```sql to_char(expression, format) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function that results in a date, time, timestamp or duration. * **format**: [Rust Chrono format string](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)to use to convert the expression. ##### Aliases * [date\_format](#date_format) [](#view-to_char-query-example) View `to_char` query example ```sql SELECT to_char('2024-01-01T12:22:01Z'::TIMESTAMP, '%a %e-%b-%Y %H:%M:%S') AS datestring ``` | datestring | |-----------------------| |Mon 1-Jan-2024 12:22:01| ## to\_date Converts a value to a date (`YYYY-MM-DD`). Supports strings and numeric types as input. Strings are parsed as `YYYY-MM-DD` unless another format is specified. Numeric values are interpreted as days since the[Unix epoch](/influxdb3/enterprise/reference/glossary/#unix-epoch). > [!Note] > `to_date` returns a `DATE32` Arrow type, which isn’t supported by InfluxDB. > To use with InfluxDB, [cast the return value to a timestamp or string](/influxdb3/enterprise/query-data/sql/cast-types/). ```sql to_date(expression[, ..., format_n]) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **format\_n**: Optional [Rust strftime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)pattern to use to parse the *string* expression. Formats are attempted in the order that they appear. The function returns the timestamp from the first format to parse successfully. If no formats parse successfully, the function returns an error. [](#view-to_date-query-example) View `to_date` query example ```sql SELECT to_date('1-Jan-2024', '%e-%b-%Y')::STRING AS date ``` | date | |----------| |2024-01-01| ## to\_local\_time Converts a timestamp with a timezone to a timestamp without a timezone (no offset or timezone information). This function accounts for time shifts like daylight saving time (DST). > [!Note] > Use `to_local_time()` with [`date_bin()`](#date_bin) and[`date_bin_gapfill`](#date_bin_gapfill) to generate window boundaries based the > local time zone rather than UTC. ```sql to_local_time(expression) ``` ### Arguments * **expression**: Time expression to operate on. Can be a constant, column, or function. [](#view-to_local_time-query-example) View `to_local_time` query example ```sql SELECT to_local_time('2024-01-01 00:00:00'::TIMESTAMP) AS "local time"; ``` | local time | |--------------------| |2024-01-01T00:00:00Z| [](#view-to_local_time-query-example-with-a-time-zone-offset) View `to_local_time` query example with a time zone offset ```sql SELECT to_local_time((arrow_cast('2024-01-01 00:00:00', 'Timestamp(Nanosecond, Some("UTC"))')) AT TIME ZONE 'America/Los_Angeles') AS "local time" ``` | local time | |--------------------| |2023-12-31T16:00:00Z| [](#view-to_local_time-query-example-with-date_bin) View `to_local_time` query example with `date_bin` ```sql SELECT date_bin(interval '1 day', time, to_local_time(0::TIMESTAMP)) AT TIME ZONE 'America/Los_Angeles' AS time, avg(f1), avg(f2) FROM (VALUES (arrow_cast('2024-01-01 12:00:00', 'Timestamp(Nanosecond, Some("UTC"))'), 1.23, 4.56), (arrow_cast('2024-01-01 13:00:00', 'Timestamp(Nanosecond, Some("UTC"))'), 2.46, 8.1), (arrow_cast('2024-01-01 14:00:00', 'Timestamp(Nanosecond, Some("UTC"))'), 4.81, 16.2) ) AS data(time, f1, f2) GROUP BY 1 ``` | time | avg(data.f1) |avg(data.f2)| |-------------------------|------------------|------------| |2023-12-31T16:00:00-08:00|2.8333333333333335| 9.62 | ## to\_timestamp Converts a value to RFC3339 timestamp format (`YYYY-MM-DDT00:00:00Z`). Supports timestamp, integer, and unsigned integer types as input. Integers and unsigned integers are parsed as[Unix nanosecond timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp)and return the corresponding RFC3339 timestamp. ```sql to_timestamp(expression) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. [](#view-to_timestamp-query-example) View `to_timestamp` query example ```sql SELECT to_timestamp(1704067200000000000) ``` |to\_timestamp(Int64(1704067200000000000))| |-----------------------------------------| | 2024-01-01T00:00:00Z | ## to\_timestamp\_micros Converts a value to RFC3339 microsecond timestamp format (`YYYY-MM-DDT00:00:00.000000Z`). Supports timestamp, integer, and unsigned integer types as input. Integers and unsigned integers are parsed as[Unix microsecond timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp)and return the corresponding RFC3339 timestamp. ```sql to_timestamp_micros(expression[, ..., format_n]) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **format\_n**: Optional [Rust strftime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)pattern to use to parse the *string* expression. Formats are attempted in the order that they appear. The function returns the timestamp from the first format to parse successfully. If no formats parse successfully, the function returns an error. [](#view-to_timestamp_micros-query-example) View `to_timestamp_micros` query example ```sql SELECT to_timestamp_micros(1704067200000001) ``` |to\_timestamp\_micros(Int64(1704067200000001))| |----------------------------------------------| | 2024-01-01T00:00:00.000001Z | [](#view-to_timestamp_micros-example-with-string-format-parsing) View `to_timestamp_micros` example with string format parsing ```sql SELECT to_timestamp_micros('01:01:59.123456789 01-01-2024', '%c', '%+', '%H:%M:%S%.f %m-%d-%Y') AS microsecond ``` | microsecond | |---------------------------| |2024-01-01T01:01:59.123456Z| ## to\_timestamp\_millis Converts a value to RFC3339 millisecond timestamp format (`YYYY-MM-DDT00:00:00.000Z`). Supports timestamp, integer, and unsigned integer types as input. Integers and unsigned integers are parsed as[Unix millisecond timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp)and return the corresponding RFC3339 timestamp. ```sql to_timestamp_millis(expression[, ..., format_n]) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **format\_n**: Optional [Rust strftime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)pattern to use to parse the *string* expression. Formats are attempted in the order that they appear. The function returns the timestamp from the first format to parse successfully. If no formats parse successfully, the function returns an error. [](#view-to_timestamp_millis-query-example) View `to_timestamp_millis` query example ```sql SELECT to_timestamp_millis(1704067200001) AS time ``` Results |to\_timestamp\_millis(Int64(1704067200001))| |-------------------------------------------| | 2024-01-01T00:00:00.001Z | [](#view-to_timestamp_millis-example-with-string-format-parsing) View `to_timestamp_millis` example with string format parsing ```sql SELECT to_timestamp_millis('01:01:59.123456789 01-01-2024', '%c', '%+', '%H:%M:%S%.f %m-%d-%Y') AS millisecond ``` | millisecond | |------------------------| |2024-01-01T01:01:59.123Z| ## to\_timestamp\_nanos Converts a value to RFC3339 nanosecond timestamp format (`YYYY-MM-DDT00:00:00.000000000Z`). Supports timestamp, integer, and unsigned integer types as input. Integers and unsigned integers are parsed as[Unix nanosecond timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp)and return the corresponding RFC3339 timestamp. ```sql to_timestamp_nanos(expression[, ..., format_n]) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **format\_n**: Optional [Rust strftime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)pattern to use to parse the *string* expression. Formats are attempted in the order that they appear. The function returns the timestamp from the first format to parse successfully. If no formats parse successfully, the function returns an error. [](#view-to_timestamp_nanos-query-example) View `to_timestamp_nanos` query example ```sql SELECT to_timestamp_nanos(1704067200000000001) ``` |to\_timestamp\_nanos(Int64(1704067200000000001))| |------------------------------------------------| | 2024-01-01T00:00:00.000000001Z | [](#view-to_timestamp_nanos-example-with-string-format-parsing) View `to_timestamp_nanos` example with string format parsing ```sql SELECT to_timestamp_nanos('01:01:59.123456789 01-01-2024', '%c', '%+', '%H:%M:%S%.f %m-%d-%Y') AS nanosecond ``` | nanosecond | |------------------------------| |2024-01-01T01:01:59.123456789Z| ## to\_timestamp\_seconds Converts a value to RFC3339 second timestamp format (`YYYY-MM-DDT00:00:00Z`). Supports timestamp, integer, and unsigned integer types as input. Integers and unsigned integers are parsed as[Unix second timestamps](/influxdb3/enterprise/reference/glossary/#unix-timestamp)and return the corresponding RFC3339 timestamp. ```sql to_timestamp_seconds(expression[, ..., format_n]) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **format\_n**: Optional [Rust strftime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)pattern to use to parse the *string* expression. Formats are attempted in the order that they appear. The function returns the timestamp from the first format to parse successfully. If no formats parse successfully, the function returns an error. [](#view-to_timestamp_seconds-query-example) View `to_timestamp_seconds` query example ```sql SELECT to_timestamp_seconds(1704067201) ``` |to\_timestamp\_seconds(Int64(1704067201))| |-----------------------------------------| | 2024-01-01T00:00:01Z | [](#view-to_timestamp_seconds-example-with-string-format-parsing) View `to_timestamp_seconds` example with string format parsing ```sql SELECT to_timestamp_seconds('01:01:59.123456789 01-01-2024', '%c', '%+', '%H:%M:%S%.f %m-%d-%Y') AS second ``` | second | |--------------------| |2024-01-01T01:01:59Z| ## to\_unixtime Converts a value to seconds since the [Unix epoch](/influxdb3/enterprise/reference/glossary/#unix-epoch). Supports strings, timestamps, and floats as input. Strings are parsed as [RFC3339Nano timestamps](/influxdb3/enterprise/reference/glossary/#rfc3339nano-timestamp) if no[Rust Chrono format strings](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)are provided. ```sql to_unixtime(expression[, ..., format_n]) ``` ### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **format\_n**: Optional [Rust strftime](https://docs.rs/chrono/latest/chrono/format/strftime/index.html)pattern to use to parse the *string* expression. Formats are attempted in the order that they appear. The function returns the timestamp from the first format to parse successfully. If no formats parse successfully, the function returns an error. ##### Related functions [from\_unixtime](#from_unixtime) [](#view-to_unixtime-query-example) View `to_unixtime` query example ```sql SELECT to_unixtime('2024-01-01T01:01:59.123456789Z') AS unixtime ``` | unixtime | |----------| |1704070919| [](#view-to_unixtime-example-with-string-format-parsing) View `to_unixtime` example with string format parsing ```sql SELECT to_unixtime('01:01:59.123456789 01-01-2024', '%c', '%+', '%H:%M:%S%.f %m-%d-%Y') AS unixtime ``` | unixtime | |----------| |1704070919| ## tz Converts a timestamp to a provided timezone. If the second argument is not provided, it defaults to UTC. ```sql tz(time_expression[, timezone]) ``` ### Arguments * **time\_expression**: time to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. * **timezone**: [Timezone string](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones)to cast the value into. Default is `'UTC'`. The function returns the timestamp cast to the specified timezone. If an incorrect timezone string is passed or the wrong datatype is provided, the function returns an error. [](#view-tz-query-example) View `tz` query example ```sql SELECT tz('2024-01-01T01:00:00Z', 'America/New_York') AS time_tz ``` | time\_tz | |-------------------------| |2024-10-01T02:00:00-04:00| [](#view-tz-query-example-from-getting-started-data) View `tz` query example from Getting Started data ```sql SELECT tz(time, 'Australia/Sydney') AS time_tz, time FROM home ORDER BY time LIMIT 3; ``` | time\_tz | time | |-----------------------------------|------------------------------| |1970-01-01T10:00:01.728979200+10:00|1970-01-01T00:00:01.728979200Z| |1970-01-01T10:00:01.728979200+10:00|1970-01-01T00:00:01.728979200Z| |1970-01-01T10:00:01.728982800+10:00|1970-01-01T00:00:01.728982800Z| ##### Differences between tz and AT TIME ZONE `tz` and [`AT TIME ZONE`](/influxdb3/enterprise/reference/sql/operators/other/#at-time-zone)differ when the input timestamp **does not** have a timezone. * When using an input timestamp that does not have a timezone (the default behavior in InfluxDB) with the`AT TIME ZONE` operator, the operator returns the same timestamp, but with a timezone offset (also known as the “wall clock” time)–for example: ``` '2024-01-01 00:00:00'::TIMESTAMP AT TIME ZONE 'America/Los_Angeles' -- Returns 2024-01-01T00:00:00-08:00 ``` * When using an input timestamp with a timezone, both the `tz()` function and the `AT TIME ZONE`operator return the timestamp converted to the time in the specified timezone–for example: ``` '2024-01-01T00:00:00-00:00' AT TIME ZONE 'America/Los_Angeles' tz('2024-01-01T00:00:00-00:00', 'America/Los_Angeles') -- Both return 2023-12-31T16:00:00-08:00 ``` * `tz()` always converts the input timestamp to the specified time zone. If the input timestamp does not have a timezone, the function assumes it is a UTC timestamp–for example: ``` tz('2024-01-01 00:00:00'::TIMESTAMP, 'America/Los_Angeles') -- Returns 2023-12-31T16:00:00-08:00 ``` ``` tz('2024-01-01T00:00:00+1:00', 'America/Los_Angeles') -- Returns 2023-12-31T15:00:00-08:00 ``` [](#view-tz-and-timestamp-comparison) View `tz` and `::timestamp` comparison ```sql SELECT '2024-04-01T00:00:20Z'::timestamp AT TIME ZONE 'Europe/Brussels' as time_timestamp, tz('2024-04-01T00:00:20', 'Europe/Brussels') as time_tz; ``` | time\_timestamp | time\_tz | |-------------------------|-------------------------| |2024-04-01T00:00:20+02:00|2024-04-01T02:00:20+02:00| | time | temp | current_date | | --- | --- | --- | | time | temp | current_date | | 2022-01-01T08:00:00Z | 21 | 2026-11-17T00:00:00Z | | 2022-01-01T09:00:00Z | 23 | 2026-11-17T00:00:00Z | | 2022-01-01T10:00:00Z | 22.7 | 2026-11-17T00:00:00Z | | time | temp | current_time | | --- | --- | --- | | time | temp | current_time | | 2022-01-01T08:00:00Z | 21 | 2021-01-01T00:00:00Z | | 2022-01-01T09:00:00Z | 23 | 2021-01-01T00:00:00Z | | 2022-01-01T10:00:00Z | 22.7 | 2021-01-01T00:00:00Z | | time | water_level_avg | | --- | --- | | time | water_level_avg | | 2019-09-17T00:00:00.000Z | 4.370175687443861 | | 2019-09-16T00:00:00.000Z | 4.6034785848437485 | | 2019-09-15T00:00:00.000Z | 4.680651501506248 | | 2019-09-14T00:00:00.000Z | 4.857683652395836 | | 2019-09-13T00:00:00.000Z | 4.911051520291668 | | 2019-09-12T00:00:00.000Z | 4.763990784533338 | | 2019-09-11T00:00:00.000Z | 4.6582452515041695 | | 2019-09-10T00:00:00.000Z | 4.608425018785421 | | _time | room | temp | | --- | --- | --- | | _time | room | temp | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | | | 2022-01-01T10:00:00Z | Living Room | 21.8 | | _time | room | AVG(home.temp) | | --- | --- | --- | | _time | room | AVG(home.temp) | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | 22 | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | 22.85 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | 21.25 | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | 21.6 | | 2022-01-01T10:00:00Z | Living Room | 21.8 | | _time | room | AVG(home.temp) | | --- | --- | --- | | _time | room | AVG(home.temp) | | 2022-01-01T08:00:00Z | Kitchen | 21 | | 2022-01-01T08:30:00Z | Kitchen | 21 | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T09:30:00Z | Kitchen | 23 | | 2022-01-01T10:00:00Z | Kitchen | 22.7 | | 2022-01-01T08:00:00Z | Living Room | 21.1 | | 2022-01-01T08:30:00Z | Living Room | 21.1 | | 2022-01-01T09:00:00Z | Living Room | 21.4 | | 2022-01-01T09:30:00Z | Living Room | 21.4 | | 2022-01-01T10:00:00Z | Living Room | 21.8 | | original_time | utc_bin_time | | --- | --- | | original_time | utc_bin_time | | 2020-10-25T02:29:00+01:00 | 2020-10-24T23:30:00Z | | 2020-10-25T02:30:00+01:00 | 2020-10-25T01:30:00Z | | time | room | avg_temp | | --- | --- | --- | | time | room | avg_temp | | 2022-01-01T00:00:00-08:00 | Kitchen | 22.61666666666667 | | 2022-01-01T12:00:00-08:00 | Kitchen | 22.7 | | 2022-01-01T00:00:00-08:00 | Living Room | 22.166666666666668 | | 2022-01-01T12:00:00-08:00 | Living Room | 22.2 | | time | room | temp | | --- | --- | --- | | time | room | temp | | 2022-01-01T00:00:00-08:00 | Kitchen | 21 | | 2022-01-01T00:30:00-08:00 | Kitchen | | | 2022-01-01T01:00:00-08:00 | Kitchen | 23 | | 2022-01-01T01:30:00-08:00 | Kitchen | | | 2022-01-01T02:00:00-08:00 | Kitchen | 22.7 | | 2022-01-01T00:00:00-08:00 | Living Room | 21.1 | | 2022-01-01T00:30:00-08:00 | Living Room | | | 2022-01-01T01:00:00-08:00 | Living Room | 21.4 | | 2022-01-01T01:30:00-08:00 | Living Room | | | 2022-01-01T02:00:00-08:00 | Living Room | 21.8 | | time | room | interpolate(avg(home.temp)) | | --- | --- | --- | | time | room | interpolate(avg(home.temp)) | | 2022-01-01T00:00:00-08:00 | Kitchen | 21 | | 2022-01-01T00:30:00-08:00 | Kitchen | 22 | | 2022-01-01T01:00:00-08:00 | Kitchen | 23 | | 2022-01-01T01:30:00-08:00 | Kitchen | 22.85 | | 2022-01-01T02:00:00-08:00 | Kitchen | 22.7 | | 2022-01-01T00:00:00-08:00 | Living Room | 21.1 | | 2022-01-01T00:30:00-08:00 | Living Room | 21.25 | | 2022-01-01T01:00:00-08:00 | Living Room | 21.4 | | 2022-01-01T01:30:00-08:00 | Living Room | 21.6 | | 2022-01-01T02:00:00-08:00 | Living Room | 21.8 | | time | room | locf(avg(home.temp)) | | --- | --- | --- | | time | room | locf(avg(home.temp)) | | 2022-01-01T00:00:00-08:00 | Kitchen | 21 | | 2022-01-01T00:30:00-08:00 | Kitchen | 21 | | 2022-01-01T01:00:00-08:00 | Kitchen | 23 | | 2022-01-01T01:30:00-08:00 | Kitchen | 23 | | 2022-01-01T02:00:00-08:00 | Kitchen | 22.7 | | 2022-01-01T00:00:00-08:00 | Living Room | 21.1 | | 2022-01-01T00:30:00-08:00 | Living Room | 21.1 | | 2022-01-01T01:00:00-08:00 | Living Room | 21.4 | | 2022-01-01T01:30:00-08:00 | Living Room | 21.4 | | 2022-01-01T02:00:00-08:00 | Living Room | 21.8 | | hour | level | | --- | --- | | hour | level | | 2019-09-10T00:00:00.000Z | 3.7248000000000006 | | 2019-09-10T01:00:00.000Z | 3.8561499999999995 | | 2019-09-10T02:00:00.000Z | 4.5405999999999995 | | 2019-09-10T03:00:00.000Z | 5.5548072072500005 | | 2019-09-10T04:00:00.000Z | 6.433900000000001 | | 2019-09-10T05:00:00.000Z | 6.810949999999998 | | level | week | | --- | --- | | level | week | | 4.3314415259020835 | 2019-08-12T00:00:00.000Z | | 4.234838403584523 | 2019-08-19T00:00:00.000Z | | 4.4184818559633925 | 2019-08-26T00:00:00.000Z | | 4.405153386766021 | 2019-09-02T00:00:00.000Z | | 4.725866897257734 | 2019-09-09T00:00:00.000Z | | 4.499938596774042 | 2019-09-16T00:00:00.000Z | | hour | time | level description | location | | --- | --- | --- | --- | | hour | time | level description | location | | 2 | 2019-08-17T02:54:00Z | between 3 and 6 feet | coyote_creek | | 2 | 2019-08-17T02:54:00Z | between 3 and 6 feet | santa_monica | | 3 | 2019-08-17T03:00:00Z | between 3 and 6 feet | coyote_creek | | 3 | 2019-08-17T03:00:00Z | between 3 and 6 feet | santa_monica | | 3 | 2019-08-17T03:06:00Z | between 3 and 6 feet | coyote_creek | | 3 | 2019-08-17T03:06:00Z | between 3 and 6 feet | santa_monica | | day | | --- | | day | | 25 | | RFC3339 | | --- | | RFC3339 | | 2023-01-01T00:00:00Z | | date | | --- | | date | | 2023-01-01 | | datestring | | --- | | datestring | | Mon 1-Jan-2024 12:22:01 | | date | | --- | | date | | 2024-01-01 | | local time | | --- | | local time | | 2024-01-01T00:00:00Z | | local time | | --- | | local time | | 2023-12-31T16:00:00Z | | time | avg(data.f1) | avg(data.f2) | | --- | --- | --- | | time | avg(data.f1) | avg(data.f2) | | 2023-12-31T16:00:00-08:00 | 2.8333333333333335 | 9.62 | | to_timestamp(Int64(1704067200000000000)) | | --- | | to_timestamp(Int64(1704067200000000000)) | | 2024-01-01T00:00:00Z | | to_timestamp_micros(Int64(1704067200000001)) | | --- | | to_timestamp_micros(Int64(1704067200000001)) | | 2024-01-01T00:00:00.000001Z | | microsecond | | --- | | microsecond | | 2024-01-01T01:01:59.123456Z | | to_timestamp_millis(Int64(1704067200001)) | | --- | | to_timestamp_millis(Int64(1704067200001)) | | 2024-01-01T00:00:00.001Z | | millisecond | | --- | | millisecond | | 2024-01-01T01:01:59.123Z | | to_timestamp_nanos(Int64(1704067200000000001)) | | --- | | to_timestamp_nanos(Int64(1704067200000000001)) | | 2024-01-01T00:00:00.000000001Z | | nanosecond | | --- | | nanosecond | | 2024-01-01T01:01:59.123456789Z | | to_timestamp_seconds(Int64(1704067201)) | | --- | | to_timestamp_seconds(Int64(1704067201)) | | 2024-01-01T00:00:01Z | | second | | --- | | second | | 2024-01-01T01:01:59Z | | unixtime | | --- | | unixtime | | 1704070919 | | unixtime | | --- | | unixtime | | 1704070919 | | time_tz | | --- | | time_tz | | 2024-10-01T02:00:00-04:00 | | time_tz | time | | --- | --- | | time_tz | time | | 1970-01-01T10:00:01.728979200+10:00 | 1970-01-01T00:00:01.728979200Z | | 1970-01-01T10:00:01.728979200+10:00 | 1970-01-01T00:00:01.728979200Z | | 1970-01-01T10:00:01.728982800+10:00 | 1970-01-01T00:00:01.728982800Z | | time_timestamp | time_tz | | --- | --- | | time_timestamp | time_tz | | 2024-04-01T00:00:20+02:00 | 2024-04-01T02:00:20+02:00 | --- # SQL window functions Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/functions/window/ Window functions let you calculate running totals, moving averages, or other aggregate-like results without collapsing rows into groups. They perform their calculations over a “window” of rows, which you can partition and order in various ways, and return a calculated value for each row in the set. Unlike non-window [aggregate functions](/influxdb3/enterprise/reference/sql/functions/aggregate/) that combine each group into a single row, window functions preserve each row’s identity and calculate an additional value for every row in the partition. For example, the following query uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data)and returns each temperature reading with the average temperature per room over the queried time range: ```sql SELECT time, room, temp, avg(temp) OVER (PARTITION BY room) AS avg_room_temp FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T09:00:00Z' ORDER BY room, time ``` | time | room |temp|avg\_room\_temp| |-------------------|-----------|----|---------------| |2022-01-01T08:00:00| Kitchen |21.0| 22.0 | |2022-01-01T09:00:00| Kitchen |23.0| 22.0 | |2022-01-01T08:00:00|Living Room|21.1| 21.25 | |2022-01-01T09:00:00|Living Room|21.4| 21.25 | ## Window frames As window functions operate on a row, there is a set of rows in the row’s partition that the window function uses to perform the operation. This set of rows is called the *window frame*. Window frame boundaries can be defined using`RANGE`, `ROW`, or `GROUPS` frame units, each relative to the current row–for example: [RANGE](#range-2)[ROWS](#rows-2)[GROUPS](#groups-2) ```sql SELECT time, temp, avg(temp) OVER ( ORDER BY time RANGE INTERVAL '3 hours' PRECEDING ) AS 3h_moving_avg FROM home WHERE room = 'Kitchen' ``` ```sql SELECT time, temp, avg(temp) OVER ( ROWS 3 PRECEDING ) AS moving_avg FROM home WHERE room = 'Kitchen' ``` ```sql SELECT time, room, temp, avg(temp) OVER ( ORDER BY room GROUPS 1 PRECEDING ) AS moving_avg FROM home ``` *For more information about how window frames work, see the [frame clause](#frame-clause).* If you don’t specify window frames, window functions use all rows in the current partition to perform their operation. ```sql function([expr]) OVER( [PARTITION BY expr[, …]] [ORDER BY expr [ ASC | DESC ][, …]] [ frame_clause ] ) ``` ### OVER clause Window functions use an `OVER` clause that directly follows the window function’s name and arguments. The `OVER` clause syntactically distinguishes a window function from a non-window or aggregate function and defines how to group and order rows for the window operation. ### PARTITION BY clause The `PARTITION BY` clause in the `OVER` clause divides the rows into groups, or partitions, that share the same values of the `PARTITION BY` expressions. The window function operates on all the rows in the same partition as the current row. ### ORDER BY clause The `ORDER BY` clause inside of the `OVER` clause controls the order that the window function processes rows in each partition. When a window clause contains an `ORDER BY` clause, the window frame boundaries may be explicit or implicit, limiting a window frame size in both directions relative to the current row. > [!Note] > The `ORDER BY` clause in an `OVER` clause determines the processing order for > rows in each partition and is separate from the `ORDER BY`clause of the query. ### Frame clause The *frame clause* defines window frame boundaries and can be one of the following: ```sql { RANGE | ROWS | GROUPS } frame_start { RANGE | ROWS | GROUPS } BETWEEN frame_start AND frame_end ``` #### Frame units When defining window frames, you can use one of the following frame units: ##### RANGE Defines frame boundaries using rows with values for columns specified in the [`ORDER BY` clause](#order-by-clause) within a value range relative to the current row value. > [!Important] > When using `RANGE` frame units, you must include an `ORDER BY` clause with*exactly one column*. The offset is the difference between the current row value and surrounding row values. `RANGE` supports the following offset types: * Numeric *(non-negative)* * Numeric string *(non-negative)* * Interval [](#see-how-range-frame-units-work-with-numeric-offsets) See how `RANGE` frame units work with numeric offsets To use a numeric offset with the `RANGE` frame unit, you must sort partitions by a numeric-type column. ```sql ... OVER ( ORDER BY wind_direction RANGE BETWEEN 45 PRECEDING AND 45 FOLLOWING ) ``` The window frame includes rows with sort column values between 45 below and 45 above the current row’s value: | time |city|wind\_direction| |-------------------|----|---------------| |2025-02-17T13:00:00|Rome| 33 | |2025-02-17T08:00:00|Rome| 34 | |2025-02-17T23:00:00|Rome| 49 | |2025-02-17T17:00:00|Rome| 86 | |2025-02-17T11:00:00|Rome| 93 | |2025-02-17T12:00:00|Rome| 115 | |2025-02-17T10:00:00|Rome| 156 | [](#see-how-range-frame-units-work-with-interval-offsets) See how `RANGE` frame units work with interval offsets To use an interval offset with the `RANGE` frame unit, you must sort partitions by `time` or a timestamp-type column. ```sql ... OVER ( ORDER BY time RANGE BETWEEN INTERVAL '3 hours' PRECEDING AND INTERVAL '1 hour' FOLLOWING ) ``` The window frame includes rows with timestamps between three hours before and one hour after the current row’s timestamp: | time | room |temp| |-------------------|-------|----| |2022-01-01T08:00:00|Kitchen|21.0| |2022-01-01T09:00:00|Kitchen|23.0| |2022-01-01T10:00:00|Kitchen|22.7| |2022-01-01T11:00:00|Kitchen|22.4| |2022-01-01T12:00:00|Kitchen|22.5| |2022-01-01T13:00:00|Kitchen|22.8| |2022-01-01T14:00:00|Kitchen|22.8| |2022-01-01T15:00:00|Kitchen|22.7| ##### ROWS Defines window frame boundaries using row positions relative to the current row. The offset is the difference in row position from the current row.`ROWS` supports the following offset types: * Numeric *(non-negative)* * Numeric string *(non-negative)* [](#see-how-rows-frame-units-work) See how `ROWS` frame units work When using the `ROWS` frame unit, row positions relative to the current row determine frame boundaries–for example: ```sql ... OVER ( ROWS BETWEEN 2 PRECEDING AND 1 FOLLOWING ) ``` The window frame includes the two rows before and the one row after the current row: | time |city|wind\_direction| |-------------------|----|---------------| |2025-02-17T08:00:00|Rome| 34 | |2025-02-17T10:00:00|Rome| 156 | |2025-02-17T11:00:00|Rome| 93 | |2025-02-17T12:00:00|Rome| 115 | |2025-02-17T13:00:00|Rome| 33 | |2025-02-17T17:00:00|Rome| 86 | |2025-02-17T23:00:00|Rome| 49 | ##### GROUPS Defines window frame boundaries using row groups. Rows with the same values for the columns in the [`ORDER BY` clause](#order-by-clause)comprise a row group. > [!Important] > When using `GROUPS` frame units, include an `ORDER BY` clause. The offset is the difference in row group position relative to the current row group.`GROUPS` supports the following offset types: * Numeric *(non-negative)* * Numeric string *(non-negative)* [](#see-how-groups-frame-units-work) See how `GROUPS` frame units work When using the `GROUPS` frame unit, unique combinations column values specified in the `ORDER BY` clause determine each row group. For example, if you sort partitions by `country` and `city`: ```sql ... OVER ( ORDER BY country, city GROUPS ... ) ``` The query defines row groups in the following way: | time |country| city |wind\_direction| |-------------------|-------|----------|---------------| |2025-02-17T00:00:00|France |Strasbourg| 181 | |2025-02-17T01:00:00|France |Strasbourg| 228 | |2025-02-17T02:00:00|France |Strasbourg| 289 | |2025-02-17T00:00:00|France | Toulouse | 24 | |2025-02-17T01:00:00|France | Toulouse | 210 | |2025-02-17T02:00:00|France | Toulouse | 206 | |2025-02-17T00:00:00| Italy | Bari | 2 | |2025-02-17T01:00:00| Italy | Bari | 57 | |2025-02-17T00:00:00| Italy | Bologna | 351 | |2025-02-17T01:00:00| Italy | Bologna | 232 | |2025-02-17T02:00:00| Italy | Bologna | 29 | You can then use group offsets to determine frame boundaries: ```sql ... OVER ( ORDER BY country, city GROUPS 2 PRECEDING ) ``` The window function uses all rows in the current row group and the two preceding row groups to perform the operation: | time |country| city |wind\_direction| |-------------------|-------|----------|---------------| |2025-02-17T00:00:00|France |Strasbourg| 181 | |2025-02-17T01:00:00|France |Strasbourg| 228 | |2025-02-17T02:00:00|France |Strasbourg| 289 | |2025-02-17T00:00:00|France | Toulouse | 24 | |2025-02-17T01:00:00|France | Toulouse | 210 | |2025-02-17T02:00:00|France | Toulouse | 206 | |2025-02-17T00:00:00| Italy | Bari | 2 | |2025-02-17T01:00:00| Italy | Bari | 57 | |2025-02-17T00:00:00| Italy | Bologna | 351 | |2025-02-17T01:00:00| Italy | Bologna | 232 | |2025-02-17T02:00:00| Italy | Bologna | 29 | #### Frame boundaries Frame boundaries (**frame\_start** and **frame\_end**) define the boundaries of each frame that the window function operates on. ##### UNBOUNDED PRECEDING Starts at the first row of the partition and ends at the current row. ```sql UNBOUNDED PRECEDING ``` ##### offset PRECEDING Starts at `offset` [frame units](#frame-units) before the current row and ends at the current row. For example, `3 PRECEDING` includes 3 rows before the current row. ```sql PRECEDING ``` ##### CURRENT ROW Both starts and ends at the current row when used as a boundary. ```sql CURRENT ROW ``` ##### offset FOLLOWING Starts at the current row and ends at `offset` [frame units](#frame-units) after the current row. For example, `3 FOLLOWING` includes 3 rows after the current row. ```sql FOLLOWING ``` ##### UNBOUNDED FOLLOWING Use the current row to the end of the current partition the frame boundary. ```sql UNBOUNDED FOLLOWING ``` ### WINDOW clause Use the `WINDOW` clause to define a reusable alias for a window specification. This is useful when multiple window functions in your query share the same window definition. Instead of repeating the same OVER clause for each function, define the window once and reference it by alias–for example: ```sql SELECT sum(net_gain) OVER w, avg(net_net) OVER w FROM finance WINDOW w AS ( PARTITION BY ticker ORDER BY time DESC); ``` ## Aggregate functions All [aggregate functions](/influxdb3/cloud-dedicated/reference/sql/functions/aggregate/)can be used as window functions. ## Ranking Functions ### cume\_dist Returns the cumulative distribution of a value within a group of values. The returned value is greater than 0 and less than or equal to 1 and represents the relative rank of the value in the set of values. The [`ORDER BY` clause](#order-by-clause) in the `OVER` clause is used to correctly calculate the cumulative distribution of the current row value. ```sql cume_dist() ``` > [!Important] > When using `cume_dist`, include an [`ORDER BY` clause](#order-by-clause) in the `OVER` clause. [](#view-cume_dist-query-example) View `cume_dist` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, room, temp, cume_dist() OVER ( PARTITION BY room ORDER BY temp ) AS cume_dist FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T12:00:00Z' ``` | time | room |temp|cume\_dist| |-------------------|-----------|----|----------| |2022-01-01T08:00:00|Living Room|21.1| 0.25 | |2022-01-01T09:00:00|Living Room|21.4| 0.5 | |2022-01-01T10:00:00|Living Room|21.8| 0.75 | |2022-01-01T11:00:00|Living Room|22.2| 1.0 | |2022-01-01T08:00:00| Kitchen |21.0| 0.25 | |2022-01-01T11:00:00| Kitchen |22.4| 0.5 | |2022-01-01T10:00:00| Kitchen |22.7| 0.75 | |2022-01-01T09:00:00| Kitchen |23.0| 1.0 | ### dense\_rank Returns the rank of the current row in its partition. Ranking is consecutive; assigns duplicate values the same rank number and the rank sequence continues with the next distinct value (unlike [`rank()`](#rank)). The [`ORDER BY` clause](#order-by-clause) in the `OVER` clause determines ranking order. ```sql dense_rank() ``` [](#view-dense_rank-query-example) View `dense_rank` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, room, temp, dense_rank() OVER ( PARTITION BY room ORDER BY temp ) AS dense_rank FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T12:00:00Z' ``` | time | room |temp|dense\_rank| |-------------------|-----------|----|-----------| |2022-01-01T08:00:00| Kitchen |21.0| 1 | |2022-01-01T11:00:00| Kitchen |22.4| 2 | |2022-01-01T10:00:00| Kitchen |22.7| 3 | |2022-01-01T09:00:00| Kitchen |23.0| 4 | |2022-01-01T08:00:00|Living Room|21.1| 1 | |2022-01-01T09:00:00|Living Room|21.4| 2 | |2022-01-01T10:00:00|Living Room|21.8| 3 | |2022-01-01T11:00:00|Living Room|22.2| 4 | [](#compare-dense_rank-rank-and-row_number-functions) Compare `dense_rank`, `rank`, and `row_number` functions Consider a table with duplicate ID values. The following query shows how each ranking function handles duplicate values: ```sql SELECT id, rank() OVER(ORDER BY id), dense_rank() OVER(ORDER BY id), row_number() OVER(ORDER BY id) FROM my_table; ``` |ID |rank|dense\_rank|row\_number| |---|----|-----------|-----------| | 1 | 1 | 1 | 1 | | 1 | 1 | 1 | 2 | | 1 | 1 | 1 | 3 | | 2 | 4 | 2 | 4 | Key differences: * [`rank()`](#rank) assigns the same rank to equal values but skips ranks for subsequent values * [`dense_rank()`](#dense_rank) assigns the same rank to equal values and uses consecutive ranks * [`row_number()`](#row_number) assigns unique sequential numbers regardless of value (non-deterministic) ### ntile Distributes the rows in an ordered partition into the specified number of groups. Each group is numbered, starting at one. For each row, `ntile` returns the group number to which the row belongs. Group numbers range from 1 to the `expression` value, dividing the partition as equally as possible. The [`ORDER BY` clause](#order-by-clause) in the `OVER` clause determines ranking order. ```sql ntile(expression) ``` #### Arguments * **expression**: An integer. The number of groups to split the partition into. [](#view-ntile-query-example) View `ntile` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, temp, ntile(4) OVER ( ORDER BY time ) AS ntile FROM home WHERE room = 'Kitchen' AND time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T15:00:00Z' ``` | time |temp|ntile| |-------------------|----|-----| |2022-01-01T08:00:00|21.0| 1 | |2022-01-01T09:00:00|23.0| 1 | |2022-01-01T10:00:00|22.7| 2 | |2022-01-01T11:00:00|22.4| 2 | |2022-01-01T12:00:00|22.5| 3 | |2022-01-01T13:00:00|22.8| 3 | |2022-01-01T14:00:00|22.8| 4 | ### percent\_rank Returns the percentage rank of the current row within its partition. The returned value is between `0` and `1`, computed as: ``` (rank - 1) / (total_rows - 1) ``` The [`ORDER BY` clause](#order-by-clause) in the `OVER` clause determines the ranking order. ```sql percent_rank() ``` [](#view-percent_rank-query-example) View `percent_rank` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, room, temp, percent_rank() OVER ( PARTITION BY room ORDER BY temp ) AS percent_rank FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ``` | time | room |temp|percent\_rank| |-------------------|-----------|----|-------------| |2022-01-01T08:00:00| Kitchen |21.0| 0.0 | |2022-01-01T10:00:00| Kitchen |22.7| 0.5 | |2022-01-01T09:00:00| Kitchen |23.0| 1.0 | |2022-01-01T08:00:00|Living Room|21.1| 0.0 | |2022-01-01T09:00:00|Living Room|21.4| 0.5 | |2022-01-01T10:00:00|Living Room|21.8| 1.0 | ### rank Returns the rank of the current row in its partition. For duplicate values, `rank` assigns them the same rank number, skips subsequent ranks (unlike [`dense_rank()`](#dense_rank)), and then continues ranking with the next distinct value. The [`ORDER BY` clause](#order-by-clause) in the `OVER` clause determines ranking order. ```sql rank() ``` [](#view-rank-query-example) View `rank` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, room, temp, rank() OVER ( PARTITION BY room ORDER BY temp ) AS rank FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ``` | time | room |temp|rank| |-------------------|-----------|----|----| |2022-01-01T08:00:00|Living Room|21.1| 1 | |2022-01-01T09:00:00|Living Room|21.4| 2 | |2022-01-01T10:00:00|Living Room|21.8| 3 | |2022-01-01T08:00:00| Kitchen |21.0| 1 | |2022-01-01T10:00:00| Kitchen |22.7| 2 | |2022-01-01T09:00:00| Kitchen |23.0| 3 | [](#compare-dense_rank-rank-and-row_number-functions) Compare `dense_rank`, `rank`, and `row_number` functions Consider a table with duplicate ID values. The following query shows how each ranking function handles duplicate values: ```sql SELECT id, rank() OVER(ORDER BY id), dense_rank() OVER(ORDER BY id), row_number() OVER(ORDER BY id) FROM my_table; ``` |ID |rank|dense\_rank|row\_number| |---|----|-----------|-----------| | 1 | 1 | 1 | 1 | | 1 | 1 | 1 | 2 | | 1 | 1 | 1 | 3 | | 2 | 4 | 2 | 4 | Key differences: * [`rank()`](#rank) assigns the same rank to equal values but skips ranks for subsequent values * [`dense_rank()`](#dense_rank) assigns the same rank to equal values and uses consecutive ranks * [`row_number()`](#row_number) assigns unique sequential numbers regardless of value (non-deterministic) ### row\_number Returns the position of the current row in its partition, counting from 1. The [`ORDER BY` clause](#order-by-clause) in the `OVER` clause determines row order. ```sql row_number() ``` [](#view-row_number-query-example) View `row_number` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, room, temp, row_number() OVER ( PARTITION BY room ORDER BY temp ) AS row_number FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ``` | time | room |temp|row\_number| |-------------------|-----------|----|-----------| |2022-01-01T08:00:00|Living Room|21.1| 1 | |2022-01-01T09:00:00|Living Room|21.4| 2 | |2022-01-01T10:00:00|Living Room|21.8| 3 | |2022-01-01T08:00:00| Kitchen |21.0| 1 | |2022-01-01T10:00:00| Kitchen |22.7| 2 | |2022-01-01T09:00:00| Kitchen |23.0| 3 | [](#compare-dense_rank-rank-and-row_number-functions) Compare `dense_rank`, `rank`, and `row_number` functions Consider a table with duplicate ID values. The following query shows how each ranking function handles duplicate values: ```sql SELECT id, rank() OVER(ORDER BY id), dense_rank() OVER(ORDER BY id), row_number() OVER(ORDER BY id) FROM my_table; ``` |ID |rank|dense\_rank|row\_number| |---|----|-----------|-----------| | 1 | 1 | 1 | 1 | | 1 | 1 | 1 | 2 | | 1 | 1 | 1 | 3 | | 2 | 4 | 2 | 4 | Key differences: * [`rank()`](#rank) assigns the same rank to equal values but skips ranks for subsequent values * [`dense_rank()`](#dense_rank) assigns the same rank to equal values and uses consecutive ranks * [`row_number()`](#row_number) assigns unique sequential numbers regardless of value (non-deterministic) ## Analytical Functions ### first\_value Returns the value from the first row of the window frame. ```sql first_value(expression) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. ##### Related functions [last\_value](#last_value) [](#view-first_value-query-example) View `first_value` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ``` SELECT time, room, temp, first_value(temp) OVER ( PARTITION BY room ORDER BY time ) AS first_value FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|first\_value| |-------------------|-----------|----|------------| |2022-01-01T08:00:00| Kitchen |21.0| 21.0 | |2022-01-01T09:00:00| Kitchen |23.0| 21.0 | |2022-01-01T10:00:00| Kitchen |22.7| 21.0 | |2022-01-01T08:00:00|Living Room|21.1| 21.1 | |2022-01-01T09:00:00|Living Room|21.4| 21.1 | |2022-01-01T10:00:00|Living Room|21.8| 21.1 | ### lag Returns the value from the row that is at the specified offset before the current row in the partition. If the offset row is outside the partition, the function returns the specified default. ```sql lag(expression, offset, default) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic or string operators. * **offset**: How many rows *before* the current row to retrieve the value of*expression* from. Default is `1`. * **default**: The default value to return if the offset is in the partition. Must be of the same type as *expression*. ##### Related functions [lead](#lead) [](#view-lag-query-example) View `lag` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ```sql SELECT time, room, temp, lag(temp, 1, 0) OVER ( PARTITION BY room ORDER BY time ) AS previous_value FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|previous\_value| |-------------------|-----------|----|---------------| |2022-01-01T08:00:00| Kitchen |21.0| 0.0 | |2022-01-01T09:00:00| Kitchen |23.0| 21.0 | |2022-01-01T10:00:00| Kitchen |22.7| 23.0 | |2022-01-01T08:00:00|Living Room|21.1| 0.0 | |2022-01-01T09:00:00|Living Room|21.4| 21.1 | |2022-01-01T10:00:00|Living Room|21.8| 21.4 | [](#calculate-the-difference-from-a-previous-value) Calculate the difference from a previous value Use `LAG` with arithmetic to calculate the difference between the current value and a previous value. This is useful for detecting changes over time. The following query calculates the temperature change from the previous reading: ```sql SELECT time, room, temp, temp - lag(temp, 1) OVER ( PARTITION BY room ORDER BY time ) AS temp_change FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|temp\_change| |-------------------|-----------|----|------------| |2022-01-01T08:00:00| Kitchen |21.0| NULL | |2022-01-01T09:00:00| Kitchen |23.0| 2.0 | |2022-01-01T10:00:00| Kitchen |22.7| \-0.3 | |2022-01-01T08:00:00|Living Room|21.1| NULL | |2022-01-01T09:00:00|Living Room|21.4| 0.3 | |2022-01-01T10:00:00|Living Room|21.8| 0.4 | [](#calculate-the-difference-from-a-value-at-a-specific-time-offset) Calculate the difference from a value at a specific time offset For regularly spaced data, use `LAG` with an offset to compare values from a specific time period ago. The following query compares temperature values that are 1 hour apart (assuming hourly data): ```sql SELECT time, room, temp, lag(temp, 1) OVER ( PARTITION BY room ORDER BY time ) AS temp_1h_ago, temp - lag(temp, 1) OVER ( PARTITION BY room ORDER BY time ) AS hourly_change FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T12:00:00Z' ORDER BY room, time ``` [](#use-a-self-join-for-irregularly-spaced-data) Use a self-join for irregularly spaced data For irregularly spaced time series data where you need to compare values from an exact time offset (like exactly 1 hour ago), use a self-join with interval arithmetic: ```sql SELECT current.time, current.room, current.temp AS current_temp, previous.temp AS temp_1h_ago, current.temp - previous.temp AS hourly_diff FROM home AS current LEFT JOIN home AS previous ON current.room = previous.room AND previous.time = current.time - INTERVAL '1 hour' WHERE current.time >= '2022-01-01T08:00:00Z' AND current.time < '2022-01-01T12:00:00Z' ORDER BY current.room, current.time ``` This approach works when your data points don’t fall at regular intervals, or when you need to compare against a specific time offset regardless of when the previous data point occurred. ### last\_value Returns the value from the last row of the window frame. ```sql last_value(expression) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic operators. ##### Related functions [first\_value](#first_value) [](#view-last_value-query-example) View `last_value` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ```sql SELECT time, room, temp, last_value(temp) OVER ( PARTITION BY room ORDER BY time ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING ) AS last_value FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|last\_value| |-------------------|-----------|----|-----------| |2022-01-01T08:00:00| Kitchen |21.0| 22.7 | |2022-01-01T09:00:00| Kitchen |23.0| 22.7 | |2022-01-01T10:00:00| Kitchen |22.7| 22.7 | |2022-01-01T08:00:00|Living Room|21.1| 21.8 | |2022-01-01T09:00:00|Living Room|21.4| 21.8 | |2022-01-01T10:00:00|Living Room|21.8| 21.8 | ### lead Returns the value from the row that is at the specified offset after the current row in the partition. If the offset row is outside the partition, the function returns the specified default. ```sql lead(expression, offset, default) ``` #### Arguments * **expression**: Expression to operate on. Can be a constant, column, or function, and any combination of arithmetic or string operators. * **offset**: How many rows *before* the current row to retrieve the value of*expression* from. Default is `1`. * **default**: The default value to return if the offset is in the partition. Must be of the same type as *expression*. ##### Related functions [lag](#lag) [](#view-lead-query-example) View `lead` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ```sql SELECT time, room, temp, lead(temp, 1, 0) OVER ( PARTITION BY room ORDER BY time ) AS next_value FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ORDER BY room, time ``` | time | room |temp|next\_value| |-------------------|-----------|----|-----------| |2022-01-01T08:00:00| Kitchen |21.0| 23.0 | |2022-01-01T09:00:00| Kitchen |23.0| 22.7 | |2022-01-01T10:00:00| Kitchen |22.7| 0.0 | |2022-01-01T08:00:00|Living Room|21.1| 21.4 | |2022-01-01T09:00:00|Living Room|21.4| 21.8 | |2022-01-01T10:00:00|Living Room|21.8| 0.0 | ### nth\_value Returns the value from the row that is the nth row of the window frame (counting from 1). If the nth row doesn’t exist, the function returns *null*. ```sql nth_value(expression, n) ``` #### Arguments * **expression**: The expression to operator on. Can be a constant, column, or function, and any combination of arithmetic or string operators. * **n**: Specifies the row number in the current frame and partition to reference. [](#view-lead-query-example) View `lead` query example The following example uses the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data). ```sql SELECT time, room, temp, nth_value(temp, 2) OVER ( PARTITION BY room ) AS second_temp FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time < '2022-01-01T11:00:00Z' ``` | time | room |temp|second\_temp| |-------------------|-----------|----|------------| |2022-01-01T08:00:00| Kitchen |21.0| 22.7 | |2022-01-01T10:00:00| Kitchen |22.7| 22.7 | |2022-01-01T09:00:00| Kitchen |23.0| 22.7 | |2022-01-01T08:00:00|Living Room|21.1| 21.8 | |2022-01-01T10:00:00|Living Room|21.8| 21.8 | |2022-01-01T09:00:00|Living Room|21.4| 21.8 | | time | room | temp | avg_room_temp | | --- | --- | --- | --- | | time | room | temp | avg_room_temp | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 22.0 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 22.0 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 21.25 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.25 | | time | city | wind_direction | | --- | --- | --- | | time | city | wind_direction | | 2025-02-17T13:00:00 | Rome | 33 | | 2025-02-17T08:00:00 | Rome | 34 | | 2025-02-17T23:00:00 | Rome | 49 | | 2025-02-17T17:00:00 | Rome | 86 | | 2025-02-17T11:00:00 | Rome | 93 | | 2025-02-17T12:00:00 | Rome | 115 | | 2025-02-17T10:00:00 | Rome | 156 | | time | room | temp | | --- | --- | --- | | time | room | temp | | 2022-01-01T08:00:00 | Kitchen | 21.0 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | | 2022-01-01T11:00:00 | Kitchen | 22.4 | | 2022-01-01T12:00:00 | Kitchen | 22.5 | | 2022-01-01T13:00:00 | Kitchen | 22.8 | | 2022-01-01T14:00:00 | Kitchen | 22.8 | | 2022-01-01T15:00:00 | Kitchen | 22.7 | | time | city | wind_direction | | --- | --- | --- | | time | city | wind_direction | | 2025-02-17T08:00:00 | Rome | 34 | | 2025-02-17T10:00:00 | Rome | 156 | | 2025-02-17T11:00:00 | Rome | 93 | | 2025-02-17T12:00:00 | Rome | 115 | | 2025-02-17T13:00:00 | Rome | 33 | | 2025-02-17T17:00:00 | Rome | 86 | | 2025-02-17T23:00:00 | Rome | 49 | | time | country | city | wind_direction | | --- | --- | --- | --- | | time | country | city | wind_direction | | 2025-02-17T00:00:00 | France | Strasbourg | 181 | | 2025-02-17T01:00:00 | France | Strasbourg | 228 | | 2025-02-17T02:00:00 | France | Strasbourg | 289 | | 2025-02-17T00:00:00 | France | Toulouse | 24 | | 2025-02-17T01:00:00 | France | Toulouse | 210 | | 2025-02-17T02:00:00 | France | Toulouse | 206 | | 2025-02-17T00:00:00 | Italy | Bari | 2 | | 2025-02-17T01:00:00 | Italy | Bari | 57 | | 2025-02-17T00:00:00 | Italy | Bologna | 351 | | 2025-02-17T01:00:00 | Italy | Bologna | 232 | | 2025-02-17T02:00:00 | Italy | Bologna | 29 | | time | country | city | wind_direction | | --- | --- | --- | --- | | time | country | city | wind_direction | | 2025-02-17T00:00:00 | France | Strasbourg | 181 | | 2025-02-17T01:00:00 | France | Strasbourg | 228 | | 2025-02-17T02:00:00 | France | Strasbourg | 289 | | 2025-02-17T00:00:00 | France | Toulouse | 24 | | 2025-02-17T01:00:00 | France | Toulouse | 210 | | 2025-02-17T02:00:00 | France | Toulouse | 206 | | 2025-02-17T00:00:00 | Italy | Bari | 2 | | 2025-02-17T01:00:00 | Italy | Bari | 57 | | 2025-02-17T00:00:00 | Italy | Bologna | 351 | | 2025-02-17T01:00:00 | Italy | Bologna | 232 | | 2025-02-17T02:00:00 | Italy | Bologna | 29 | | time | room | temp | cume_dist | | --- | --- | --- | --- | | time | room | temp | cume_dist | | 2022-01-01T08:00:00 | Living Room | 21.1 | 0.25 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 0.5 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 0.75 | | 2022-01-01T11:00:00 | Living Room | 22.2 | 1.0 | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 0.25 | | 2022-01-01T11:00:00 | Kitchen | 22.4 | 0.5 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 0.75 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 1.0 | | time | room | temp | dense_rank | | --- | --- | --- | --- | | time | room | temp | dense_rank | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 1 | | 2022-01-01T11:00:00 | Kitchen | 22.4 | 2 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 3 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 4 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 1 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 2 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 3 | | 2022-01-01T11:00:00 | Living Room | 22.2 | 4 | | ID | rank | dense_rank | row_number | | --- | --- | --- | --- | | ID | rank | dense_rank | row_number | | 1 | 1 | 1 | 1 | | 1 | 1 | 1 | 2 | | 1 | 1 | 1 | 3 | | 2 | 4 | 2 | 4 | | time | temp | ntile | | --- | --- | --- | | time | temp | ntile | | 2022-01-01T08:00:00 | 21.0 | 1 | | 2022-01-01T09:00:00 | 23.0 | 1 | | 2022-01-01T10:00:00 | 22.7 | 2 | | 2022-01-01T11:00:00 | 22.4 | 2 | | 2022-01-01T12:00:00 | 22.5 | 3 | | 2022-01-01T13:00:00 | 22.8 | 3 | | 2022-01-01T14:00:00 | 22.8 | 4 | | time | room | temp | percent_rank | | --- | --- | --- | --- | | time | room | temp | percent_rank | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 0.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 0.5 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 1.0 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 0.0 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 0.5 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 1.0 | | time | room | temp | rank | | --- | --- | --- | --- | | time | room | temp | rank | | 2022-01-01T08:00:00 | Living Room | 21.1 | 1 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 2 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 3 | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 1 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 2 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 3 | | ID | rank | dense_rank | row_number | | --- | --- | --- | --- | | ID | rank | dense_rank | row_number | | 1 | 1 | 1 | 1 | | 1 | 1 | 1 | 2 | | 1 | 1 | 1 | 3 | | 2 | 4 | 2 | 4 | | time | room | temp | row_number | | --- | --- | --- | --- | | time | room | temp | row_number | | 2022-01-01T08:00:00 | Living Room | 21.1 | 1 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 2 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 3 | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 1 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 2 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 3 | | ID | rank | dense_rank | row_number | | --- | --- | --- | --- | | ID | rank | dense_rank | row_number | | 1 | 1 | 1 | 1 | | 1 | 1 | 1 | 2 | | 1 | 1 | 1 | 3 | | 2 | 4 | 2 | 4 | | time | room | temp | first_value | | --- | --- | --- | --- | | time | room | temp | first_value | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 21.0 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 21.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 21.0 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 21.1 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.1 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 21.1 | | time | room | temp | previous_value | | --- | --- | --- | --- | | time | room | temp | previous_value | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 0.0 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 21.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 23.0 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 0.0 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.1 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 21.4 | | time | room | temp | temp_change | | --- | --- | --- | --- | | time | room | temp | temp_change | | 2022-01-01T08:00:00 | Kitchen | 21.0 | NULL | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 2.0 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | -0.3 | | 2022-01-01T08:00:00 | Living Room | 21.1 | NULL | | 2022-01-01T09:00:00 | Living Room | 21.4 | 0.3 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 0.4 | | time | room | temp | last_value | | --- | --- | --- | --- | | time | room | temp | last_value | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 22.7 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 22.7 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 22.7 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 21.8 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.8 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 21.8 | | time | room | temp | next_value | | --- | --- | --- | --- | | time | room | temp | next_value | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 23.0 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 22.7 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 0.0 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 21.4 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.8 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 0.0 | | time | room | temp | second_temp | | --- | --- | --- | --- | | time | room | temp | second_temp | | 2022-01-01T08:00:00 | Kitchen | 21.0 | 22.7 | | 2022-01-01T10:00:00 | Kitchen | 22.7 | 22.7 | | 2022-01-01T09:00:00 | Kitchen | 23.0 | 22.7 | | 2022-01-01T08:00:00 | Living Room | 21.1 | 21.8 | | 2022-01-01T10:00:00 | Living Room | 21.8 | 21.8 | | 2022-01-01T09:00:00 | Living Room | 21.4 | 21.8 | --- # GROUP BY clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/group-by/ Use the `GROUP BY` clause to group data by values. `GROUP BY` is an optional clause used to group rows that have the same values for all columns and expressions in the list. To output an aggregation for each group, include an aggregate or selector function in the `SELECT` statement. When `GROUP BY` appears in a query, the `SELECT` list can only use columns that appear in the `GROUP BY` list or in aggregate expressions. > [!Note] > #### Group by aliases > > * `GROUP BY` can use column aliases that are defined in the `SELECT` clause. > * `GROUP BY` won’t use an aliased value if the alias is the same as the > original column name. `GROUP BY` uses the original value of the column, > not the transformed, aliased value. We recommended using column ordinals in > in the `GROUP BY` clause to group by transformed values and maintain the > alias identifier. ## Syntax ```sql SELECT AGGREGATE_FN(field1), tag1 FROM measurement GROUP BY tag1 ``` ## Examples ### Group data by tag values ```sql SELECT AVG(water_level) AS avg_water_level, location FROM h2o_feet GROUP BY location ``` [](#view-example-results) View example results |avg\_water\_level| location | |-----------------|-------------| |5.359142420303919|coyote\_creek| |3.530712094245885|santa\_monica| ### Group data into 15 minute time intervals by tag ``` SELECT location, DATE_BIN(INTERVAL '15 minutes', time) AS time, COUNT(water_level) AS count FROM h2o_feet WHERE time >= timestamp '2019-09-17T00:00:00Z' AND time <= timestamp '2019-09-17T01:00:00Z' GROUP BY 1, location ORDER BY location, 1 ``` [](#view-example-results) View example results The query uses the `COUNT()` function to count the number of `water_level` points per 15 minute interval. Results are then ordered by location and time. | location | time |count| |-------------|--------------------|-----| |coyote\_creek|2019-09-16T23:45:00Z| 1 | |coyote\_creek|2019-09-17T00:00:00Z| 2 | |coyote\_creek|2019-09-17T00:15:00Z| 3 | |coyote\_creek|2019-09-17T00:30:00Z| 2 | |coyote\_creek|2019-09-17T00:45:00Z| 3 | |santa\_monica|2019-09-16T23:45:00Z| 1 | |santa\_monica|2019-09-17T00:00:00Z| 2 | |santa\_monica|2019-09-17T00:15:00Z| 3 | |santa\_monica|2019-09-17T00:30:00Z| 2 | |santa\_monica|2019-09-17T00:45:00Z| 3 | | avg_water_level | location | | --- | --- | | avg_water_level | location | | 5.359142420303919 | coyote_creek | | 3.530712094245885 | santa_monica | | location | time | count | | --- | --- | --- | | location | time | count | | coyote_creek | 2019-09-16T23:45:00Z | 1 | | coyote_creek | 2019-09-17T00:00:00Z | 2 | | coyote_creek | 2019-09-17T00:15:00Z | 3 | | coyote_creek | 2019-09-17T00:30:00Z | 2 | | coyote_creek | 2019-09-17T00:45:00Z | 3 | | santa_monica | 2019-09-16T23:45:00Z | 1 | | santa_monica | 2019-09-17T00:00:00Z | 2 | | santa_monica | 2019-09-17T00:15:00Z | 3 | | santa_monica | 2019-09-17T00:30:00Z | 2 | | santa_monica | 2019-09-17T00:45:00Z | 3 | --- # HAVING clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/having/ The `HAVING` clause places conditions on results created by an aggregate operation on groups. The `HAVING` clause must follow the `GROUP BY` clause and precede the `ORDER BY` clause. > [!Note] > The `WHERE` clause filters rows based on specified conditions *before* the aggregate operation. > The `HAVING` clause filters rows based on specified conditions *after* the aggregate operation has taken place. ## Syntax ```sql SELECT_clause FROM_clause [WHERE_clause] [GROUP_BY_clause] [HAVING_clause] [ORDER_BY_clause] ``` ## Examples ### Return rows with an aggregate value greater than a specified number ``` SELECT MEAN("water_level") AS "mean_water_level", "location" FROM "h2o_feet" GROUP BY "location" HAVING "mean_water_level" > 5 ``` [](#view-example-results) View example results The query returns on rows with values in the `mean_water_level` greater than 5 *after* the aggregate operation. | location |mean\_water\_level| |-------------|------------------| |coyote\_creek|5.359142420303919 | ### Return the average result greater than a specified number from a specific time range ``` SELECT AVG("water_level") AS "avg_water_level", "time" FROM "h2o_feet" WHERE time >= '2019-09-01T00:00:00Z' AND time <= '2019-09-02T00:00:00Z' GROUP BY "time" HAVING "avg_water_level" > 6.82 ORDER BY "time" ``` [](#view-example-results) View example results The query calculates the average water level per time and only returns rows with an average greater than 6.82 during the specified time range. | time |avg\_water\_level | |--------------------|------------------| |2019-09-01T22:06:00Z| 6.8225 | |2019-09-01T22:12:00Z|6.8405000000000005| |2019-09-01T22:30:00Z| 6.8505 | |2019-09-01T22:36:00Z| 6.8325 | #### Related * [Subqueries](/influxdb3/enterprise/reference/sql/subqueries/) | location | mean_water_level | | --- | --- | | location | mean_water_level | | coyote_creek | 5.359142420303919 | | time | avg_water_level | | --- | --- | | time | avg_water_level | | 2019-09-01T22:06:00Z | 6.8225 | | 2019-09-01T22:12:00Z | 6.8405000000000005 | | 2019-09-01T22:30:00Z | 6.8505 | | 2019-09-01T22:36:00Z | 6.8325 | --- # Information schema Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/information-schema/ The underlying query engine for the InfluxDB SQL implementation,[DataFusion](https://arrow.apache.org/datafusion/index.html), provides commands that return metadata related to your data schema. To access this information, use the `SHOW TABLES`, `SHOW COLUMNS`, and`SHOW ALL` commands or query views in the [ISO](https://www.iso.org/) SQL`information_schema` schema. In the context of InfluxDB, a [measurement](/influxdb3/enterprise/reference/glossary/#measurement)is represented as a table. Time, [tags](/influxdb3/enterprise/reference/glossary/#tag), and [fields](/influxdb3/enterprise/reference/glossary/#field) are each represented by columns in a table. ## SHOW TABLES Returns information about tables (measurements) in an InfluxDB bucket. ```sql SHOW TABLES ``` You can also query the `information_schema.tables` view: ```sql SELECT * FROM information_schema.tables ``` #### Example SHOW TABLES output *Measurements are those that use the **`iox` table schema**.* |table\_catalog| table\_schema |table\_name |table\_type| |--------------|-------------------|------------|-----------| | public | iox | home |BASE TABLE | | public | system | queries |BASE TABLE | | public |information\_schema| tables | VIEW | | public |information\_schema| views | VIEW | | public |information\_schema| columns | VIEW | | public |information\_schema|df\_settings| VIEW | ## SHOW COLUMNS Returns information about the schema of a table (measurement) in an InfluxDB bucket. ```sql SHOW COLUMNS FROM example_table ``` You can also query the `information_schema.columns` view: ```sql SELECT table_catalog, table_schema, table_name, column_name, data_type, is_nullable FROM information_schema.columns WHERE table_name = 'example_table' ``` #### Example SHOW COLUMNS output |table\_catalog|table\_schema|table\_name|column\_name| data\_type |is\_nullable| |--------------|-------------|-----------|------------|---------------------------|------------| | public | iox | home | co | Int64 | YES | | public | iox | home | hum | Float64 | YES | | public | iox | home | room | Dictionary(Int32, Utf8) | YES | | public | iox | home | temp | Float64 | YES | | public | iox | home | time |Timestamp(Nanosecond, None)| NO | ## SHOW ALL Returns the configuration options of the current session. ```sql SHOW ALL ``` You can also query the `information_schema.df_settings` view: ```sql SELECT * FROM information_schema.df_settings ``` [](#view-show-all-example-output) View `SHOW ALL` example output | name |setting | |-------------------------------------------------------------|--------| | datafusion.catalog.create\_default\_catalog\_and\_schema | true | | datafusion.catalog.default\_catalog | public | | datafusion.catalog.default\_schema | iox | | datafusion.catalog.format | | | datafusion.catalog.has\_header | false | | datafusion.catalog.information\_schema | true | | datafusion.catalog.location | | | datafusion.execution.batch\_size | 8192 | | datafusion.execution.coalesce\_batches | true | | datafusion.execution.collect\_statistics | false | | datafusion.execution.parquet.enable\_page\_index | false | | datafusion.execution.parquet.metadata\_size\_hint | | | datafusion.execution.parquet.pruning | true | | datafusion.execution.parquet.pushdown\_filters | true | | datafusion.execution.parquet.reorder\_filters | true | | datafusion.execution.parquet.skip\_metadata | true | | datafusion.execution.target\_partitions | 4 | | datafusion.execution.time\_zone |\+00:00 | | datafusion.explain.logical\_plan\_only | false | | datafusion.explain.physical\_plan\_only | false | | datafusion.optimizer.enable\_round\_robin\_repartition | true | | datafusion.optimizer.filter\_null\_join\_keys | false | |datafusion.optimizer.hash\_join\_single\_partition\_threshold|1048576 | | datafusion.optimizer.max\_passes | 3 | | datafusion.optimizer.prefer\_hash\_join | true | | datafusion.optimizer.repartition\_aggregations | true | | datafusion.optimizer.repartition\_file\_min\_size |10485760| | datafusion.optimizer.repartition\_file\_scans | true | | datafusion.optimizer.repartition\_joins | true | | datafusion.optimizer.repartition\_sorts | false | | datafusion.optimizer.repartition\_windows | true | | datafusion.optimizer.skip\_failed\_rules | true | | datafusion.optimizer.top\_down\_join\_key\_reordering | true | | datafusion.sql\_parser.enable\_ident\_normalization | true | | datafusion.sql\_parser.parse\_float\_as\_decimal | false | | table_catalog | table_schema | table_name | table_type | | --- | --- | --- | --- | | table_catalog | table_schema | table_name | table_type | | public | iox | home | BASE TABLE | | public | system | queries | BASE TABLE | | public | information_schema | tables | VIEW | | public | information_schema | views | VIEW | | public | information_schema | columns | VIEW | | public | information_schema | df_settings | VIEW | | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | | --- | --- | --- | --- | --- | --- | | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | | public | iox | home | co | Int64 | YES | | public | iox | home | hum | Float64 | YES | | public | iox | home | room | Dictionary(Int32, Utf8) | YES | | public | iox | home | temp | Float64 | YES | | public | iox | home | time | Timestamp(Nanosecond, None) | NO | | name | setting | | --- | --- | | name | setting | | datafusion.catalog.create_default_catalog_and_schema | true | | datafusion.catalog.default_catalog | public | | datafusion.catalog.default_schema | iox | | datafusion.catalog.format | | | datafusion.catalog.has_header | false | | datafusion.catalog.information_schema | true | | datafusion.catalog.location | | | datafusion.execution.batch_size | 8192 | | datafusion.execution.coalesce_batches | true | | datafusion.execution.collect_statistics | false | | datafusion.execution.parquet.enable_page_index | false | | datafusion.execution.parquet.metadata_size_hint | | | datafusion.execution.parquet.pruning | true | | datafusion.execution.parquet.pushdown_filters | true | | datafusion.execution.parquet.reorder_filters | true | | datafusion.execution.parquet.skip_metadata | true | | datafusion.execution.target_partitions | 4 | | datafusion.execution.time_zone | +00:00 | | datafusion.explain.logical_plan_only | false | | datafusion.explain.physical_plan_only | false | | datafusion.optimizer.enable_round_robin_repartition | true | | datafusion.optimizer.filter_null_join_keys | false | | datafusion.optimizer.hash_join_single_partition_threshold | 1048576 | | datafusion.optimizer.max_passes | 3 | | datafusion.optimizer.prefer_hash_join | true | | datafusion.optimizer.repartition_aggregations | true | | datafusion.optimizer.repartition_file_min_size | 10485760 | | datafusion.optimizer.repartition_file_scans | true | | datafusion.optimizer.repartition_joins | true | | datafusion.optimizer.repartition_sorts | false | | datafusion.optimizer.repartition_windows | true | | datafusion.optimizer.skip_failed_rules | true | | datafusion.optimizer.top_down_join_key_reordering | true | | datafusion.sql_parser.enable_ident_normalization | true | | datafusion.sql_parser.parse_float_as_decimal | false | --- # JOIN clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/join/ Use the `JOIN` clause to join data from different tables together based on logical relationships. * [Syntax](#syntax) * [Join types](#join-types) * [INNER JOIN](#inner-join) * [LEFT [OUTER] JOIN](#left-outer-join) * [RIGHT [OUTER] JOIN](#right-outer-join) * [FULL [OUTER] JOIN](#full-outer-join) * [Troubleshoot joins](#troubleshoot-joins) ## Syntax ```sql SELECT_clause FROM [INNER | LEFT [OUTER] | RIGHT [OUTER] | FULL [OUTER]] JOIN ON [WHERE_clause] [GROUP_BY_clause] [HAVING_clause] [ORDER_BY_clause] ``` ### Arguments * **left\_join\_items**: One or more tables specified in the `FROM` clause that represent the left side of the join. * **right\_join\_items**: One or more tables specified in the `JOIN` clause that represent the right side of the join. * **join\_condition**: A predicate expression in the `ON` clause that uses the`=` (equal to) comparison operator to compare column values from the left side of the join to column values on the right side of the join. Rows with values that match the defined predicate are joined using the specified[join type](#join-types). > [!Note] > If both sides of the join include columns with the same name, you need to > use the fully-qualified reference to prevent ambiguity. > A *fully-qualified reference* uses dot notation to reference both the table name > and the column name–for example: `table_name.column_name` ## Join types The following joins types are supported: [ **INNER JOIN** ](#inner-join) [ **LEFT [OUTER] JOIN** ](#left-outer-join) [ **RIGHT [OUTER] JOIN** ](#right-outer-join) [ **FULL [OUTER] JOIN** ](#full-outer-join) #### Join sample tables The examples below illustrate join methods using the following tables: ##### prod\_line | time |station|produced| |--------------------|-------|--------| |2022-01-01T08:00:00Z| B1 | 26 | |2022-01-01T09:00:00Z| B1 | 54 | |2022-01-01T10:00:00Z| B1 | 56 | |2022-01-01T11:00:00Z| B1 | | |2022-01-01T12:00:00Z| B1 | 82 | ##### errors | time |station|level| message | |--------------------|-------|-----|--------------------| |2022-01-01T10:00:00Z| B1 |warn |Maintenance required| |2022-01-01T11:00:00Z| B1 |crit | Station offline | ### INNER JOIN Inner joins combine rows from tables on the left and right side of the join based on common column values defined in the `ON` clause. Rows that don’t have matching column values are not included in the output table. #### Inner join example [View sample tables](#join-sample-tables) ```sql SELECT * FROM prod_line RIGHT JOIN errors ON prod_line.time = errors.time AND prod_line.station = errors.station ORDER BY prod_line.time ``` ##### Inner join results | time |station|produced| time |station|level| message | |--------------------|-------|--------|--------------------|-------|-----|--------------------| |2022-01-01T10:00:00Z| B1 | 56 |2022-01-01T10:00:00Z| B1 |warn |Maintenance required| |2022-01-01T11:00:00Z| B1 | |2022-01-01T11:00:00Z| B1 |crit | Station offline | ### LEFT [OUTER] JOIN A left outer join returns all rows from the left side of the join and only returns data from the right side of the join in rows with matching column values defined in the `ON` clause. #### Left outer join example [View sample tables](#join-sample-tables) ```sql SELECT * FROM prod_line LEFT JOIN errors ON prod_line.time = errors.time AND prod_line.station = errors.station ORDER BY prod_line.time ``` ##### Left outer join results | time |station|produced| time |station|level| message | |--------------------|-------|--------|--------------------|-------|-----|--------------------| |2022-01-01T08:00:00Z| B1 | 26 | | | | | |2022-01-01T09:00:00Z| B1 | 54 | | | | | |2022-01-01T10:00:00Z| B1 | 56 |2022-01-01T10:00:00Z| B1 |warn |Maintenance required| |2022-01-01T11:00:00Z| B1 | |2022-01-01T11:00:00Z| B1 |crit | Station offline | |2022-01-01T12:00:00Z| B1 | 82 | | | | | ### RIGHT [OUTER] JOIN A right outer join returns all rows from the right side of the join and only returns data from the left side of the join in rows with matching column values defined in the `ON` clause. #### Right outer join example [View sample tables](#join-sample-tables) ```sql SELECT * FROM prod_line RIGHT JOIN errors ON prod_line.time = errors.time AND prod_line.station = errors.station ORDER BY prod_line.time ``` ##### Right outer join results | time |station|produced| time |station|level| message | |--------------------|-------|--------|--------------------|-------|-----|--------------------| |2022-01-01T10:00:00Z| B1 | 56 |2022-01-01T10:00:00Z| B1 |warn |Maintenance required| |2022-01-01T11:00:00Z| B1 | |2022-01-01T11:00:00Z| B1 |crit | Station offline | ### FULL [OUTER] JOIN A full outer join returns all data from the left and right sides of the join and combines rows with matching column values defined in the `ON` clause. Data that is not available on each respective side of the join is NULL. #### Full outer join example [View sample tables](#join-sample-tables) ```sql SELECT * FROM prod_line FULL JOIN errors ON prod_line.time = errors.time AND prod_line.station = errors.station ORDER BY time ``` ##### Full outer join results | time |station|produced| time |station|level| message | |--------------------|-------|--------|--------------------|-------|-----|--------------------| |2022-01-01T08:00:00Z| B1 | 26 | | | | | |2022-01-01T09:00:00Z| B1 | 54 | | | | | |2022-01-01T10:00:00Z| B1 | 56 |2022-01-01T10:00:00Z| B1 |warn |Maintenance required| |2022-01-01T11:00:00Z| B1 | |2022-01-01T11:00:00Z| B1 |crit | Station offline | |2022-01-01T12:00:00Z| B1 | 82 | | | | | ## Troubleshoot joins ### Ambiguous reference to unqualified field If a column exists on both sides of the join and is used in the `SELECT`,`ON`, `WHERE`, `HAVING`, `GROUP BY`, or `ORDER BY` clause, you must use a[fully-qualified reference](#fully-qualified-reference). For example, if both sides of the join have a `time` column and you want to explicitly select a time column, you must specifiy which side of the join to use the time column from: ```sql SELECT prod_line.time, produced, message, FROM prod_line INNER JOIN errors ON -- ... ``` | time | station | produced | | --- | --- | --- | | time | station | produced | | 2022-01-01T08:00:00Z | B1 | 26 | | 2022-01-01T09:00:00Z | B1 | 54 | | 2022-01-01T10:00:00Z | B1 | 56 | | 2022-01-01T11:00:00Z | B1 | | | 2022-01-01T12:00:00Z | B1 | 82 | | time | station | level | message | | --- | --- | --- | --- | | time | station | level | message | | 2022-01-01T10:00:00Z | B1 | warn | Maintenance required | | 2022-01-01T11:00:00Z | B1 | crit | Station offline | | time | station | produced | time | station | level | message | | --- | --- | --- | --- | --- | --- | --- | | time | station | produced | time | station | level | message | | 2022-01-01T10:00:00Z | B1 | 56 | 2022-01-01T10:00:00Z | B1 | warn | Maintenance required | | 2022-01-01T11:00:00Z | B1 | | 2022-01-01T11:00:00Z | B1 | crit | Station offline | | time | station | produced | time | station | level | message | | --- | --- | --- | --- | --- | --- | --- | | time | station | produced | time | station | level | message | | 2022-01-01T08:00:00Z | B1 | 26 | | | | | | 2022-01-01T09:00:00Z | B1 | 54 | | | | | | 2022-01-01T10:00:00Z | B1 | 56 | 2022-01-01T10:00:00Z | B1 | warn | Maintenance required | | 2022-01-01T11:00:00Z | B1 | | 2022-01-01T11:00:00Z | B1 | crit | Station offline | | 2022-01-01T12:00:00Z | B1 | 82 | | | | | | time | station | produced | time | station | level | message | | --- | --- | --- | --- | --- | --- | --- | | time | station | produced | time | station | level | message | | 2022-01-01T10:00:00Z | B1 | 56 | 2022-01-01T10:00:00Z | B1 | warn | Maintenance required | | 2022-01-01T11:00:00Z | B1 | | 2022-01-01T11:00:00Z | B1 | crit | Station offline | | time | station | produced | time | station | level | message | | --- | --- | --- | --- | --- | --- | --- | | time | station | produced | time | station | level | message | | 2022-01-01T08:00:00Z | B1 | 26 | | | | | | 2022-01-01T09:00:00Z | B1 | 54 | | | | | | 2022-01-01T10:00:00Z | B1 | 56 | 2022-01-01T10:00:00Z | B1 | warn | Maintenance required | | 2022-01-01T11:00:00Z | B1 | | 2022-01-01T11:00:00Z | B1 | crit | Station offline | | 2022-01-01T12:00:00Z | B1 | 82 | | | | | --- # LIMIT clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/limit/ The `LIMIT` clause limits the number of rows returned by a query to a specified non-negative integer. ## Syntax ```sql SELECT_clause FROM_clause [WHERE_clause] [GROUP_BY_clause] [ORDER_BY_clause] LIMIT ``` ## Examples ### Limit results to a maximum of five rows ``` SELECT "water_level","location", "time" FROM "h2o_feet" LIMIT 5 ``` [](#view-example-results) View example results The query returns a maximum of 5 results. | location | time |water\_level| |-------------|------------------------|------------| |coyote\_creek|2019-08-28T00:00:00.000Z| 4.206 | |coyote\_creek|2019-08-28T00:06:00.000Z| 4.052 | |coyote\_creek|2019-08-28T00:12:00.000Z| 3.901 | |coyote\_creek|2019-08-28T00:18:00.000Z| 3.773 | |coyote\_creek|2019-08-28T00:24:00.000Z| 3.632 | ### Sort and limit results Use the `ORDER BY` and `LIMIT` clauses to first sort results by specified columns, then limit the sorted results by a specified number. ``` SELECT "water_level", "location", "time" FROM "h2o_feet" ORDER BY "water_level" DESC LIMIT 3 ``` [](#view-example-results) View example results The query returns the highest 3 `water_level` readings in the `h2o_feet` measurement. | location | time |water\_level| |-------------|------------------------|------------| |coyote\_creek|2019-08-27T13:42:00.000Z| \-0.561 | |coyote\_creek|2019-08-29T15:24:00.000Z| \-0.571 | |coyote\_creek|2019-08-28T14:24:00.000Z| \-0.587 | | location | time | water_level | | --- | --- | --- | | location | time | water_level | | coyote_creek | 2019-08-28T00:00:00.000Z | 4.206 | | coyote_creek | 2019-08-28T00:06:00.000Z | 4.052 | | coyote_creek | 2019-08-28T00:12:00.000Z | 3.901 | | coyote_creek | 2019-08-28T00:18:00.000Z | 3.773 | | coyote_creek | 2019-08-28T00:24:00.000Z | 3.632 | | location | time | water_level | | --- | --- | --- | | location | time | water_level | | coyote_creek | 2019-08-27T13:42:00.000Z | -0.561 | | coyote_creek | 2019-08-29T15:24:00.000Z | -0.571 | | coyote_creek | 2019-08-28T14:24:00.000Z | -0.587 | --- # SQL operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/operators/ SQL operators are reserved words or characters which perform certain operations, including comparisons and arithmetic. * [Arithmetic operators](#arithmetic-operators) * [Comparison operators](#comparison-operators) * [Logical operators](#logical-operators) * [Bitwise operators](#bitwise-operators) * [Other operators](#other-operators) ## [Arithmetic operators](/influxdb3/enterprise/reference/sql/operators/arithmetic/) Arithmetic operators take two numeric values (either literals or variables) and perform a calculation that returns a single numeric value. |Operator| Description |Example|Result| |--------|--------------|-------|------| | `+` | Addition |`2 + 2`| `4` | | `-` | Subtraction |`4 - 2`| `2` | | `*` |Multiplication|`2 * 3`| `6` | | `/` | Division |`6 / 3`| `2` | | `%` | Modulo |`7 % 2`| `1` | ## [Comparison operators](/influxdb3/enterprise/reference/sql/operators/comparison/) Comparison operators evaluate the relationship between the left and right operands and return `true` or `false`. | Operator | Meaning | Example | |----------------------|--------------------------------------------------------|--------------------------| | `=` | Equal to | `123 = 123` | | `<>` | Not equal to | `123 <> 456` | | `!=` | Not equal to | `123 != 456` | | `>` | Greater than | `3 > 2` | | `>=` | Greater than or equal to | `3 >= 2` | | `<` | Less than | `1 < 2` | | `<=` | Less than or equal to | `1 <= 2` | | `IS DISTINCT FROM` | Is distinct from | `0 IS DISTINCT FROM 1` | |`IS NOT DISTINCT FROM`| Is not distinct from |`0 IS NOT DISTINCT FROM 1`| | `~` | Matches a regular expression | `'abc' ~ 'a.*'` | | `~*` | Matches a regular expression *(case-insensitive)* | `'Abc' ~* 'A.*'` | | `!~` | Does not match a regular expression | `'abc' !~ 'd.*'` | | `!~*` |Does not match a regular expression *(case-insensitive)*| `'Abc' !~* 'a.*'` | ## [Logical operators](/influxdb3/enterprise/reference/sql/operators/logical/) Logical operators combine or manipulate conditions in a SQL query. |Operator | Meaning | |---------|--------------------------------------------------------------------------| | `AND` | Returns true if both operands are true. Otherwise, returns false. | |`BETWEEN`|Returns true if the left operand is within the range of the right operand.| |`EXISTS` | Returns true if the results of a subquery are not empty. | | `IN` | Returns true if the left operand is in the right operand list. | | `LIKE` |Returns true if the left operand matches the right operand pattern string.| | `NOT` | Negates the subsequent expression. | | `OR` | Returns true if any operand is true. Otherwise, returns false. | ## [Bitwise operators](/influxdb3/enterprise/reference/sql/operators/bitwise/) Bitwise operators perform bitwise operations on bit patterns or binary numerals. |Operator| Meaning |Example |Result| |--------|-------------------|--------|------| | `&` | Bitwise and |`5 & 3` | `1` | | `|` | Bitwise or |`5 | 3` | `7` | | `^` | Bitwise xor |`5 ^ 3` | `6` | | `>>` |Bitwise shift right|`5 >> 3`| `0` | | `<<` |Bitwise shift left |`5 << 3`| `40` | ## [Other operators](/influxdb3/enterprise/reference/sql/operators/other/) SQL supports other miscellaneous operators that perform various operations. | Operator | Meaning | Example | Result | |--------------|------------------------|-----------------------------------------------------------------------------------|-------------| | `||` | Concatenate strings | `'Hello' || ' world'` |`Hello world`| |`AT TIME ZONE`|Apply a time zone offset|*[View example](/influxdb3/enterprise/reference/sql/operators/other/#at-time-zone)*| | | Operator | Description | Example | Result | | --- | --- | --- | --- | | Operator | Description | Example | Result | | + | Addition | 2 + 2 | 4 | | - | Subtraction | 4 - 2 | 2 | | * | Multiplication | 2 * 3 | 6 | | / | Division | 6 / 3 | 2 | | % | Modulo | 7 % 2 | 1 | | Operator | Meaning | Example | | --- | --- | --- | | Operator | Meaning | Example | | = | Equal to | 123 = 123 | | <> | Not equal to | 123 <> 456 | | != | Not equal to | 123 != 456 | | > | Greater than | 3 > 2 | | >= | Greater than or equal to | 3 >= 2 | | < | Less than | 1 < 2 | | <= | Less than or equal to | 1 <= 2 | | IS DISTINCT FROM | Is distinct from | 0 IS DISTINCT FROM 1 | | IS NOT DISTINCT FROM | Is not distinct from | 0 IS NOT DISTINCT FROM 1 | | ~ | Matches a regular expression | 'abc' ~ 'a.*' | | ~* | Matches a regular expression (case-insensitive) | 'Abc' ~* 'A.*' | | !~ | Does not match a regular expression | 'abc' !~ 'd.*' | | !~* | Does not match a regular expression (case-insensitive) | 'Abc' !~* 'a.*' | | Operator | Meaning | | --- | --- | | Operator | Meaning | | AND | Returns true if both operands are true. Otherwise, returns false. | | BETWEEN | Returns true if the left operand is within the range of the right operand. | | EXISTS | Returns true if the results of a subquery are not empty. | | IN | Returns true if the left operand is in the right operand list. | | LIKE | Returns true if the left operand matches the right operand pattern string. | | NOT | Negates the subsequent expression. | | OR | Returns true if any operand is true. Otherwise, returns false. | | Operator | Meaning | Example | Result | | --- | --- | --- | --- | | Operator | Meaning | Example | Result | | & | Bitwise and | 5 & 3 | 1 | | | | Bitwise or | 5 | 3 | 7 | | ^ | Bitwise xor | 5 ^ 3 | 6 | | >> | Bitwise shift right | 5 >> 3 | 0 | | << | Bitwise shift left | 5 << 3 | 40 | | Operator | Meaning | Example | Result | | --- | --- | --- | --- | | Operator | Meaning | Example | Result | | || | Concatenate strings | 'Hello' || ' world' | Hello world | | AT TIME ZONE | Apply a time zone offset | View example | | --- # SQL arithmetic operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/operators/arithmetic/ Arithmetic operators take two numeric values (either literals or variables) and perform a calculation that returns a single numeric value. |Operator| Description | | |--------|--------------|-------------------| | `+` | Addition | [](#addition) | | `-` | Subtraction | [](#subtraction) | | `*` |Multiplication|[](#multiplication)| | `/` | Division | [](#division) | | `%` | Modulo | [](#modulo) | ## \+ The `+` operator adds two operands together and returns the sum. ```sql SELECT 1 + 2 ``` |int64(1) + int64(2)| |-------------------| | 3 | ## \- The `-` operator subtracts the right operand from the left operand and returns the difference. ```sql SELECT 4 - 2 ``` |int64(4) - int64(2)| |-------------------| | 2 | ## \* The `*` operator multiplies two operands together and returns the product. ```sql SELECT 2 * 3 ``` |int64(2) \* int64(3)| |--------------------| | 6 | / ---------- The `/` operator divides the left operand by the right operand and returns the quotient. ```sql SELECT 6 / 3 ``` |int64(6) / int64(3)| |-------------------| | 2 | % ---------- The `%` (modulo) operator divides the left operand by the right operand and returns the remainder. If the left operand is not divisible by the right operand, it returns the left operand. ```sql SELECT 8 % 3 ``` |Int64(8) % Int64(3)| |-------------------| | 2 | ```sql SELECT 3 % 8 ``` |Int64(3) % Int64(8)| |-------------------| | 3 | | Operator | Description | | | --- | --- | --- | | Operator | Description | | | + | Addition | | | - | Subtraction | | | * | Multiplication | | | / | Division | | | % | Modulo | | | int64(1) + int64(2) | | --- | | int64(1) + int64(2) | | 3 | | int64(4) - int64(2) | | --- | | int64(4) - int64(2) | | 2 | | int64(2) * int64(3) | | --- | | int64(2) * int64(3) | | 6 | | int64(6) / int64(3) | | --- | | int64(6) / int64(3) | | 2 | | Int64(8) % Int64(3) | | --- | | Int64(8) % Int64(3) | | 2 | | Int64(3) % Int64(8) | | --- | | Int64(3) % Int64(8) | | 3 | --- # SQL bitwise operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/operators/bitwise/ Bitwise operators perform bitwise operations on bit patterns or binary numerals. |Operator| Meaning | | |--------|-------------------|------------------------| | `&` | Bitwise and | [](#bitwise-and) | | `|` | Bitwise or | [](#bitwise-or) | | `^` | Bitwise xor | [](#bitwise-xor) | | `>>` |Bitwise shift right|[](#bitwise-shift-right)| | `<<` |Bitwise shift left |[](#bitwise-shift-left) | & ---------- The `&` (bitwise AND) operator compares each bit of the left operand to the corresponding bit of the right operand. If both bits are 1, the corresponding result bit is set to 1. Otherwise, the corresponding result bit is set to 0. ```sql SELECT 5 & 3 ``` |Int64(5) & Int64(3)| |-------------------| | 1 | | ---------- The `|` (bitwise OR or inclusive OR) operator compares each bit of the left operand to the corresponding bit of the right operand. If either bit is 1, the corresponding result bit is set to 1. Otherwise, the corresponding result bit is set to 0. ```sql SELECT 5 | 3 ``` |Int64(5) | Int64(3)| |-------------------| | 7 | ^ ---------- The `^` (bitwise XOR or exclusive OR) operator compares each bit of the left operand to the corresponding bit of the right operand. If the bit in one of the operands is 0 and the bit in the other operand is 1, the corresponding result bit is set to 1. Otherwise, the corresponding result bit is set to 0. ```sql SELECT 5 ^ 3 ``` |Int64(5) BIT\_XOR Int64(3)| |--------------------------| | 6 | ## \>\> The `>>` (bitwise shift right) operator shifts the bits in the left operand to the right by the number of positions specified in the right operand. For unsigned numbers, bit positions vacated by the shift operation are filled with 0. For signed numbers, the sign bit is used to fill the vacated bit positions. If the number is positive, the bit position is filled with 0. If the number is negative, the bit position is filled with 1. ```sql SELECT 5 >> 3 ``` |Int64(5) \>\> Int64(3)| |----------------------| | 0 | ## \<\< The `<<` (bitwise shift left) operator shifts the bits in the left operand to the left by the number of positions specified in the right operand. Bit positions vacated by the shift operation are filled with 0. Bits that shift off the end are discarded, including the sign bit. ```sql SELECT 5 << 3 ``` |Int64(5) \<\< Int64(3)| |----------------------| | 40 | | Operator | Meaning | | | --- | --- | --- | | Operator | Meaning | | | & | Bitwise and | | | | | Bitwise or | | | ^ | Bitwise xor | | | >> | Bitwise shift right | | | << | Bitwise shift left | | | Int64(5) & Int64(3) | | --- | | Int64(5) & Int64(3) | | 1 | | Int64(5) | Int64(3) | | --- | | Int64(5) | Int64(3) | | 7 | | Int64(5) BIT_XOR Int64(3) | | --- | | Int64(5) BIT_XOR Int64(3) | | 6 | | Int64(5) >> Int64(3) | | --- | | Int64(5) >> Int64(3) | | 0 | | Int64(5) << Int64(3) | | --- | | Int64(5) << Int64(3) | | 40 | --- # SQL comparison operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/operators/comparison/ Comparison operators evaluate the relationship between the left and right operands and returns `true` or `false`. | Operator | Meaning | | |----------------------|--------------------------------------------------------|------------------------------------| | `=` | Equal to | [](#equal-to) | | `<>` | Not equal to | [](#not-equal-to) | | `!=` | Not equal to | [](#not-equal-to) | | `>` | Greater than | [](#greater-than) | | `>=` | Greater than or equal to | [](#greater-than-or-equal) | | `<` | Less than | [](#less-than) | | `<=` | Less than or equal to | [](#less-than-or-equal) | | `IS DISTINCT FROM` | Is distinct from | [](#is-distinct-from) | |`IS NOT DISTINCT FROM`| Is not distinct from | [](#is-not-distinct-from) | | `~` | Matches a regular expression | [](#regexp-match) | | `~*` | Matches a regular expression *(case-insensitive)* | [](#regexp-match-case-insensitive) | | `!~` | Does not match a regular expression | [](#regexp-nomatch) | | `!~*` |Does not match a regular expression *(case-insensitive)*|[](#regexp-nomatch-case-insensitive)| ## \= The `=` operator compares the left and right operands and, if equal, returns `true`. Otherwise returns `false`. ```sql SELECT 123 = 123 ``` |Int64(123) = Int64(123)| |-----------------------| | true | ## !=, \<\> The `!=` and `<>` operators compare the left and right operands and, if not equal, returns `true`. Otherwise returns `false`. ```sql SELECT 123 != 456 ``` |Int64(123) != Int64(456)| |------------------------| | true | ```sql SELECT 123 <> 456 ``` |Int64(123) != Int64(456)| |------------------------| | true | ## \> The `>` operator compares the left and right operands and, if the left operand is greater than the right operand, returns `true`. Otherwise returns `false`. ```sql SELECT 3 > 2 ``` |Int64(3) \> Int64(2)| |--------------------| | true | ## \>= The `>=` operator compares the left and right operands and, if the left operand is greater than or equal to the right operand, returns `true`. Otherwise returns `false`. ```sql SELECT 3 >= 2 ``` |Int64(3) \>= Int64(2)| |---------------------| | true | ## \< The `<` operator compares the left and right operands and, if the left operand is less than the right operand, returns `true`. Otherwise returns `false`. ```sql SELECT 1 < 2 ``` |Int641(1) \< Int64(2)| |---------------------| | true | ## \<= The `<=` operator compares the left and right operands and, if the left operand is less than or equal to the right operand, returns `true`. Otherwise returns `false`. ```sql SELECT 1 <= 2 ``` |Int641(1) \<= Int64(2)| |----------------------| | true | ## IS DISTINCT FROM The `IS DISTINCT FROM` operator is a *NULL*-safe operator that returns`true` if both operands are not equal; otherwise, it returns `false`. This operator guarantees the result of a comparison is `true` or `false` and not an empty set. ```sql SELECT 0 IS DISTINCT FROM NULL ``` |Int64(0) IS DISTINCT FROM NULL| |------------------------------| | true | ## IS NOT DISTINCT FROM The `IS NOT DISTINCT FROM` operator is a *NULL*-safe operator that returns`true` if both operands are equal or *NULL*; otherwise, it returns `false`. This operator negates [`IS DISTINCT FROM`](#is-distinct-from). ```sql SELECT NULL IS NOT DISTINCT FROM NULL ``` |NULL IS NOT DISTINCT FROM NULL| |------------------------------| | true | ## \~ The `~` operator compares the left string operand to the right regular expression operand and, if it matches (case-sensitive), returns `true`. Otherwise returns `false`. ```sql SELECT 'abc' ~ 'a.*' ``` |Utf8(“abc”) \~ Utf8(“a.\*”)| |---------------------------| | true | ## \~\* The `~*` operator compares the left string operand to the right regular expression operand and, if it matches (case-insensitive), returns `true`. Otherwise returns `false`. ```sql SELECT 'Abc' ~* 'A.*' ``` |Utf8(“Abc”) \~\* Utf8(“A.\*”)| |-----------------------------| | true | ## !\~ The `!~` operator compares the left string operand to the right regular expression operand and, if it does not match (case-sensitive), returns `true`. Otherwise returns `false`. ```sql SELECT 'abc' !~ 'd.*' ``` |Utf8(“abc”) !\~ Utf8(“d.\*”)| |----------------------------| | true | ## !\~\* The `!~*` operator compares the left string operand to the right regular expression operand and, if it does not match (case-insensitive), returns `true`. Otherwise returns `false`. ```sql SELECT 'Abc' !~* 'a.*' ``` |Utf8(“Abc”) !\~\* Utf8(“a.\*”)| |------------------------------| | false | | Operator | Meaning | | | --- | --- | --- | | Operator | Meaning | | | = | Equal to | | | <> | Not equal to | | | != | Not equal to | | | > | Greater than | | | >= | Greater than or equal to | | | < | Less than | | | <= | Less than or equal to | | | IS DISTINCT FROM | Is distinct from | | | IS NOT DISTINCT FROM | Is not distinct from | | | ~ | Matches a regular expression | | | ~* | Matches a regular expression (case-insensitive) | | | !~ | Does not match a regular expression | | | !~* | Does not match a regular expression (case-insensitive) | | | Int64(123) = Int64(123) | | --- | | Int64(123) = Int64(123) | | true | | Int64(123) != Int64(456) | | --- | | Int64(123) != Int64(456) | | true | | Int64(123) != Int64(456) | | --- | | Int64(123) != Int64(456) | | true | | Int64(3) > Int64(2) | | --- | | Int64(3) > Int64(2) | | true | | Int64(3) >= Int64(2) | | --- | | Int64(3) >= Int64(2) | | true | | Int641(1) < Int64(2) | | --- | | Int641(1) < Int64(2) | | true | | Int641(1) <= Int64(2) | | --- | | Int641(1) <= Int64(2) | | true | | Int64(0) IS DISTINCT FROM NULL | | --- | | Int64(0) IS DISTINCT FROM NULL | | true | | NULL IS NOT DISTINCT FROM NULL | | --- | | NULL IS NOT DISTINCT FROM NULL | | true | | Utf8(“abc”) ~ Utf8(“a.*”) | | --- | | Utf8(“abc”) ~ Utf8(“a.*”) | | true | | Utf8(“Abc”) ~* Utf8(“A.*”) | | --- | | Utf8(“Abc”) ~* Utf8(“A.*”) | | true | | Utf8(“abc”) !~ Utf8(“d.*”) | | --- | | Utf8(“abc”) !~ Utf8(“d.*”) | | true | | Utf8(“Abc”) !~* Utf8(“a.*”) | | --- | | Utf8(“Abc”) !~* Utf8(“a.*”) | | false | --- # SQL logical operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/operators/logical/ Logical operators combine or manipulate conditions in a SQL query. |Operator | Meaning | | |---------|--------------------------------------------------------------------------|------------| | `AND` | Returns true if both operands are true. Otherwise, returns false. | [](#and) | |`BETWEEN`|Returns true if the left operand is within the range of the right operand.|[](#between)| |`EXISTS` | Returns true if the results of a subquery are not empty. |[](#exists) | | `IN` | Returns true if the left operand is in the right operand list. | [](#in) | | `LIKE` |Returns true if the left operand matches the right operand pattern string.| [](#like) | | `NOT` | Negates the subsequent expression. | [](#not) | | `OR` | Returns true if any operand is true. Otherwise, returns false. | [](#or) | > [!Note] > #### Sample data > > Query examples on this page use the following sample data sets: > > * [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data) > * [Home sensor actions sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-actions-data) ## AND The `AND` operand returns `true` if both operands are `true`. Otherwise, it returns false. This operator is typically used in the [`WHERE` clause](/influxdb3/enterprise/reference/sql/where/)to combine multiple conditions. ```sql SELECT true AND false AS "AND condition" ``` |AND condition| |-------------| | false | ##### Examples [](#and-operator-in-the-where-clause) `AND` operator in the `WHERE` clause ```sql SELECT * FROM home WHERE co > 10 AND room = 'Kitchen' ``` |co |hum | room |temp| time | |---|----|-------|----|--------------------| |18 |36.9|Kitchen|23.3|2022-01-01T18:00:00Z| |22 |36.6|Kitchen|23.1|2022-01-01T19:00:00Z| |26 |36.5|Kitchen|22.7|2022-01-01T20:00:00Z| ## BETWEEN The `BETWEEN` operator returns `true` if the left numeric operand is within the range specified in the right operand. Otherwise, it returns `false` ```sql SELECT 6 BETWEEN 5 AND 8 AS "BETWEEN condition" ``` |BETWEEN condition| |-----------------| | true | ##### Examples [](#between-operator-in-the-where-clause) `BETWEEN` operator in the `WHERE` clause ```sql SELECT * FROM home WHERE co BETWEEN 5 AND 10 ``` |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 7 | 36 | Kitchen |22.4|2022-01-01T16:00:00Z| | 9 | 36 | Kitchen |22.7|2022-01-01T17:00:00Z| | 5 |35.9|Living Room|22.6|2022-01-01T17:00:00Z| | 9 |36.2|Living Room|22.8|2022-01-01T18:00:00Z| ## EXISTS The `EXISTS` operator returns `true` if result of a[correlated subquery](/influxdb3/enterprise/reference/sql/subqueries/#correlated-subqueries)is not empty. Otherwise it returns `false`. *See [SQL subquery operators](/influxdb3/enterprise/reference/sql/subqueries/#subquery-operators).* ##### Examples [](#exists-operator-with-a-subquery-in-the-where-clause) `EXISTS` operator with a subquery in the `WHERE` clause ```sql SELECT * FROM home home_actions WHERE EXISTS ( SELECT * FROM home WHERE home.co = home_actions.co - 1 ) ORDER BY time ``` |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 1 |36.5| Kitchen |22.8|2022-01-01T13:00:00Z| | 1 |36.3| Kitchen |22.8|2022-01-01T14:00:00Z| | 1 |36.1|Living Room|22.3|2022-01-01T15:00:00Z| | 4 | 36 |Living Room|22.4|2022-01-01T16:00:00Z| | 5 |35.9|Living Room|22.6|2022-01-01T17:00:00Z| |18 |36.9| Kitchen |23.3|2022-01-01T18:00:00Z| ## IN The `IN` operator returns `true` if the left operand is in the right operand list or subquery result. Otherwise, it returns `false`. ```sql SELECT 'John' IN ('Jane', 'John') AS "IN condition" ``` |IN condition| |------------| | true | *See [SQL subquery operators](/influxdb3/enterprise/reference/sql/subqueries/#subquery-operators).* ##### Examples [](#in-operator-with-a-list-in-the-where-clause) `IN` operator with a list in the `WHERE` clause ```sql SELECT * FROM home WHERE room IN ('Bathroom', 'Bedroom', 'Kitchen') LIMIT 4 ``` |co |hum | room |temp| time | |---|----|-------|----|--------------------| | 0 |35.9|Kitchen| 21 |2022-01-01T08:00:00Z| | 0 |36.2|Kitchen| 23 |2022-01-01T09:00:00Z| | 0 |36.1|Kitchen|22.7|2022-01-01T10:00:00Z| | 0 | 36 |Kitchen|22.4|2022-01-01T11:00:00Z| [](#in-operator-with-a-subquery-in-the-where-clause) `IN` operator with a subquery in the `WHERE` clause ```sql SELECT * FROM home WHERE room IN ( SELECT DISTINCT room FROM home_actions ) ORDER BY time LIMIT 4 ``` |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 0 |35.9|Living Room|21.1|2022-01-01T08:00:00Z| | 0 |35.9| Kitchen | 21 |2022-01-01T08:00:00Z| | 0 |35.9|Living Room|21.4|2022-01-01T09:00:00Z| | 0 |36.2| Kitchen | 23 |2022-01-01T09:00:00Z| ## LIKE The `LIKE` operator returns `true` if the left operand matches the string pattern specified in the right operand.`LIKE` expressions support [SQL wildcard characters](#sql-wildcard-characters). ```sql SELECT 'John' LIKE 'J_%n' AS "LIKE condition" ``` |LIKE condition| |--------------| | true | [](#like-operator-in-the-where-clause) `LIKE` operator in the `WHERE` clause ```sql SELECT * FROM home WHERE room LIKE '%Room' LIMIT 4 ``` |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 0 |35.9|Living Room|21.1|2022-01-01T08:00:00Z| | 0 |35.9|Living Room|21.4|2022-01-01T09:00:00Z| | 0 | 36 |Living Room|21.8|2022-01-01T10:00:00Z| | 0 | 36 |Living Room|22.2|2022-01-01T11:00:00Z| ### SQL wildcard characters The InfluxDB SQL implementation supports the following wildcard characters when using the `LIKE` operator to match strings to a pattern. |Character| Description | |---------|----------------------------------| | `%` |Represents zero or more characters| | `_` | Represents any single character | ## NOT The `NOT` operator negates the subsequent expression. ```sql SELECT NOT true AS "NOT condition" ``` |NOT condition| |-------------| | false | ##### Examples [](#not-in) `NOT IN` ```sql SELECT * FROM home WHERE room NOT IN ('Kitchen', 'Bathroom') LIMIT 4 ``` |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 0 |35.9|Living Room|21.1|2022-01-01T08:00:00Z| | 0 |35.9|Living Room|21.4|2022-01-01T09:00:00Z| | 0 | 36 |Living Room|21.8|2022-01-01T10:00:00Z| | 0 | 36 |Living Room|22.2|2022-01-01T11:00:00Z| [](#not-exists) `NOT EXISTS` ```sql SELECT * FROM home home_actions WHERE NOT EXISTS ( SELECT * FROM home WHERE home.co = home_actions.co + 4 ) ORDER BY time ``` |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 7 | 36 | Kitchen |22.4|2022-01-01T16:00:00Z| | 4 | 36 |Living Room|22.4|2022-01-01T16:00:00Z| | 9 | 36 | Kitchen |22.7|2022-01-01T17:00:00Z| | 9 |36.2|Living Room|22.8|2022-01-01T18:00:00Z| |17 |36.4|Living Room|22.2|2022-01-01T20:00:00Z| |26 |36.5| Kitchen |22.7|2022-01-01T20:00:00Z| [](#not-between) `NOT BETWEEN` ```sql SELECT * FROM home WHERE co NOT BETWEEN 1 AND 22 AND room = 'Kitchen' ``` |co |hum | room |temp| time | |---|----|-------|----|--------------------| | 0 |35.9|Kitchen| 21 |2022-01-01T08:00:00Z| | 0 |36.2|Kitchen| 23 |2022-01-01T09:00:00Z| | 0 |36.1|Kitchen|22.7|2022-01-01T10:00:00Z| | 0 | 36 |Kitchen|22.4|2022-01-01T11:00:00Z| | 0 | 36 |Kitchen|22.5|2022-01-01T12:00:00Z| |26 |36.5|Kitchen|22.7|2022-01-01T20:00:00Z| ## OR The `OR` operator returns `true` if any operand is `true`. Otherwise, it returns `false`. This operator is typically used in the [`WHERE` clause](/influxdb3/enterprise/reference/sql/where/)to combine multiple conditions. ```sql SELECT true OR false AS "OR condition" ``` |OR condition| |------------| | true | ##### Examples [](#or-in-the-where-clause) `OR` in the `WHERE` clause ```sql SELECT * FROM home WHERE co > 20 OR temp > 23 ``` |co |hum | room |temp| time | |---|----|-------|----|--------------------| |18 |36.9|Kitchen|23.3|2022-01-01T18:00:00Z| |22 |36.6|Kitchen|23.1|2022-01-01T19:00:00Z| |26 |36.5|Kitchen|22.7|2022-01-01T20:00:00Z| #### Related * [WHERE clause](/influxdb3/enterprise/reference/sql/where/) * [Subquery operators](/influxdb3/enterprise/reference/sql/subqueries/#subquery-operators) | Operator | Meaning | | | --- | --- | --- | | Operator | Meaning | | | AND | Returns true if both operands are true. Otherwise, returns false. | | | BETWEEN | Returns true if the left operand is within the range of the right operand. | | | EXISTS | Returns true if the results of a subquery are not empty. | | | IN | Returns true if the left operand is in the right operand list. | | | LIKE | Returns true if the left operand matches the right operand pattern string. | | | NOT | Negates the subsequent expression. | | | OR | Returns true if any operand is true. Otherwise, returns false. | | | AND condition | | --- | | AND condition | | false | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 18 | 36.9 | Kitchen | 23.3 | 2022-01-01T18:00:00Z | | 22 | 36.6 | Kitchen | 23.1 | 2022-01-01T19:00:00Z | | 26 | 36.5 | Kitchen | 22.7 | 2022-01-01T20:00:00Z | | BETWEEN condition | | --- | | BETWEEN condition | | true | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 7 | 36 | Kitchen | 22.4 | 2022-01-01T16:00:00Z | | 9 | 36 | Kitchen | 22.7 | 2022-01-01T17:00:00Z | | 5 | 35.9 | Living Room | 22.6 | 2022-01-01T17:00:00Z | | 9 | 36.2 | Living Room | 22.8 | 2022-01-01T18:00:00Z | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 1 | 36.5 | Kitchen | 22.8 | 2022-01-01T13:00:00Z | | 1 | 36.3 | Kitchen | 22.8 | 2022-01-01T14:00:00Z | | 1 | 36.1 | Living Room | 22.3 | 2022-01-01T15:00:00Z | | 4 | 36 | Living Room | 22.4 | 2022-01-01T16:00:00Z | | 5 | 35.9 | Living Room | 22.6 | 2022-01-01T17:00:00Z | | 18 | 36.9 | Kitchen | 23.3 | 2022-01-01T18:00:00Z | | IN condition | | --- | | IN condition | | true | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 35.9 | Kitchen | 21 | 2022-01-01T08:00:00Z | | 0 | 36.2 | Kitchen | 23 | 2022-01-01T09:00:00Z | | 0 | 36.1 | Kitchen | 22.7 | 2022-01-01T10:00:00Z | | 0 | 36 | Kitchen | 22.4 | 2022-01-01T11:00:00Z | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 35.9 | Living Room | 21.1 | 2022-01-01T08:00:00Z | | 0 | 35.9 | Kitchen | 21 | 2022-01-01T08:00:00Z | | 0 | 35.9 | Living Room | 21.4 | 2022-01-01T09:00:00Z | | 0 | 36.2 | Kitchen | 23 | 2022-01-01T09:00:00Z | | LIKE condition | | --- | | LIKE condition | | true | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 35.9 | Living Room | 21.1 | 2022-01-01T08:00:00Z | | 0 | 35.9 | Living Room | 21.4 | 2022-01-01T09:00:00Z | | 0 | 36 | Living Room | 21.8 | 2022-01-01T10:00:00Z | | 0 | 36 | Living Room | 22.2 | 2022-01-01T11:00:00Z | | Character | Description | | --- | --- | | Character | Description | | % | Represents zero or more characters | | _ | Represents any single character | | NOT condition | | --- | | NOT condition | | false | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 35.9 | Living Room | 21.1 | 2022-01-01T08:00:00Z | | 0 | 35.9 | Living Room | 21.4 | 2022-01-01T09:00:00Z | | 0 | 36 | Living Room | 21.8 | 2022-01-01T10:00:00Z | | 0 | 36 | Living Room | 22.2 | 2022-01-01T11:00:00Z | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 7 | 36 | Kitchen | 22.4 | 2022-01-01T16:00:00Z | | 4 | 36 | Living Room | 22.4 | 2022-01-01T16:00:00Z | | 9 | 36 | Kitchen | 22.7 | 2022-01-01T17:00:00Z | | 9 | 36.2 | Living Room | 22.8 | 2022-01-01T18:00:00Z | | 17 | 36.4 | Living Room | 22.2 | 2022-01-01T20:00:00Z | | 26 | 36.5 | Kitchen | 22.7 | 2022-01-01T20:00:00Z | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 35.9 | Kitchen | 21 | 2022-01-01T08:00:00Z | | 0 | 36.2 | Kitchen | 23 | 2022-01-01T09:00:00Z | | 0 | 36.1 | Kitchen | 22.7 | 2022-01-01T10:00:00Z | | 0 | 36 | Kitchen | 22.4 | 2022-01-01T11:00:00Z | | 0 | 36 | Kitchen | 22.5 | 2022-01-01T12:00:00Z | | 26 | 36.5 | Kitchen | 22.7 | 2022-01-01T20:00:00Z | | OR condition | | --- | | OR condition | | true | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 18 | 36.9 | Kitchen | 23.3 | 2022-01-01T18:00:00Z | | 22 | 36.6 | Kitchen | 23.1 | 2022-01-01T19:00:00Z | | 26 | 36.5 | Kitchen | 22.7 | 2022-01-01T20:00:00Z | --- # Other SQL operators Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/operators/other/ SQL supports miscellaneous operators that perform various operations. |Operator| Meaning | | |--------|-------------------|------------------------| | `||` |Concatenate strings|[](#concatenate-strings)| || ---------- The `||` operator concatenates two string operands into a single string. ```sql SELECT 'Hello' || ' world' AS "Concatenated" ``` |Concatenated| |------------| |Hello world | ## AT TIME ZONE The `AT TIME ZONE` operator takes the timestamp in the left operand and returns an equivalent timestamp with the updated time and offset of the time zone specified in the right operand. If no time zone is included in the input timestamp’s[Arrow data type](/influxdb3/enterprise/reference/sql/data-types/#sql-and-arrow-data-types), the operator assumes the time is in the time zone specified. Time zone offsets are provided by the operating system time zone database. ```sql SELECT time AT TIME ZONE 'America/Los_Angeles' FROM home ``` [](#convert-a-utc-timestamp-to-a-specified-timezone) Convert a UTC timestamp to a specified timezone ```sql SELECT arrow_cast('2024-01-01 00:00:00', 'Timestamp(Nanosecond, Some("UTC"))') AT TIME ZONE 'America/Los_Angeles' AS 'Time with TZ offset' ``` | Time with TZ offset | |-------------------------| |2023-12-31T16:00:00-08:00| [](#add-a-time-zone-offset-to-a-timestamp-without-a-specified-timezone) Add a time zone offset to a timestamp without a specified timezone ```sql SELECT '2024-01-01 00:00:00' AT TIME ZONE 'America/Los_Angeles' AS 'Local time with TZ offset' ``` |Local time with TZ offset| |-------------------------| |2024-01-01T00:00:00-08:00| | Operator | Meaning | | | --- | --- | --- | | Operator | Meaning | | | || | Concatenate strings | | | Concatenated | | --- | | Concatenated | | Hello world | | Time with TZ offset | | --- | | Time with TZ offset | | 2023-12-31T16:00:00-08:00 | | Local time with TZ offset | | --- | | Local time with TZ offset | | 2024-01-01T00:00:00-08:00 | --- # ORDER BY clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/order-by/ The `ORDER BY` clause sort results by specified columns and order. Sort data based on fields, tags, and timestamps. The following orders are supported: * `ASC`: ascending *(default)* * `DESC`: descending ## Syntax ```sql [SELECT CLAUSE] [FROM CLAUSE] [ ORDER BY expression [ ASC | DESC ][, …] ] ``` > [!Note] > **Note:** If your query includes a `GROUP BY` clause, the `ORDER BY` clause > must appear **after** the `GROUP BY` clause. ## Examples ### Sort data by time with the most recent first ``` SELECT "water_level", "time" FROM "h2o_feet" WHERE "location" = 'coyote_creek' ORDER BY time DESC ``` [](#view-example-results) View example results | time |water\_level| |------------------------|------------| |2019-09-17T16:24:00.000Z| 3.235 | |2019-09-17T16:18:00.000Z| 3.314 | |2019-09-17T16:12:00.000Z| 3.402 | |2019-09-17T16:06:00.000Z| 3.497 | |2019-09-17T16:00:00.000Z| 3.599 | |2019-09-17T15:54:00.000Z| 3.704 | ### Sort data by tag or field values ``` SELECT "water_level", "time", "location" FROM "h2o_feet" ORDER BY "location", "water_level" DESC ``` ### Sort data by selection order ```sql SELECT "location","water_level", "time" FROM "h2o_feet" ORDER BY 1, 2 ``` [](#view-example-results) View example results The query sorts results the location of a column in the `SELECT` statement: first by `location` (1), and second by `water_level` (2). | location | time |water\_level| |-------------|------------------------|------------| |coyote\_creek|2019-08-28T14:30:00.000Z| \-0.61 | |coyote\_creek|2019-08-29T15:18:00.000Z| \-0.594 | |coyote\_creek|2019-08-28T14:36:00.000Z| \-0.591 | |coyote\_creek|2019-08-28T14:24:00.000Z| \-0.587 | |coyote\_creek|2019-08-29T15:24:00.000Z| \-0.571 | |coyote\_creek|2019-08-27T13:42:00.000Z| \-0.561 | | time | water_level | | --- | --- | | time | water_level | | 2019-09-17T16:24:00.000Z | 3.235 | | 2019-09-17T16:18:00.000Z | 3.314 | | 2019-09-17T16:12:00.000Z | 3.402 | | 2019-09-17T16:06:00.000Z | 3.497 | | 2019-09-17T16:00:00.000Z | 3.599 | | 2019-09-17T15:54:00.000Z | 3.704 | | location | time | water_level | | --- | --- | --- | | location | time | water_level | | coyote_creek | 2019-08-28T14:30:00.000Z | -0.61 | | coyote_creek | 2019-08-29T15:18:00.000Z | -0.594 | | coyote_creek | 2019-08-28T14:36:00.000Z | -0.591 | | coyote_creek | 2019-08-28T14:24:00.000Z | -0.587 | | coyote_creek | 2019-08-29T15:24:00.000Z | -0.571 | | coyote_creek | 2019-08-27T13:42:00.000Z | -0.561 | --- # SELECT statement Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/select/ Use the `SELECT` statement to query data from an InfluxDB measurement. The `SELECT` clause is required when querying data in SQL. ### Syntax ```sql SELECT a, b, "time" FROM ``` The SELECT clause supports the following: * `SELECT *` - return all tags, fields and timestamps. * `SELECT DISTINCT` to return all distinct (different) values. * `SELECT <"field" or "tag">` - returns a specified field or tag. * `SELECT <"field" or "tag">, <"field" or "tag">` - returns more than one tag or field. * `SELECT <"field"> AS a `- return the field as the alias. ## Examples The following examples use data from the NOAA database. To download the NOAA test data see [NOAA water sample data](/influxdb/v2/reference/sample-data/#noaa-water-sample-data). ### Select all fields and tags from a measurement ```sql SELECT * FROM h2o_feet LIMIT 10 ``` [](#view-example-results) View example results | level description | location | time |water\_level| |-------------------------|-------------|------------------------|------------| |at or greater than 9 feet|coyote\_creek|2019-09-01T00:00:00.000Z|9.126144144 | |at or greater than 9 feet|coyote\_creek|2019-09-01T00:06:00.000Z| 9.009 | | between 6 and 9 feet |coyote\_creek|2019-09-01T00:12:00.000Z| 8.862 | | between 6 and 9 feet |coyote\_creek|2019-09-01T00:18:00.000Z| 8.714 | ### Select specific tags and fields from a measurement ```sql SELECT "location", "water_level" FROM "h2o_feet" ``` [](#view-example-results-1) View example results | location |water\_level| |-------------|------------| |coyote\_creek|9.126144144 | |coyote\_creek| 9.009 | |coyote\_creek| 8.862 | |coyote\_creek| 8.714 | |coyote\_creek| 8.547 | ### Select a field, tag and timestamp from a measurement ```sql SELECT "water_level", "location", "time" FROM "h2o_feet" ``` [](#view-example-results-2) View example results | location | time |water\_level| |-------------|------------------------|------------| |coyote\_creek|2019-08-20T00:00:00.000Z| 8.638 | |coyote\_creek|2019-08-20T00:06:00.000Z| 8.658 | |coyote\_creek|2019-08-20T00:12:00.000Z| 8.678 | ### Select a field and perform basic arithmetic The following query takes the value of water\_level, multiplies it by 3 and adds 5 to the result. ```sql SELECT ("water_level" * 3) + 5 FROM "h2o_feet" ``` [](#view-example-results-3) View example results | water\_level | |------------------| | 30.128 | |30.641000000000002| |31.142000000000003| | 31.586 | | 32.027 | | 32.378432432 | #### Related * [Subqueries](/influxdb3/enterprise/reference/sql/subqueries/) | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | at or greater than 9 feet | coyote_creek | 2019-09-01T00:00:00.000Z | 9.126144144 | | at or greater than 9 feet | coyote_creek | 2019-09-01T00:06:00.000Z | 9.009 | | between 6 and 9 feet | coyote_creek | 2019-09-01T00:12:00.000Z | 8.862 | | between 6 and 9 feet | coyote_creek | 2019-09-01T00:18:00.000Z | 8.714 | | location | water_level | | --- | --- | | location | water_level | | coyote_creek | 9.126144144 | | coyote_creek | 9.009 | | coyote_creek | 8.862 | | coyote_creek | 8.714 | | coyote_creek | 8.547 | | location | time | water_level | | --- | --- | --- | | location | time | water_level | | coyote_creek | 2019-08-20T00:00:00.000Z | 8.638 | | coyote_creek | 2019-08-20T00:06:00.000Z | 8.658 | | coyote_creek | 2019-08-20T00:12:00.000Z | 8.678 | | water_level | | --- | | water_level | | 30.128 | | 30.641000000000002 | | 31.142000000000003 | | 31.586 | | 32.027 | | 32.378432432 | --- # Subqueries Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/subqueries/ Subqueries (also known as inner queries or nested queries) are queries within a query. Subqueries can be used in `SELECT`, `FROM`, `WHERE`, and `HAVING` clauses. * [Subquery operators](#subquery-operators) * [[ NOT ] EXISTS](#-not--exists) * [[ NOT ] IN](#-not--in) * [SELECT clause subqueries](#select-clause-subqueries) * [FROM clause subqueries](#from-clause-subqueries) * [WHERE clause subqueries](#where-clause-subqueries) * [HAVING clause subqueries](#having-clause-subqueries) * [Subquery categories](#subquery-categories) * [Correlated subqueries](#correlated-subqueries) * [Non-correlated subqueries](#non-correlated-subqueries) * [Scalar subqueries](#scalar-subqueries) * [Non-scalar subqueries](#non-scalar-subqueries) > [!Note] > #### Sample data > > Query examples on this page use the following sample data sets: > > * [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data) > * [Home sensor actions sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-actions-data) > * [NOAA Bay Area weather sample data](/influxdb3/enterprise/reference/sample-data/#noaa-bay-area-weather-data) ## Subquery operators * [[ NOT ] EXISTS](#-not--exists) * [[ NOT ] IN](#-not--in) ### [ NOT ] EXISTS The `EXISTS` operator returns all rows where a*[correlated subquery](#correlated-subqueries)* produces one or more matches for that row. `NOT EXISTS` returns all rows where a *correlated subquery* produces zero matches for that row. Only *correlated subqueries* are supported. #### Syntax ```sql [NOT] EXISTS (subquery) ``` ### [ NOT ] IN The `IN` operator returns all rows where a given expression’s value can be found in the results of a *[correlated subquery](#correlated-subqueries)*.`NOT IN` returns all rows where a given expression’s value cannot be found in the results of a subquery or list of values. #### Syntax ```sql expression [NOT] IN (subquery|list-literal) ``` #### Examples [](#view-in-examples-using-a-query) View `IN` examples using a query [IN](#in)[NOT IN](#not-in) ```sql SELECT time, room, temp FROM home WHERE room IN ( SELECT DISTINCT room FROM home_actions ) ``` ```sql SELECT time, room, temp FROM home WHERE room NOT IN ( SELECT DISTINCT room FROM home_actions ) ``` [](#view-in-examples-using-a-list-literal) View `IN` examples using a list literal [IN](#in-2)[NOT IN](#not-in-2) ```sql SELECT time, room, temp FROM home WHERE room IN ('Bathroom', 'Bedroom', 'Kitchen') ``` ```sql SELECT time, room, temp FROM home WHERE room NOT IN ('Bathroom', 'Bedroom', 'Kitchen') ``` ## SELECT clause subqueries `SELECT` clause subqueries use values returned from the inner query as part of the outer query’s `SELECT` list. The `SELECT` clause only supports [scalar subqueries](#scalar-subqueries) that return a single value per execution of the inner query. The returned value can be unique per row. ### Syntax ```sql SELECT [expression1[, expression2, ..., expressionN],] () ``` > [!Note] > `SELECT` clause subqueries can be used as an alternative to `JOIN` operations. ### Examples [](#select-clause-with-correlated-subquery) `SELECT` clause with correlated subquery ```sql SELECT time, room, co, ( SELECT MAX(description) FROM home_actions WHERE time = home.time AND room = home.room AND level != 'ok' ) AS "Alert Description" FROM home ORDER BY room, time ``` #### Inner query results Because the inner query is a [correlated subquery](#correlated-subqueries), the result depends on the values of `room` and `time` columns in the outer query. The results below represent the action description for each `room` and `time`combination with a `level` value that does not equal `ok`. | time | room | MAX(home\_actions.description) | |--------------------|-----------|-------------------------------------------| |2022-01-01T18:00:00Z| Kitchen |Carbon monoxide level above normal: 18 ppm.| |2022-01-01T19:00:00Z| Kitchen |Carbon monoxide level above normal: 22 ppm.| |2022-01-01T20:00:00Z| Kitchen |Carbon monoxide level above normal: 26 ppm.| |2022-01-01T19:00:00Z|Living Room|Carbon monoxide level above normal: 14 ppm.| |2022-01-01T20:00:00Z|Living Room|Carbon monoxide level above normal: 17 ppm.| #### Outer query results | time | room |co | Alert Description | |--------------------|-----------|---|-------------------------------------------| |2022-01-01T08:00:00Z| Kitchen | 0 | | |2022-01-01T09:00:00Z| Kitchen | 0 | | |2022-01-01T10:00:00Z| Kitchen | 0 | | |2022-01-01T11:00:00Z| Kitchen | 0 | | |2022-01-01T12:00:00Z| Kitchen | 0 | | |2022-01-01T13:00:00Z| Kitchen | 1 | | |2022-01-01T14:00:00Z| Kitchen | 1 | | |2022-01-01T15:00:00Z| Kitchen | 3 | | |2022-01-01T16:00:00Z| Kitchen | 7 | | |2022-01-01T17:00:00Z| Kitchen | 9 | | |2022-01-01T18:00:00Z| Kitchen |18 |Carbon monoxide level above normal: 18 ppm.| |2022-01-01T19:00:00Z| Kitchen |22 |Carbon monoxide level above normal: 22 ppm.| |2022-01-01T20:00:00Z| Kitchen |26 |Carbon monoxide level above normal: 26 ppm.| |2022-01-01T08:00:00Z|Living Room| 0 | | |2022-01-01T09:00:00Z|Living Room| 0 | | |2022-01-01T10:00:00Z|Living Room| 0 | | |2022-01-01T11:00:00Z|Living Room| 0 | | |2022-01-01T12:00:00Z|Living Room| 0 | | |2022-01-01T13:00:00Z|Living Room| 0 | | |2022-01-01T14:00:00Z|Living Room| 0 | | |2022-01-01T15:00:00Z|Living Room| 1 | | |2022-01-01T16:00:00Z|Living Room| 4 | | |2022-01-01T17:00:00Z|Living Room| 5 | | |2022-01-01T18:00:00Z|Living Room| 9 | | |2022-01-01T19:00:00Z|Living Room|14 |Carbon monoxide level above normal: 14 ppm.| |2022-01-01T20:00:00Z|Living Room|17 |Carbon monoxide level above normal: 17 ppm.| ## FROM clause subqueries `FROM` clause subqueries return a set of results that is then queried and operated on by the outer query. ### Syntax ```sql SELECT expression1[, expression2, ..., expressionN] FROM () ``` ### Examples [](#view-from-clause-subquery-example) View `FROM` clause subquery example The following query returns the average of maximum values per room. The inner query returns the maximum value for each field from each room. The outer query uses the results of the inner query and returns the average maximum value for each field. ```sql SELECT AVG(max_co) AS avg_max_co, AVG(max_hum) AS avg_max_hum, AVG(max_temp) AS avg_max_temp FROM ( SELECT room, MAX(co) AS max_co, MAX(hum) AS max_hum, MAX(temp) AS max_temp FROM home GROUP BY room ) ``` #### Inner query results | room |max\_co|max\_hum|max\_temp| |-----------|-------|--------|---------| |Living Room| 17 | 36.4 | 22.8 | | Kitchen | 26 | 36.9 | 23.3 | #### Outer query results |avg\_max\_co|avg\_max\_hum|avg\_max\_temp| |------------|-------------|--------------| | 21.5 | 36.7 | 23.1 | ## WHERE clause subqueries [`WHERE` clause](/influxdb3/enterprise/reference/sql/where/) subqueries compare an expression to the result of the subquery and return *true* or *false*. Rows that evaluate to *false* or NULL are filtered from results. The `WHERE` clause supports correlated and non-correlated subqueries as well as scalar and non-scalar subqueries (depending on the the operator used in the predicate expression). ### Syntax ```sql SELECT expression1[, expression2, ..., expressionN] FROM WHERE expression operator () ``` > [!Note] > `WHERE` clause subqueries can be used as an alternative to `JOIN` operations. ### Examples [](#where-clause-with-scalar-subquery) `WHERE` clause with scalar subquery The following query returns all points with `temp` values above the average of all `temp` values. The subquery returns the average `temp` value. ```sql SELECT * FROM home WHERE temp > ( SELECT AVG(temp) FROM home ) ``` #### Inner query result | AVG(home.temp) | |------------------| |22.396153846153844| #### Outer query result |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 0 |36.2| Kitchen | 23 |2022-01-01T09:00:00Z| | 0 |36.1| Kitchen |22.7|2022-01-01T10:00:00Z| | 0 | 36 | Kitchen |22.4|2022-01-01T11:00:00Z| | 0 | 36 | Kitchen |22.5|2022-01-01T12:00:00Z| | 1 |36.5| Kitchen |22.8|2022-01-01T13:00:00Z| | 1 |36.3| Kitchen |22.8|2022-01-01T14:00:00Z| | 3 |36.2| Kitchen |22.7|2022-01-01T15:00:00Z| | 7 | 36 | Kitchen |22.4|2022-01-01T16:00:00Z| | 9 | 36 | Kitchen |22.7|2022-01-01T17:00:00Z| |18 |36.9| Kitchen |23.3|2022-01-01T18:00:00Z| |22 |36.6| Kitchen |23.1|2022-01-01T19:00:00Z| |26 |36.5| Kitchen |22.7|2022-01-01T20:00:00Z| | 0 | 36 |Living Room|22.4|2022-01-01T13:00:00Z| | 4 | 36 |Living Room|22.4|2022-01-01T16:00:00Z| | 5 |35.9|Living Room|22.6|2022-01-01T17:00:00Z| | 9 |36.2|Living Room|22.8|2022-01-01T18:00:00Z| |14 |36.3|Living Room|22.5|2022-01-01T19:00:00Z| [](#where-clause-with-non-scalar-subquery) `WHERE` clause with non-scalar subquery Non-scalar subqueries must use the `[NOT] IN` or `[NOT] EXISTS` operators and can only return a single column. The values in the returned column are evaluated as a list. The following query returns all points in the `home` measurement associated with the same timestamps as `warn` level alerts in the `home_actions` measurement. ```sql SELECT * FROM home WHERE time IN ( SELECT DISTINCT time FROM home_actions WHERE level = 'warn' ) ``` #### Inner query result | time | |--------------------| |2022-01-01T18:00:00Z| |2022-01-01T19:00:00Z| |2022-01-01T20:00:00Z| #### Outer query result |co |hum | room |temp| time | |---|----|-----------|----|--------------------| |18 |36.9| Kitchen |23.3|2022-01-01T18:00:00Z| | 9 |36.2|Living Room|22.8|2022-01-01T18:00:00Z| |26 |36.5| Kitchen |22.7|2022-01-01T20:00:00Z| |17 |36.4|Living Room|22.2|2022-01-01T20:00:00Z| |22 |36.6| Kitchen |23.1|2022-01-01T19:00:00Z| |14 |36.3|Living Room|22.5|2022-01-01T19:00:00Z| [](#where-clause-with-correlated-subquery) `WHERE` clause with correlated subquery The following query returns rows with temperature values greater than the median temperature value for each room. The subquery in the `WHERE` clause uses the`room` value from the outer query to return the median `temp` value for that specific room. ```sql SELECT time, room, temp FROM home outer_query WHERE temp > ( SELECT median(temp) AS temp FROM home WHERE room = outer_query.room GROUP BY room ) ORDER BY room, time ``` #### Inner query result The result of the inner query depends on the value of `room` in the outer query, but the following table contains the median `temp` value for each room. | room |temp| |-----------|----| |Living Room|22.3| | Kitchen |22.7| #### Outer query result | time | room |temp| |--------------------|-----------|----| |2022-01-01T09:00:00Z| Kitchen | 23 | |2022-01-01T13:00:00Z| Kitchen |22.8| |2022-01-01T14:00:00Z| Kitchen |22.8| |2022-01-01T18:00:00Z| Kitchen |23.3| |2022-01-01T19:00:00Z| Kitchen |23.1| |2022-01-01T13:00:00Z|Living Room|22.4| |2022-01-01T16:00:00Z|Living Room|22.4| |2022-01-01T17:00:00Z|Living Room|22.6| |2022-01-01T18:00:00Z|Living Room|22.8| |2022-01-01T19:00:00Z|Living Room|22.5| ## HAVING clause subqueries [`HAVING` clause](/influxdb3/enterprise/reference/sql/having/) subqueries compare an expression that uses aggregate values returned by aggregate functions in the `SELECT` clause to the result of the subquery and return *true* or *false*. Rows that evaluate to *false* or NULL are filtered from results. The `HAVING` clause supports correlated and non-correlated subqueries as well as scalar and non-scalar subqueries (depending on the the operator used in the predicate expression). ### Syntax ```sql SELECT aggregate_expression1[, aggregate_expression2, ..., aggregate_expressionN] FROM WHERE GROUP BY column_expression1[, column_expression2, ..., column_expressionN] HAVING expression operator () ``` ### Examples [](#having-clause-with-scalar-subquery) `HAVING` clause with scalar subquery The following query returns all two hour blocks of time with average `temp` values greater then the median `temp` value. ```sql SELECT DATE_BIN(INTERVAL '2 hours', time) AS "2-hour block", AVG(temp) AS avg_temp FROM home GROUP BY 1 HAVING avg_temp > ( SELECT MEDIAN(temp) FROM home ) ``` #### Inner query result |MEDIAN(home.temp)| |-----------------| | 22.45 | #### Outer query result | 2-hour block |avg\_temp| |--------------------|---------| |2022-01-01T12:00:00Z| 22.475 | |2022-01-01T16:00:00Z| 22.525 | |2022-01-01T18:00:00Z| 22.925 | |2022-01-01T14:00:00Z| 22.525 | [](#having-clause-with-non-scalar-subquery) `HAVING` clause with non-scalar subquery Non-scalar subqueries must use the `[NOT] IN` or `[NOT] EXISTS` operators and can only return a single column. The values in the returned column are evaluated as a list. The following query returns the maximum `co` and `temp` values within 2-hour windows of time where the `time` value associated with time window is also associated with a warning in the `home_actions` measurement. ```sql SELECT date_bin(INTERVAL '2 hours', time) AS "2-hour block", max(co) AS max_co, max(temp) as max_temp FROM home GROUP BY 1, room HAVING "2-hour block" IN ( SELECT DISTINCT time FROM home_actions WHERE level = 'warn' ) ``` #### Inner query result | time | |--------------------| |2022-01-01T18:00:00Z| |2022-01-01T19:00:00Z| |2022-01-01T20:00:00Z| #### Outer query result | 2-hour block |max\_co|max\_temp| |--------------------|-------|---------| |2022-01-01T18:00:00Z| 14 | 22.8 | |2022-01-01T18:00:00Z| 22 | 23.3 | |2022-01-01T20:00:00Z| 17 | 22.2 | |2022-01-01T20:00:00Z| 26 | 22.7 | [](#having-clause-with-correlated-subquery) `HAVING` clause with correlated subquery The following query returns 2-hour windows of time with average `temp` values greater than the median `temp` value for each room. The subquery in the `HAVING`clause uses the `room` value from the outer query to return the median `temp` value for that specific room. ```sql SELECT time, room, temp FROM home outer_query WHERE temp > ( SELECT median(temp) AS temp FROM home WHERE room = outer_query.room GROUP BY room ) ORDER BY room, time ``` #### Inner query result The result of the inner query depends on the value of `room` in the outer query, but the following table contains the median `temp` value for each room. | room |temp| |-----------|----| |Living Room|22.3| | Kitchen |22.7| #### Outer query result | 2-hour block | room | avg\_temp | |--------------------|-----------|------------------| |2022-01-01T14:00:00Z| Kitchen | 22.75 | |2022-01-01T18:00:00Z| Kitchen |23.200000000000003| |2022-01-01T16:00:00Z|Living Room| 22.5 | |2022-01-01T18:00:00Z|Living Room| 22.65 | ## Subquery categories SQL subqueries can be categorized as one or more of the following based on the behavior of the subquery: * [correlated](#correlated-subqueries) or [non-correlated](#non-correlated-subqueries) * [scalar](#scalar-subqueries) or [non-scalar](#non-scalar-subqueries) ### Correlated subqueries In a **correlated** subquery, the inner query depends on the values of the current row being processed. In the query below, the inner query (`SELECT temp_avg FROM weather WHERE location = home.room`) depends on data (`home.room`) from the outer query (`SELECT time, room, temp FROM home`) and is therefore a *correlated* subquery. ```sql SELECT time, room, temp FROM home WHERE temp = ( SELECT temp_avg FROM weather WHERE location = home.room ) ``` > [!Important] > #### Correlated subquery performance > > Because correlated subqueries depend on the outer query and typically must > execute for each row returned by the outer query, correlated subqueries are**less performant** than non-correlated subqueries. ### Non-correlated subqueries In a **non-correlated** subquery, the inner query *doesn’t* depend on the outer query and executes independently. The inner query executes first, and then passes the results to the outer query. In the query below, the inner query (`SELECT MIN(temp_avg) FROM weather`) can run independently from the outer query (`SELECT time, temp FROM home`) and is therefore a *non-correlated* subquery. ```sql SELECT time, temp FROM home WHERE temp < ( SELECT MIN(temp_avg) FROM weather ) ``` ### Scalar subqueries A **scalar** subquery returns a single value (one column of one row). If no rows are returned, the subquery returns NULL. The example subquery below returns the average value of a specified column. This value is a single scalar value. ```sql SELECT * FROM home WHERE co > (SELECT avg(co) FROM home) ``` ### Non-scalar subqueries A **non-scalar** subquery returns 0, 1, or multiple rows, each of which may contain 1 or multiple columns. For each column, if there is no value to return, the subquery returns NULL. If no rows qualify to be returned, the subquery returns 0 rows. The example subquery below returns all distinct values in a column. Multiple values are returned. ```sql SELECT * FROM home WHERE room IN (SELECT DISTINCT room FROM home_actions) ``` #### Related * [Query data with SQL](/influxdb3/enterprise/query-data/sql/) * [SELECT statement](/influxdb3/enterprise/reference/sql/select/) * [WHERE clause](/influxdb3/enterprise/reference/sql/where/) * [HAVING clause](/influxdb3/enterprise/reference/sql/having/) | time | room | MAX(home_actions.description) | | --- | --- | --- | | time | room | MAX(home_actions.description) | | 2022-01-01T18:00:00Z | Kitchen | Carbon monoxide level above normal: 18 ppm. | | 2022-01-01T19:00:00Z | Kitchen | Carbon monoxide level above normal: 22 ppm. | | 2022-01-01T20:00:00Z | Kitchen | Carbon monoxide level above normal: 26 ppm. | | 2022-01-01T19:00:00Z | Living Room | Carbon monoxide level above normal: 14 ppm. | | 2022-01-01T20:00:00Z | Living Room | Carbon monoxide level above normal: 17 ppm. | | time | room | co | Alert Description | | --- | --- | --- | --- | | time | room | co | Alert Description | | 2022-01-01T08:00:00Z | Kitchen | 0 | | | 2022-01-01T09:00:00Z | Kitchen | 0 | | | 2022-01-01T10:00:00Z | Kitchen | 0 | | | 2022-01-01T11:00:00Z | Kitchen | 0 | | | 2022-01-01T12:00:00Z | Kitchen | 0 | | | 2022-01-01T13:00:00Z | Kitchen | 1 | | | 2022-01-01T14:00:00Z | Kitchen | 1 | | | 2022-01-01T15:00:00Z | Kitchen | 3 | | | 2022-01-01T16:00:00Z | Kitchen | 7 | | | 2022-01-01T17:00:00Z | Kitchen | 9 | | | 2022-01-01T18:00:00Z | Kitchen | 18 | Carbon monoxide level above normal: 18 ppm. | | 2022-01-01T19:00:00Z | Kitchen | 22 | Carbon monoxide level above normal: 22 ppm. | | 2022-01-01T20:00:00Z | Kitchen | 26 | Carbon monoxide level above normal: 26 ppm. | | 2022-01-01T08:00:00Z | Living Room | 0 | | | 2022-01-01T09:00:00Z | Living Room | 0 | | | 2022-01-01T10:00:00Z | Living Room | 0 | | | 2022-01-01T11:00:00Z | Living Room | 0 | | | 2022-01-01T12:00:00Z | Living Room | 0 | | | 2022-01-01T13:00:00Z | Living Room | 0 | | | 2022-01-01T14:00:00Z | Living Room | 0 | | | 2022-01-01T15:00:00Z | Living Room | 1 | | | 2022-01-01T16:00:00Z | Living Room | 4 | | | 2022-01-01T17:00:00Z | Living Room | 5 | | | 2022-01-01T18:00:00Z | Living Room | 9 | | | 2022-01-01T19:00:00Z | Living Room | 14 | Carbon monoxide level above normal: 14 ppm. | | 2022-01-01T20:00:00Z | Living Room | 17 | Carbon monoxide level above normal: 17 ppm. | | room | max_co | max_hum | max_temp | | --- | --- | --- | --- | | room | max_co | max_hum | max_temp | | Living Room | 17 | 36.4 | 22.8 | | Kitchen | 26 | 36.9 | 23.3 | | avg_max_co | avg_max_hum | avg_max_temp | | --- | --- | --- | | avg_max_co | avg_max_hum | avg_max_temp | | 21.5 | 36.7 | 23.1 | | AVG(home.temp) | | --- | | AVG(home.temp) | | 22.396153846153844 | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 36.2 | Kitchen | 23 | 2022-01-01T09:00:00Z | | 0 | 36.1 | Kitchen | 22.7 | 2022-01-01T10:00:00Z | | 0 | 36 | Kitchen | 22.4 | 2022-01-01T11:00:00Z | | 0 | 36 | Kitchen | 22.5 | 2022-01-01T12:00:00Z | | 1 | 36.5 | Kitchen | 22.8 | 2022-01-01T13:00:00Z | | 1 | 36.3 | Kitchen | 22.8 | 2022-01-01T14:00:00Z | | 3 | 36.2 | Kitchen | 22.7 | 2022-01-01T15:00:00Z | | 7 | 36 | Kitchen | 22.4 | 2022-01-01T16:00:00Z | | 9 | 36 | Kitchen | 22.7 | 2022-01-01T17:00:00Z | | 18 | 36.9 | Kitchen | 23.3 | 2022-01-01T18:00:00Z | | 22 | 36.6 | Kitchen | 23.1 | 2022-01-01T19:00:00Z | | 26 | 36.5 | Kitchen | 22.7 | 2022-01-01T20:00:00Z | | 0 | 36 | Living Room | 22.4 | 2022-01-01T13:00:00Z | | 4 | 36 | Living Room | 22.4 | 2022-01-01T16:00:00Z | | 5 | 35.9 | Living Room | 22.6 | 2022-01-01T17:00:00Z | | 9 | 36.2 | Living Room | 22.8 | 2022-01-01T18:00:00Z | | 14 | 36.3 | Living Room | 22.5 | 2022-01-01T19:00:00Z | | time | | --- | | time | | 2022-01-01T18:00:00Z | | 2022-01-01T19:00:00Z | | 2022-01-01T20:00:00Z | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 18 | 36.9 | Kitchen | 23.3 | 2022-01-01T18:00:00Z | | 9 | 36.2 | Living Room | 22.8 | 2022-01-01T18:00:00Z | | 26 | 36.5 | Kitchen | 22.7 | 2022-01-01T20:00:00Z | | 17 | 36.4 | Living Room | 22.2 | 2022-01-01T20:00:00Z | | 22 | 36.6 | Kitchen | 23.1 | 2022-01-01T19:00:00Z | | 14 | 36.3 | Living Room | 22.5 | 2022-01-01T19:00:00Z | | room | temp | | --- | --- | | room | temp | | Living Room | 22.3 | | Kitchen | 22.7 | | time | room | temp | | --- | --- | --- | | time | room | temp | | 2022-01-01T09:00:00Z | Kitchen | 23 | | 2022-01-01T13:00:00Z | Kitchen | 22.8 | | 2022-01-01T14:00:00Z | Kitchen | 22.8 | | 2022-01-01T18:00:00Z | Kitchen | 23.3 | | 2022-01-01T19:00:00Z | Kitchen | 23.1 | | 2022-01-01T13:00:00Z | Living Room | 22.4 | | 2022-01-01T16:00:00Z | Living Room | 22.4 | | 2022-01-01T17:00:00Z | Living Room | 22.6 | | 2022-01-01T18:00:00Z | Living Room | 22.8 | | 2022-01-01T19:00:00Z | Living Room | 22.5 | | MEDIAN(home.temp) | | --- | | MEDIAN(home.temp) | | 22.45 | | 2-hour block | avg_temp | | --- | --- | | 2-hour block | avg_temp | | 2022-01-01T12:00:00Z | 22.475 | | 2022-01-01T16:00:00Z | 22.525 | | 2022-01-01T18:00:00Z | 22.925 | | 2022-01-01T14:00:00Z | 22.525 | | time | | --- | | time | | 2022-01-01T18:00:00Z | | 2022-01-01T19:00:00Z | | 2022-01-01T20:00:00Z | | 2-hour block | max_co | max_temp | | --- | --- | --- | | 2-hour block | max_co | max_temp | | 2022-01-01T18:00:00Z | 14 | 22.8 | | 2022-01-01T18:00:00Z | 22 | 23.3 | | 2022-01-01T20:00:00Z | 17 | 22.2 | | 2022-01-01T20:00:00Z | 26 | 22.7 | | room | temp | | --- | --- | | room | temp | | Living Room | 22.3 | | Kitchen | 22.7 | | 2-hour block | room | avg_temp | | --- | --- | --- | | 2-hour block | room | avg_temp | | 2022-01-01T14:00:00Z | Kitchen | 22.75 | | 2022-01-01T18:00:00Z | Kitchen | 23.200000000000003 | | 2022-01-01T16:00:00Z | Living Room | 22.5 | | 2022-01-01T18:00:00Z | Living Room | 22.65 | --- # Table value constructor Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/table-value-constructor/ The table value constructor (TVC) uses the `VALUES` keyword to specify a set of row value expressions to construct into a table. The TVC can be used in the `FROM` clause to build an ad hoc table at query time. ```sql VALUES (row_value_list)[,...n] ``` ##### Arguments * **row\_value\_list**: Comma-delimited list of column values. Enclose each list in parentheses and separate multiple lists with commas. Each list must have the same number of values and values must be in the same order as columns in the table. Each list must contain a value for each column. ## Usage ```sql SELECT expression[,...n] FROM (VALUES (row_value_list)[,...n]) [AS] table_name(column_name[,...n]) ``` > [!Note] > When using the TVC, the `AS` keyword is optional and implied when naming the > table and providing column names. ## Examples ### Select data from an ad hoc table ```sql SELECT * FROM (VALUES ('2023-01-01 12:00:00'::TIMESTAMP, 1.23, 4.56), ('2023-01-01 13:00:00'::TIMESTAMP, 2.46, 8.1), ('2023-01-01 13:00:00'::TIMESTAMP, 4.81, 16.2) ) AS data(time, f1, f2) ``` | time | f1 | f2 | |--------------------|----|----| |2023-01-01T12:00:00Z|1.23|4.56| |2023-01-01T13:00:00Z|2.46|8.1 | |2023-01-01T13:00:00Z|4.81|16.2| | time | f1 | f2 | | --- | --- | --- | | time | f1 | f2 | | 2023-01-01T12:00:00Z | 1.23 | 4.56 | | 2023-01-01T13:00:00Z | 2.46 | 8.1 | | 2023-01-01T13:00:00Z | 4.81 | 16.2 | --- # UNION clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/union/ The `UNION` clause combines the results of two or more `SELECT` statements into a single result set. By default, `UNION` only keeps unique rows. To keep all rows, including duplicates, use `UNION ALL`. **When using the `UNION` clause**: * The number of columns in each result set must be the same. * Columns must be in the same order and of the same or compatible data types. ## Syntax ```sql SELECT expression[,...n] FROM measurement_1 UNION [ALL] SELECT expression[,...n] FROM measurement_2 ``` ## Examples ### Union results from different measurements ```sql ( SELECT 'h2o_pH' AS measurement, time, "pH" AS "water_pH" FROM "h2o_pH" LIMIT 4 ) UNION ( SELECT 'h2o_quality' AS measurement, time, index FROM h2o_quality LIMIT 4 ) ``` [](#view-example-results) View example results |measurement | time |water\_pH| |------------|--------------------|---------| | h2o\_pH |2019-08-27T00:12:00Z| 7 | | h2o\_pH |2019-08-27T00:18:00Z| 8 | |h2o\_quality|2019-09-11T01:06:00Z| 89 | | h2o\_pH |2019-08-27T00:06:00Z| 7 | |h2o\_quality|2019-09-11T00:00:00Z| 26 | |h2o\_quality|2019-09-11T01:00:00Z| 19 | |h2o\_quality|2019-09-11T00:48:00Z| 65 | | h2o\_pH |2019-08-27T00:00:00Z| 8 | ### Return the highest and lowest three results in a single result set *The following example uses the sample data set provided in[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol).* ``` ( SELECT 'low' as type, time, co FROM home ORDER BY co ASC LIMIT 3 ) UNION ( SELECT 'high' as type, time, co FROM home ORDER BY co DESC LIMIT 3 ) ``` [](#view-example-results) View example results |type| time |co | |----|--------------------|---| |high|2022-01-01T20:00:00Z|26 | |high|2022-01-01T19:00:00Z|22 | |high|2022-01-01T18:00:00Z|18 | |low |2022-01-01T14:00:00Z| 0 | |low |2022-01-01T10:00:00Z| 0 | |low |2022-01-01T08:00:00Z| 0 | ### Union query results with custom data *The following example uses the sample data set provided in[Get started with InfluxDB tutorial](/influxdb3/enterprise/get-started/write/#construct-line-protocol). It also uses the [table value constructor](/influxdb3/enterprise/reference/sql/table-value-constructor/)to build a table with custom data.* ```sql SELECT * FROM home WHERE time >= '2022-01-01T08:00:00Z' AND time <= '2022-01-01T12:00:00Z' UNION SELECT * FROM (VALUES (0, 34.2, 'Bedroom', 21.1, '2022-01-01T08:00:00Z'::TIMESTAMP), (0, 34.5, 'Bedroom', 21.2, '2022-01-01T09:00:00Z'::TIMESTAMP), (0, 34.6, 'Bedroom', 21.5, '2022-01-01T10:00:00Z'::TIMESTAMP), (0, 34.5, 'Bedroom', 21.8, '2022-01-01T11:00:00Z'::TIMESTAMP), (0, 33.9, 'Bedroom', 22.0, '2022-01-01T12:00:00Z'::TIMESTAMP) ) newRoom(co, hum, room, temp, time) ORDER BY room, time ``` [](#view-example-results) View example results |co |hum | room |temp| time | |---|----|-----------|----|--------------------| | 0 |34.2| Bedroom |21.1|2022-01-01T08:00:00Z| | 0 |34.5| Bedroom |21.2|2022-01-01T09:00:00Z| | 0 |34.6| Bedroom |21.5|2022-01-01T10:00:00Z| | 0 |34.5| Bedroom |21.8|2022-01-01T11:00:00Z| | 0 |33.9| Bedroom | 22 |2022-01-01T12:00:00Z| | 0 |35.9| Kitchen | 21 |2022-01-01T08:00:00Z| | 0 |36.2| Kitchen | 23 |2022-01-01T09:00:00Z| | 0 |36.1| Kitchen |22.7|2022-01-01T10:00:00Z| | 0 | 36 | Kitchen |22.4|2022-01-01T11:00:00Z| | 0 | 36 | Kitchen |22.5|2022-01-01T12:00:00Z| | 0 |35.9|Living Room|21.1|2022-01-01T08:00:00Z| | 0 |35.9|Living Room|21.4|2022-01-01T09:00:00Z| | 0 | 36 |Living Room|21.8|2022-01-01T10:00:00Z| | 0 | 36 |Living Room|22.2|2022-01-01T11:00:00Z| | 0 |35.9|Living Room|22.2|2022-01-01T12:00:00Z| | measurement | time | water_pH | | --- | --- | --- | | measurement | time | water_pH | | h2o_pH | 2019-08-27T00:12:00Z | 7 | | h2o_pH | 2019-08-27T00:18:00Z | 8 | | h2o_quality | 2019-09-11T01:06:00Z | 89 | | h2o_pH | 2019-08-27T00:06:00Z | 7 | | h2o_quality | 2019-09-11T00:00:00Z | 26 | | h2o_quality | 2019-09-11T01:00:00Z | 19 | | h2o_quality | 2019-09-11T00:48:00Z | 65 | | h2o_pH | 2019-08-27T00:00:00Z | 8 | | type | time | co | | --- | --- | --- | | type | time | co | | high | 2022-01-01T20:00:00Z | 26 | | high | 2022-01-01T19:00:00Z | 22 | | high | 2022-01-01T18:00:00Z | 18 | | low | 2022-01-01T14:00:00Z | 0 | | low | 2022-01-01T10:00:00Z | 0 | | low | 2022-01-01T08:00:00Z | 0 | | co | hum | room | temp | time | | --- | --- | --- | --- | --- | | co | hum | room | temp | time | | 0 | 34.2 | Bedroom | 21.1 | 2022-01-01T08:00:00Z | | 0 | 34.5 | Bedroom | 21.2 | 2022-01-01T09:00:00Z | | 0 | 34.6 | Bedroom | 21.5 | 2022-01-01T10:00:00Z | | 0 | 34.5 | Bedroom | 21.8 | 2022-01-01T11:00:00Z | | 0 | 33.9 | Bedroom | 22 | 2022-01-01T12:00:00Z | | 0 | 35.9 | Kitchen | 21 | 2022-01-01T08:00:00Z | | 0 | 36.2 | Kitchen | 23 | 2022-01-01T09:00:00Z | | 0 | 36.1 | Kitchen | 22.7 | 2022-01-01T10:00:00Z | | 0 | 36 | Kitchen | 22.4 | 2022-01-01T11:00:00Z | | 0 | 36 | Kitchen | 22.5 | 2022-01-01T12:00:00Z | | 0 | 35.9 | Living Room | 21.1 | 2022-01-01T08:00:00Z | | 0 | 35.9 | Living Room | 21.4 | 2022-01-01T09:00:00Z | | 0 | 36 | Living Room | 21.8 | 2022-01-01T10:00:00Z | | 0 | 36 | Living Room | 22.2 | 2022-01-01T11:00:00Z | | 0 | 35.9 | Living Room | 22.2 | 2022-01-01T12:00:00Z | --- # WHERE clause Source: https://docs.influxdata.com/influxdb3/enterprise/reference/sql/where/ Use the `WHERE` clause to filter results based on fields, tags, or timestamps. ## Syntax ```sql SELECT_clause FROM_clause WHERE [(AND|OR) [...]] ``` > [!Note] > **Note:** Unlike InfluxQL, SQL **supports** `OR` in the `WHERE` clause to > specify multiple conditions, including time ranges. ## Examples Note that single quotes are required for string literals in the `WHERE` clause. ### Filter data based on field values ``` SELECT * FROM "h2o_feet" WHERE "water_level" >= 9.78 ``` [](#view-example-results) View example results The query returns data from the `h2o_feet` measurement with `water_level` field values that are greater than or equal to 9.78. | level description | location | time |water\_level| |-------------------------|-------------|------------------------|------------| |at or greater than 9 feet|coyote\_creek|2019-09-01T23:06:00.000Z| 9.8 | |at or greater than 9 feet|coyote\_creek|2019-09-01T23:12:00.000Z| 9.829 | |at or greater than 9 feet|coyote\_creek|2019-09-01T23:18:00.000Z| 9.862 | |at or greater than 9 feet|coyote\_creek|2019-09-01T23:24:00.000Z| 9.892 | |at or greater than 9 feet|coyote\_creek|2019-09-01T23:30:00.000Z| 9.902 | |at or greater than 9 feet|coyote\_creek|2019-09-01T23:36:00.000Z| 9.898 | ### Filter data based on specific tag and field values ``` SELECT * FROM "h2o_feet" WHERE "location" = 'santa_monica' and "level description" = 'below 3 feet' ``` [](#view-example-results) View example results The query returns all data from the `h2o_feet` measurement with the `location` tag key, `santa_monica`, and a `level description` field value that equals `below 3 feet`. |level description| location | time |water\_level| |-----------------|-------------|------------------------|------------| | below 3 feet |santa\_monica|2019-09-01T00:00:00.000Z| 1.529 | | below 3 feet |santa\_monica|2019-09-01T00:06:00.000Z| 1.444 | | below 3 feet |santa\_monica|2019-09-01T00:12:00.000Z| 1.335 | | below 3 feet |santa\_monica|2019-09-01T00:18:00.000Z| 1.345 | | below 3 feet |santa\_monica|2019-09-01T00:24:00.000Z| 1.27 | ### Filter data within a specific time period ``` SELECT * FROM h2o_feet WHERE "location" = 'santa_monica' AND "time" >= '2019-08-19T12:00:00Z' AND "time" <= '2019-08-19T13:00:00Z' ``` [](#view-example-results) View example results The query returns results with timestamps greater than or equal to `08-19-2019T12:00:00Z` and less than or equal to `08-19-2019T13:00:00Z`. |level description| location | time |water\_level| |-----------------|-------------|------------------------|------------| | below 3 feet |santa\_monica|2019-08-19T12:00:00.000Z| 2.533 | | below 3 feet |santa\_monica|2019-08-19T12:06:00.000Z| 2.543 | | below 3 feet |santa\_monica|2019-08-19T12:12:00.000Z| 2.385 | | below 3 feet |santa\_monica|2019-08-19T12:18:00.000Z| 2.362 | | below 3 feet |santa\_monica|2019-08-19T12:24:00.000Z| 2.405 | | below 3 feet |santa\_monica|2019-08-19T12:30:00.000Z| 2.398 | ### Filter data by dynamic date ranges Use date and time functions to filter data by relative time periods that automatically update. #### Get data from yesterday ``` SELECT * FROM h2o_feet WHERE "location" = 'santa_monica' AND time >= DATE_TRUNC('day', NOW() - INTERVAL '1 day') AND time < DATE_TRUNC('day', NOW()) ``` [](#view-example-results) View example results This query filters data to include only records from the previous calendar day: * `NOW() - INTERVAL '1 day'` calculates yesterday’s timestamp * `DATE_TRUNC('day', ...)` truncates to the start of that day (00:00:00) * The range spans from yesterday at 00:00:00 to today at 00:00:00 |level description| location | time |water\_level| |-----------------|-------------|------------------------|------------| | below 3 feet |santa\_monica|2019-08-18T12:00:00.000Z| 2.533 | | below 3 feet |santa\_monica|2019-08-18T12:06:00.000Z| 2.543 | | below 3 feet |santa\_monica|2019-08-18T12:12:00.000Z| 2.385 | | below 3 feet |santa\_monica|2019-08-18T12:18:00.000Z| 2.362 | | below 3 feet |santa\_monica|2019-08-18T12:24:00.000Z| 2.405 | | below 3 feet |santa\_monica|2019-08-18T12:30:00.000Z| 2.398 | #### Get data from the last 24 hours ``` SELECT * FROM h2o_feet WHERE time >= NOW() - INTERVAL '1 day' AND location = 'santa_monica' ``` [](#view-example-results) View example results This query returns data from exactly 24 hours before the current time. Unlike the “yesterday” example, this creates a rolling 24-hour window that moves with the current time. |level description| location | time |water\_level| |-----------------|-------------|------------------------|------------| | below 3 feet |santa\_monica|2019-08-18T18:00:00.000Z| 2.120 | | below 3 feet |santa\_monica|2019-08-18T18:06:00.000Z| 2.028 | | below 3 feet |santa\_monica|2019-08-18T18:12:00.000Z| 1.982 | | below 3 feet |santa\_monica|2019-08-19T06:00:00.000Z| 1.825 | | below 3 feet |santa\_monica|2019-08-19T06:06:00.000Z| 1.753 | | below 3 feet |santa\_monica|2019-08-19T06:12:00.000Z| 1.691 | #### Get data from the current week ``` SELECT * FROM h2o_feet WHERE time >= DATE_TRUNC('week', NOW()) AND location = 'santa_monica' ``` [](#view-example-results) View example results This query returns all data from the start of the current week (Monday at 00:00:00) to the current time. The DATE\_TRUNC(‘week’, NOW()) function truncates the current timestamp to the beginning of the week. |level description| location | time |water\_level| |-----------------|-------------|------------------------|------------| | below 3 feet |santa\_monica|2019-08-12T00:00:00.000Z| 2.064 | | below 3 feet |santa\_monica|2019-08-14T09:30:00.000Z| 2.116 | | below 3 feet |santa\_monica|2019-08-16T15:45:00.000Z| 1.952 | | below 3 feet |santa\_monica|2019-08-18T12:00:00.000Z| 2.533 | | below 3 feet |santa\_monica|2019-08-18T18:00:00.000Z| 2.385 | | below 3 feet |santa\_monica|2019-08-19T10:30:00.000Z| 1.691 | ### Filter data using the OR operator ```sql SELECT * FROM "h2o_feet" WHERE "level description" = 'less than 3 feet' OR "water_level" < 2.5 ``` [](#view-example-results) View example results The query returns results with a `level description` field value equal to `less than 3 feet` or a `water_level` field value less than 2.5. |level description| location | time |water\_level| |-----------------|-------------|------------------------|------------| | below 3 feet |coyote\_creek|2019-08-25T10:06:00.000Z| 2.398 | | below 3 feet |coyote\_creek|2019-08-25T10:12:00.000Z| 2.234 | | below 3 feet |coyote\_creek|2019-08-25T10:18:00.000Z| 2.064 | | below 3 feet |coyote\_creek|2019-08-25T10:24:00.000Z| 1.893 | #### Related * [Subqueries](/influxdb3/enterprise/reference/sql/subqueries/) | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | at or greater than 9 feet | coyote_creek | 2019-09-01T23:06:00.000Z | 9.8 | | at or greater than 9 feet | coyote_creek | 2019-09-01T23:12:00.000Z | 9.829 | | at or greater than 9 feet | coyote_creek | 2019-09-01T23:18:00.000Z | 9.862 | | at or greater than 9 feet | coyote_creek | 2019-09-01T23:24:00.000Z | 9.892 | | at or greater than 9 feet | coyote_creek | 2019-09-01T23:30:00.000Z | 9.902 | | at or greater than 9 feet | coyote_creek | 2019-09-01T23:36:00.000Z | 9.898 | | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | below 3 feet | santa_monica | 2019-09-01T00:00:00.000Z | 1.529 | | below 3 feet | santa_monica | 2019-09-01T00:06:00.000Z | 1.444 | | below 3 feet | santa_monica | 2019-09-01T00:12:00.000Z | 1.335 | | below 3 feet | santa_monica | 2019-09-01T00:18:00.000Z | 1.345 | | below 3 feet | santa_monica | 2019-09-01T00:24:00.000Z | 1.27 | | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | below 3 feet | santa_monica | 2019-08-19T12:00:00.000Z | 2.533 | | below 3 feet | santa_monica | 2019-08-19T12:06:00.000Z | 2.543 | | below 3 feet | santa_monica | 2019-08-19T12:12:00.000Z | 2.385 | | below 3 feet | santa_monica | 2019-08-19T12:18:00.000Z | 2.362 | | below 3 feet | santa_monica | 2019-08-19T12:24:00.000Z | 2.405 | | below 3 feet | santa_monica | 2019-08-19T12:30:00.000Z | 2.398 | | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | below 3 feet | santa_monica | 2019-08-18T12:00:00.000Z | 2.533 | | below 3 feet | santa_monica | 2019-08-18T12:06:00.000Z | 2.543 | | below 3 feet | santa_monica | 2019-08-18T12:12:00.000Z | 2.385 | | below 3 feet | santa_monica | 2019-08-18T12:18:00.000Z | 2.362 | | below 3 feet | santa_monica | 2019-08-18T12:24:00.000Z | 2.405 | | below 3 feet | santa_monica | 2019-08-18T12:30:00.000Z | 2.398 | | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | below 3 feet | santa_monica | 2019-08-18T18:00:00.000Z | 2.120 | | below 3 feet | santa_monica | 2019-08-18T18:06:00.000Z | 2.028 | | below 3 feet | santa_monica | 2019-08-18T18:12:00.000Z | 1.982 | | below 3 feet | santa_monica | 2019-08-19T06:00:00.000Z | 1.825 | | below 3 feet | santa_monica | 2019-08-19T06:06:00.000Z | 1.753 | | below 3 feet | santa_monica | 2019-08-19T06:12:00.000Z | 1.691 | | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | below 3 feet | santa_monica | 2019-08-12T00:00:00.000Z | 2.064 | | below 3 feet | santa_monica | 2019-08-14T09:30:00.000Z | 2.116 | | below 3 feet | santa_monica | 2019-08-16T15:45:00.000Z | 1.952 | | below 3 feet | santa_monica | 2019-08-18T12:00:00.000Z | 2.533 | | below 3 feet | santa_monica | 2019-08-18T18:00:00.000Z | 2.385 | | below 3 feet | santa_monica | 2019-08-19T10:30:00.000Z | 1.691 | | level description | location | time | water_level | | --- | --- | --- | --- | | level description | location | time | water_level | | below 3 feet | coyote_creek | 2019-08-25T10:06:00.000Z | 2.398 | | below 3 feet | coyote_creek | 2019-08-25T10:12:00.000Z | 2.234 | | below 3 feet | coyote_creek | 2019-08-25T10:18:00.000Z | 2.064 | | below 3 feet | coyote_creek | 2019-08-25T10:24:00.000Z | 1.893 | --- # Storage engine configuration reference Source: https://docs.influxdata.com/influxdb3/enterprise/reference/storage-engine-config-options/ For background on the upgraded storage engine, see[Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/). This page provides a complete reference for the upgraded storage engine configuration options. If an option is omitted, the server either derives a value from the existing`influxdb3 serve` configuration or falls back to an engine-specific default that balances resource usage and throughput. Most of the options on this page are tuning options that do not appear in the default `influxdb3 serve --help` output; use `--help-all` to list them. > [!Important] > #### The `pt-` option prefix was removed > > Upgraded storage engine options no longer use the `pt-` prefix—for example,`--pt-snapshot-size` is now `--snapshot-size`. > There is no backward compatibility for the old option names: > old `--pt-*` flags cause a startup error, and legacy `INFLUXDB3_PT_*` and`INFLUXDB3_ENTERPRISE_PT_*` environment variables are ignored. > > The server logs a warning at startup for each `INFLUXDB3_PT_*` or`INFLUXDB3_ENTERPRISE_PT_*` environment variable that is still set. > > The following options were folded into engine-agnostic options > shared with the Parquet-based engine: > > * `--pt-wal-flush-interval` → [`--wal-flush-interval`](/influxdb3/enterprise/reference/config-options/#wal-flush-interval) > * `--pt-wal-replication-interval` → [`--replication-interval`](/influxdb3/enterprise/reference/config-options/#replication-interval) > * `--pt-file-cache-size` → [`--file-cache-size`](/influxdb3/enterprise/reference/config-options/#file-cache-size) > * `--pt-file-cache-recency` → [`--file-cache-recency`](/influxdb3/enterprise/reference/config-options/#file-cache-recency) > * `--pt-disable-data-file-cache` → [`--disable-file-cache`](/influxdb3/enterprise/reference/config-options/#disable-file-cache) > > For the complete old-to-new name table, see[Migrate from pt- option names](#migrate-from-pt-option-names). > [!Note] > #### Runtime defaults and thread counts > > On InfluxDB 3 Enterprise 3.11+ clusters running the upgraded storage engine, the IO and DataFusion > runtimes each default to the licensed core count. > Thread counts set above the licensed core count are capped with a startup > warning. * [General](#general) * [WAL](#wal) * [Snapshot](#snapshot) * [Gen0](#gen0) * [File cache](#file-cache) * [Replication (query nodes)](#replication-query-nodes) * [Compactor](#compactor) * [L1-L4 level tuning](#l1-l4-level-tuning) * [Example configurations](#example-configurations) * [Downgrade options](#downgrade-options) * [Migrate from pt- option names](#migrate-from-pt-option-names) ## General | Option | Description | Default | |--------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| |[`--upgrade-pacha-tree`](/influxdb3/enterprise/reference/config-options/#upgrade-pacha-tree)| Migrate the cluster’s existing Parquet data to the upgraded storage engine. New clusters default to the upgraded engine and do not need this flag. | `false` | | `--engine-path-prefix` |Optional path prefix for all engine data (WAL and compaction generations). Max 32 characters. Must start and end with alphanumeric; inner characters allow `[a-zA-Z0-9._-]`. Shorter paths improve partitioning in object stores.| No prefix | | `--max-total-columns` | Maximum total columns across the entire instance (distinct from the per-table [`--num-total-columns-per-table-limit`](/influxdb3/enterprise/reference/config-options/#num-total-columns-per-table-limit)). Must be at least 2. |`10,000,000` (10M)| | `--enable-retention` | Enable retention enforcement. | `true` | | `--disable-hybrid-query` | Disable hybrid query mode. During and after the storage engine upgrade, queries normally merge results across both Parquet and `.pt` files. Set this flag to query only `.pt` data. | `false` | | `--enable-auto-dvc` | Enable automatic distinct value caching for `SHOW TAG VALUES` queries and the `tag_values()` SQL function. | Disabled | | `--auto-dvc-max-cardinality` | Maximum cardinality for auto-created distinct value caches. Requires `--enable-auto-dvc`. | `100000` | | `--auto-dvc-refresh-interval` | Background refresh interval for auto-created distinct value caches; minimum `1s`. Requires `--enable-auto-dvc`. | `10m` | | `--upgrade-poll-interval` | Polling interval for storage engine upgrade status monitoring. See [Upgrade from Parquet](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet). | `5s` | ### Engine path prefix Use a short prefix to improve partitioning in object stores: ```bash influxdb3 serve \ # ... --engine-path-prefix mydata ``` ### Hybrid query mode During and after the storage engine upgrade, hybrid query mode merges results across both Parquet and `.pt` files. Disable hybrid mode to query only `.pt` data: ```bash influxdb3 serve \ # ... --disable-hybrid-query ``` ## WAL Configure Write-Ahead Log (WAL) behavior for durability and performance. | Option | Description | Default | |--------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| |[`--wal-flush-interval`](/influxdb3/enterprise/reference/config-options/#wal-flush-interval)| Flush interval for the WAL. Shared with the Parquet-based engine (previously `--pt-wal-flush-interval`). | `1s` | | `--wal-flush-concurrency` | WAL flush concurrency. |`max(io_threads - 2, 2)`| | `--wal-buffer-size` |Maximum in-memory WAL buffer before a flush is triggered regardless of the flush interval. Increase this if WAL files are flushed before the interval elapses.| `15MB` | | `--wal-snapshots-to-keep` | Number of snapshot manifests worth of WAL history to retain. Must be greater than 0. | `5` | ### WAL buffer size The WAL buffer accumulates incoming writes before flushing to object storage. Larger buffers reduce flush frequency and produce larger WAL files, but increase memory usage: ```bash influxdb3 serve \ # ... --wal-buffer-size 30MB ``` ### Flush interval and concurrency Control how frequently the WAL flushes and how many workers run flushes in parallel: ```bash influxdb3 serve \ # ... --wal-flush-interval 2s \ --wal-flush-concurrency 8 ``` ### Write timestamp bounds Reject writes with explicit point timestamps outside an acceptable window around the time the server receives the request. Lines without an explicit timestamp are exempt, and the bounds don’t apply to the `_internal` database. | Option | Description | Default | |------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|--------------------| | [`--write-timestamp-max-past`](/influxdb3/enterprise/reference/config-options/#write-timestamp-max-past) |Reject explicit point timestamps older than this duration relative to the time the write request was received.|*Not set (disabled)*| |[`--write-timestamp-max-future`](/influxdb3/enterprise/reference/config-options/#write-timestamp-max-future)|Reject explicit point timestamps newer than this duration relative to the time the write request was received.|*Not set (disabled)*| ```bash influxdb3 serve \ # ... --write-timestamp-max-past 24h \ --write-timestamp-max-future 10m ``` ## Snapshot Configure snapshot buffer behavior, which controls how WAL files are merged into Gen0 files. | Option | Description | Default | |----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------| | `--snapshot-size` | Maximum size of the active snapshot bucket before it is rotated for snapshotting. | `250MB` | | `--snapshot-duration` |Time-based snapshot rotation trigger. Controls how often the ingester creates snapshots. Also used on query nodes as the bucket rotation interval for the replica buffer.| `10s` | |`--max-concurrent-snapshots`| Maximum number of concurrent snapshot operations before applying backpressure to writers. | `5` | | `--merge-threshold-size` | Maximum unmerged file size before triggering a merge operation. |`--snapshot-size` / 4 (62.5MB)| ### Snapshot size and duration Control when snapshot rotation triggers: ```bash influxdb3 serve \ # ... --snapshot-size 500MB \ --snapshot-duration 15s ``` ### Merge threshold Set the size threshold that triggers background merge operations. Lower values result in more frequent merges: ```bash influxdb3 serve \ # ... --merge-threshold-size 125MB ``` ## Gen0 Control the size of Gen0 files produced during merge operations. | Option | Description | Default | |--------------------------|--------------------------------------------------------|----------------| |`--gen0-max-rows-per-file`|Upper bound on rows per Gen0 file emitted during merge. |`10000000` (10M)| | `--gen0-max-file-size` |Upper bound on bytes per Gen0 file emitted during merge.| `100MB` | ### Gen0 file size limits Control the size of Gen0 files for query and compaction performance: ```bash influxdb3 serve \ # ... --gen0-max-rows-per-file 5000000 \ --gen0-max-file-size 50MB ``` ## File cache Configure data file caching for query performance. | Option | Description |Default| |--------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------| | [`--file-cache-size`](/influxdb3/enterprise/reference/config-options/#file-cache-size) |Total data file cache budget, shared with the Parquet-based engine. During the storage engine upgrade with hybrid query enabled, the budget is split 50/50 between the two engine caches; otherwise, the active engine receives the full budget. Provide a unit suffix or percentage (bare numbers are rejected). Set to `0b` on dedicated ingest nodes.| `20%` | |[`--disable-file-cache`](/influxdb3/enterprise/reference/config-options/#disable-file-cache)| Disable data file caching in both engines. Set to `true` on dedicated ingest nodes. |`false`| |[`--file-cache-recency`](/influxdb3/enterprise/reference/config-options/#file-cache-recency)| Only cache files newer than this age. Pre-caching on all-in-one and query nodes is based on this value. Shared with the Parquet-based engine. | `3d` | | `--file-cache-evict-after` | Evict cached files that have not been read within this duration. | `24h` | > [!Note] > #### Dedicated ingest nodes > > On dedicated ingest nodes (`--mode ingest`), disable the data file cache to avoid > wasting memory on data that ingest nodes never query. > Set `--file-cache-size 0b` or use `--disable-file-cache`. > These options must be explicitly set—they are not applied automatically when`--mode ingest` is used. > See [Disable caching on ingest nodes](#disable-caching-on-ingest-nodes) for an example. > [!Important] > #### In v3.11.6+: `--file-cache-size` is a hard limit > > In v3.11.6 and later, the upgraded storage engine enforces a hard cache limit. > The budget includes cached files and files open in running queries. When file access would exceed the budget, InfluxDB 3 Enterprise fails the query. The error identifies whether the budget is exhausted or a file exceeds the total budget: ```text file access cache budget exhausted: ``` ```text file larger than the file access cache budget: ``` Flight (gRPC) clients receive a `RESOURCE_EXHAUSTED` error. For HTTP queries in CSV, JSON, or JSON Lines format: * Before v3.12.1, the response returns HTTP 200 and ends early. * In v3.12.1 and later, a failure before the first row returns HTTP 429. A failure after the response starts still ends the response early. Retry the query after other queries finish. Set the cache size to accommodate the files accessed by concurrent queries. ### File cache size Set the maximum size for the data file cache: ```bash influxdb3 serve \ # ... --file-cache-size 8GB ``` ### Cache recency filter Only cache files containing data within a recent time window: ```bash influxdb3 serve \ # ... --file-cache-recency 24h ``` ### Disable caching on ingest nodes For dedicated ingest nodes, disable the data file cache to save memory: ```bash influxdb3 serve \ # ... --mode ingest \ --disable-file-cache ``` ## Replication (query nodes) Configure replication behavior for query nodes in distributed deployments. | Option | Description | Default | |------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------| |[`--replication-interval`](/influxdb3/enterprise/reference/config-options/#replication-interval)|Polling interval to check for new WAL files to replicate from ingest nodes. Shared with the Parquet-based engine (previously `--pt-wal-replication-interval`).| `250ms` | | `--wal-replica-recovery-concurrency` | Number of concurrent WAL file fetches during replica recovery or catchup. | `8` | | `--wal-replica-steady-concurrency` | Number of concurrent WAL file fetches during steady-state replication. | `8` | | `--wal-replica-queue-length` | Length of the queue between WAL file fetching and replica buffer merging (previously `--pt-wal-replica-queue-size`). | `100` | | `--wal-replica-recovery-tail-skip-limit` | Number of consecutive missing WAL files before stopping replica recovery. | `128` | | `--replica-gen0-load-concurrency` | Limit on the number of Gen0 files loaded concurrently when the replica starts. | `16` | | `--replica-max-buffer-size` | Maximum replica buffer size. Requires a unit suffix or percentage. Used by query nodes to store WAL files replicated from ingest nodes. |50% of available memory, capped at 16GB| ### Recovery concurrency Control parallelism during query node recovery or catchup: ```bash influxdb3 serve \ # ... --mode query \ --wal-replica-recovery-concurrency 16 ``` ### Steady-state replication Configure ongoing replication performance: ```bash influxdb3 serve \ # ... --mode query \ --wal-replica-steady-concurrency 4 \ --wal-replica-queue-length 200 ``` ### Replica buffer size Control the maximum buffer size for replicated data on query nodes: ```bash influxdb3 serve \ # ... --mode query \ --replica-max-buffer-size 8GB ``` ## Compactor Configure background compaction behavior. The compactor organizes data into fixed 24-hour UTC windows and progresses data through four compaction levels (L1 through L4). | Option | Description | Default | |-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------| | `--shard-count` | Target number of shards per compaction window. | `1` | |`--compactor-input-size-budget`| Maximum total input bytes across all active compaction jobs. Acts as an admission control budget for the compactor scheduler. | 50% of system memory at startup | |`--compactor-import-batch-size`|Largest batch of staged data that one bulk import merge reads at once. Still capped at one quarter of `--compactor-input-size-budget`. Available in v3.12.1 and later.| `4gb` | | `--final-compaction-age` | Age threshold for final compaction. When all L1-L3 run sets in a window are older than this, a final compaction merges everything into L4. |`4h` (time-disjoint layout); `72h` (legacy layout)| |`--compactor-cleanup-cooldown` | Cooldown after checkpoint publish before replaced files can be cleaned up. | `10min` | > [!Note] > **InfluxDB 3.10**: The `--pt-partition-count` option was renamed to`--pt-shard-count`; the option is now named `--shard-count`. ### Distributed compaction InfluxDB 3 Enterprise 3.12 adds beta support for running compaction across multiple nodes instead of only on the compactor primary. For an overview of the primary/worker model and how to enable it, see[Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/). | Option | Description |Default| |--------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|-------| | [`--compactor-dispatch-target`](/influxdb3/enterprise/reference/config-options/#compactor-dispatch-target) | Where compactions run: `local` (the primary only), `remote` (other `compact`-mode nodes only), or `all` (both). Beta. |`local`| |[`--compactor-dispatch-ack-timeout`](/influxdb3/enterprise/reference/config-options/#compactor-dispatch-ack-timeout)|How long a remote output-group dispatch can sit unacknowledged before the worker is declared unresponsive and its work redispatches elsewhere.| `5s` | Workers are nodes running in `compact` mode that advertise an internode address with `--internode-bind-addr` (and [`--conn-info`](/influxdb3/enterprise/reference/config-options/#conn-info)if needed). Set `--compactor-dispatch-target` to the same value on every compact node; the primary reads its own value to decide where work goes. ```bash influxdb3 serve \ # ... --mode compact \ --compactor-dispatch-target remote ``` ### Orphan sweep The compactor’s primary node periodically sweeps its output prefixes for orphaned files (objects that no published state references) and records them in a durable audit trail. | Option | Description |Default| |----------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|-------| | [`--compactor-sweep-interval`](/influxdb3/enterprise/reference/config-options/#compactor-sweep-interval) | Orphan-sweep cadence. Set to `off` to disable the scheduled sweep. | `7d` | | [`--compactor-sweep-mode`](/influxdb3/enterprise/reference/config-options/#compactor-sweep-mode) | `armed` enqueues confirmed orphans for deletion; `dry-run` only records them in the audit trail. |`armed`| | [`--compactor-sweep-grace`](/influxdb3/enterprise/reference/config-options/#compactor-sweep-grace) |Objects modified within this window before a pass’s capture instant are never deletion candidates. Values below `6h` are raised to `6h`.| `24h` | | [`--compactor-sweep-auto-resume`](/influxdb3/enterprise/reference/config-options/#compactor-sweep-auto-resume) | Resume an unfinished scheduled sweep pass after a restart instead of waiting a full interval. |`true` | |[`--compactor-sweep-audit-retention`](/influxdb3/enterprise/reference/config-options/#compactor-sweep-audit-retention)| Retention for the sweep’s audit-trail objects, expired at the start of each pass. | `7d` | > [!Warning] > #### Keep `--shard-count` at 1 > > In InfluxDB 3.10, running the storage engine upgrade with more than one shard > (`--shard-count` greater than `1`) can cause data loss and a bootstrap > deadlock. Leave `--shard-count` at its default value of `1`. ### Compaction budget Control total memory allocated to active compaction jobs: ```bash influxdb3 serve \ # ... --compactor-input-size-budget 8GB ``` ### Final compaction age Control when windows receive their final compaction into L4: ```bash influxdb3 serve \ # ... --final-compaction-age 48h ``` ### Deferred snapshot recovery When a snapshot fails to compact, the compactor defers it and records it in`system.pt_compaction_deferred_snapshots`.`--recover-deferred-snapshots` returns deferred snapshots to compaction, but only while every ingest node has fewer snapshots in flight than the recovery in-flight limit. | Option | Description |Default| |----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------| |`--recover-deferred-snapshots-in-flight-limit`|Number of snapshots an ingest node can have in flight before deferred snapshot recovery pauses. Valid values are `1` to `4096`, validated at startup. Applies to recovery only–compaction scheduling keeps the built-in limit of three.| `3` | To set the limit through the environment, use`INFLUXDB3_RECOVER_DEFERRED_SNAPSHOTS_IN_FLIGHT_LIMIT`. If your ingest nodes snapshot faster than compaction completes batches, no node drops below the default limit, so recovery never runs and`deferred_snapshot_count` in[`system.pt_compaction_ingest_nodes`](/influxdb3/enterprise/admin/query-system-data/#query-system-tables)doesn’t fall. Raise the limit to let recovery proceed: ```bash influxdb3 serve \ # ... --recover-deferred-snapshots \ --recover-deferred-snapshots-in-flight-limit 32 ``` A higher limit increases snapshot lag while recovery runs. Discovery lookahead and per-cycle feed limits are unchanged, so recovery can’t overwhelm the compaction pipeline. ## L1-L4 level tuning For what these levels mean and how the time-disjoint compaction model uses them, see[Upgraded storage engine compaction](/influxdb3/enterprise/reference/internals/durability/#upgraded-storage-engine-compaction). These per-level options apply within whichever compaction layout a cluster is running. Clusters using time-disjoint two-level compaction and clusters still on the legacy four-level layout both use the same L1-L4 tunables—the layouts organize the levels differently, but the options below configure each level the same way in either case. |Level | Role |Default tail target|Default file size|Promotion trigger| |------|--------------------------|-------------------|-----------------|-----------------| |**L1**| Ingest landing zone | 600MB | 25MB | 3 run sets | |**L2**| First promotion tier | 1.2GB | 40MB | 3 run sets | |**L3**| Second promotion tier | 2.5GB | 75MB | 4 run sets | |**L4**|Terminal (fully compacted)| 50GB | 125MB | N/A | ### L1 options | Option | Description |Default| |-----------------------|----------------------------------------------------|-------| |`--l1-tail-target-size`| L1 tail run set target size. |`600MB`| |`--l1-target-file-size`| L1 target file size. |`25MB` | |`--l1-promotion-count` |Number of L1 run sets that triggers promotion to L2.| `3` | ### L2 options | Option | Description |Default| |-----------------------|----------------------------------------------------|-------| |`--l2-tail-target-size`| L2 tail run set target size. |`1.2GB`| |`--l2-target-file-size`| L2 target file size. |`40MB` | |`--l2-promotion-count` |Number of L2 run sets that triggers promotion to L3.| `3` | ### L3 options | Option | Description |Default| |-----------------------|----------------------------------------------------|-------| |`--l3-tail-target-size`| L3 tail run set target size. |`2.5GB`| |`--l3-target-file-size`| L3 target file size. |`75MB` | |`--l3-promotion-count` |Number of L3 run sets that triggers promotion to L4.| `4` | ### L4 options | Option | Description |Default| |-----------------------|----------------------------|-------| |`--l4-tail-target-size`|L4 tail run set target size.|`50GB` | |`--l4-target-file-size`| L4 target file size. |`125MB`| ## Example configurations ### Development (minimal resources) ```bash influxdb3 serve \ --node-id dev01 \ --cluster-id dev-cluster \ --object-store file \ --data-dir ~/.influxdb3 \ --num-io-threads 2 \ --file-cache-size 512MB \ --wal-buffer-size 5MB \ --snapshot-size 100MB ``` ### Production all-in-one (8 cores, 32 GB RAM) ```bash influxdb3 serve \ --node-id prod01 \ --cluster-id prod-cluster \ --object-store s3 \ --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --num-io-threads 8 \ --file-cache-size 8GB \ --wal-buffer-size 30MB \ --snapshot-size 500MB \ --wal-flush-concurrency 4 ``` ### High-throughput ingest node (16 cores) ```bash influxdb3 serve \ --node-id ingest01 \ --cluster-id prod-cluster \ --object-store s3 \ --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --mode ingest \ --num-io-threads 16 \ --wal-buffer-size 50MB \ --wal-flush-interval 2s \ --wal-flush-concurrency 8 \ --snapshot-size 1GB \ --disable-file-cache ``` ### Query-optimized node (16 cores) ```bash influxdb3 serve \ --node-id query01 \ --cluster-id prod-cluster \ --object-store s3 \ --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --mode query \ --num-io-threads 16 \ --file-cache-size 16GB \ --file-cache-recency 24h \ --replica-max-buffer-size 8GB ``` ### Dedicated compactor (8 cores) ```bash influxdb3 serve \ --node-id compact01 \ --cluster-id prod-cluster \ --object-store s3 \ --bucket S3_BUCKET \ --aws-access-key-id AWS_ACCESS_KEY_ID \ --aws-secret-access-key AWS_SECRET_ACCESS_KEY \ --mode compact \ --num-io-threads 8 \ --compactor-input-size-budget 12GB ``` ## Downgrade options The `influxdb3 downgrade-to-parquet` command reverts a cluster from the upgraded storage engine back to standard Parquet storage. For the downgrade procedure, see[Downgrade to Parquet](/influxdb3/enterprise/reference/internals/storage-engine/#downgrade-to-parquet). | Option | Description | |------------------|--------------------------------------------------------------------------------------------------------------------| | `--cluster-id` | *(Required)* Cluster identifier. | | `--object-store` | *(Required)* Object storage type (`file`, `s3`, `gcs`, `azure`). | | `--data-dir` | Location of data files for a local (`file`) object store. | | `--bucket` | Object store bucket name (for `s3`, `gcs`, `azure`). | | `--dry-run` | Preview mode–list files that would be deleted without making changes. | | `--yes` | Skip the confirmation prompt. | |`--ignore-running`|Proceed even if nodes appear to be running. **Warning:** may cause data inconsistency if nodes are actively writing.| ## Migrate from pt- option names Use the following table to update startup scripts, Helm values, or systemd environment files that use the removed `--pt-*` option names. There are no aliases: old `--pt-*` flags cause a startup error, and legacy`INFLUXDB3_PT_*` and `INFLUXDB3_ENTERPRISE_PT_*` environment variables are ignored (the server logs a warning at startup for each one that is still set). Environment variable names follow the option names—for example,`INFLUXDB3_PT_SNAPSHOT_SIZE` becomes `INFLUXDB3_SNAPSHOT_SIZE`. Any `--pt-*` option not listed below drops the `pt-` prefix without any other change to its name. | Old name | New name | |-------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------| | `--pt-compactor-cleanup-cooldown` | `--compactor-cleanup-cooldown` | | `--pt-compactor-input-size-budget` | `--compactor-input-size-budget` | | `--pt-disable-data-file-cache` | [`--disable-file-cache`](/influxdb3/enterprise/reference/config-options/#disable-file-cache) (shared with the Parquet-based engine) | | `--pt-disable-hybrid-query` | `--disable-hybrid-query` | | `--pt-enable-auto-dvc` | `--enable-auto-dvc` | | `--pt-enable-retention` | `--enable-retention` | | `--pt-engine-path-prefix` | `--engine-path-prefix` | | `--pt-file-cache-evict-after` | `--file-cache-evict-after` | | `--pt-file-cache-recency` | [`--file-cache-recency`](/influxdb3/enterprise/reference/config-options/#file-cache-recency) (shared with the Parquet-based engine) | | `--pt-file-cache-size` | [`--file-cache-size`](/influxdb3/enterprise/reference/config-options/#file-cache-size) (shared with the Parquet-based engine) | | `--pt-final-compaction-age` | `--final-compaction-age` | | `--pt-gen0-max-bytes-per-file` | `--gen0-max-file-size` | | `--pt-gen0-max-rows-per-file` | `--gen0-max-rows-per-file` | | `--pt-l1-promotion-count` | `--l1-promotion-count` | | `--pt-l1-tail-target-bytes` | `--l1-tail-target-size` | | `--pt-l1-target-file-bytes` | `--l1-target-file-size` | | `--pt-l2-promotion-count` | `--l2-promotion-count` | | `--pt-l2-tail-target-bytes` | `--l2-tail-target-size` | | `--pt-l2-target-file-bytes` | `--l2-target-file-size` | | `--pt-l3-promotion-count` | `--l3-promotion-count` | | `--pt-l3-tail-target-bytes` | `--l3-tail-target-size` | | `--pt-l3-target-file-bytes` | `--l3-target-file-size` | | `--pt-l4-tail-target-bytes` | `--l4-tail-target-size` | | `--pt-l4-target-file-bytes` | `--l4-target-file-size` | | `--pt-max-columns` | `--max-total-columns` | | `--pt-max-concurrent-snapshots` | `--max-concurrent-snapshots` | | `--pt-merge-threshold-size` | `--merge-threshold-size` | | `--pt-replica-gen0-load-concurrency` | `--replica-gen0-load-concurrency` | | `--pt-replica-max-buffer-size` | `--replica-max-buffer-size` | | `--pt-row-delete-min-age` | `--row-delete-min-age` | | `--pt-shard-count` | `--shard-count` | | `--pt-snapshot-duration` | `--snapshot-duration` | | `--pt-snapshot-size` | `--snapshot-size` | | `--pt-upgrade-poll-interval` | `--upgrade-poll-interval` | | `--pt-wal-flush-concurrency` | `--wal-flush-concurrency` | | `--pt-wal-flush-interval` | [`--wal-flush-interval`](/influxdb3/enterprise/reference/config-options/#wal-flush-interval) (shared with the Parquet-based engine) | | `--pt-wal-max-buffer-size` | `--wal-buffer-size` | | `--pt-wal-replica-queue-size` | `--wal-replica-queue-length` | | `--pt-wal-replica-recovery-concurrency` | `--wal-replica-recovery-concurrency` | |`--pt-wal-replica-recovery-tail-skip-limit`| `--wal-replica-recovery-tail-skip-limit` | | `--pt-wal-replica-steady-concurrency` | `--wal-replica-steady-concurrency` | | `--pt-wal-replication-interval` |[`--replication-interval`](/influxdb3/enterprise/reference/config-options/#replication-interval) (shared with the Parquet-based engine)| | `--pt-wal-snapshots-to-keep` | `--wal-snapshots-to-keep` | #### Related * [Storage engine](/influxdb3/enterprise/reference/internals/storage-engine/) * [Query system data](/influxdb3/enterprise/admin/query-system-data/) * [InfluxDB 3 Enterprise configuration options](/influxdb3/enterprise/reference/config-options/) * [Performance tuning](/influxdb3/enterprise/admin/performance-tuning/) * [Distributed compaction](/influxdb3/enterprise/admin/distributed-compaction/) [storage](/influxdb3/enterprise/tags/storage/)[configuration](/influxdb3/enterprise/tags/configuration/)[reference](/influxdb3/enterprise/tags/reference/) | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --upgrade-pacha-tree | Migrate the cluster’s existing Parquet data to the upgraded storage engine. New clusters default to the upgraded engine and do not need this flag. | false | | --engine-path-prefix | Optional path prefix for all engine data (WAL and compaction generations). Max 32 characters. Must start and end with alphanumeric; inner characters allow [a-zA-Z0-9._-] . Shorter paths improve partitioning in object stores. | No prefix | | --max-total-columns | Maximum total columns across the entire instance (distinct from the per-table --num-total-columns-per-table-limit ). Must be at least 2. | 10,000,000 (10M) | | --enable-retention | Enable retention enforcement. | true | | --disable-hybrid-query | Disable hybrid query mode. During and after the storage engine upgrade, queries normally merge results across both Parquet and .pt files. Set this flag to query only .pt data. | false | | --enable-auto-dvc | Enable automatic distinct value caching for SHOW TAG VALUES queries and the tag_values() SQL function. | Disabled | | --auto-dvc-max-cardinality | Maximum cardinality for auto-created distinct value caches. Requires --enable-auto-dvc . | 100000 | | --auto-dvc-refresh-interval | Background refresh interval for auto-created distinct value caches; minimum 1s . Requires --enable-auto-dvc . | 10m | | --upgrade-poll-interval | Polling interval for storage engine upgrade status monitoring. See Upgrade from Parquet . | 5s | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --wal-flush-interval | Flush interval for the WAL. Shared with the Parquet-based engine (previously --pt-wal-flush-interval ). | 1s | | --wal-flush-concurrency | WAL flush concurrency. | max(io_threads - 2, 2) | | --wal-buffer-size | Maximum in-memory WAL buffer before a flush is triggered regardless of the flush interval. Increase this if WAL files are flushed before the interval elapses. | 15MB | | --wal-snapshots-to-keep | Number of snapshot manifests worth of WAL history to retain. Must be greater than 0. | 5 | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --write-timestamp-max-past | Reject explicit point timestamps older than this duration relative to the time the write request was received. | Not set (disabled) | | --write-timestamp-max-future | Reject explicit point timestamps newer than this duration relative to the time the write request was received. | Not set (disabled) | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --snapshot-size | Maximum size of the active snapshot bucket before it is rotated for snapshotting. | 250MB | | --snapshot-duration | Time-based snapshot rotation trigger. Controls how often the ingester creates snapshots. Also used on query nodes as the bucket rotation interval for the replica buffer. | 10s | | --max-concurrent-snapshots | Maximum number of concurrent snapshot operations before applying backpressure to writers. | 5 | | --merge-threshold-size | Maximum unmerged file size before triggering a merge operation. | --snapshot-size / 4 (62.5MB) | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --gen0-max-rows-per-file | Upper bound on rows per Gen0 file emitted during merge. | 10000000 (10M) | | --gen0-max-file-size | Upper bound on bytes per Gen0 file emitted during merge. | 100MB | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --file-cache-size | Total data file cache budget, shared with the Parquet-based engine. During the storage engine upgrade with hybrid query enabled, the budget is split 50/50 between the two engine caches; otherwise, the active engine receives the full budget. Provide a unit suffix or percentage (bare numbers are rejected). Set to 0b on dedicated ingest nodes. | 20% | | --disable-file-cache | Disable data file caching in both engines. Set to true on dedicated ingest nodes. | false | | --file-cache-recency | Only cache files newer than this age. Pre-caching on all-in-one and query nodes is based on this value. Shared with the Parquet-based engine. | 3d | | --file-cache-evict-after | Evict cached files that have not been read within this duration. | 24h | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --replication-interval | Polling interval to check for new WAL files to replicate from ingest nodes. Shared with the Parquet-based engine (previously --pt-wal-replication-interval ). | 250ms | | --wal-replica-recovery-concurrency | Number of concurrent WAL file fetches during replica recovery or catchup. | 8 | | --wal-replica-steady-concurrency | Number of concurrent WAL file fetches during steady-state replication. | 8 | | --wal-replica-queue-length | Length of the queue between WAL file fetching and replica buffer merging (previously --pt-wal-replica-queue-size ). | 100 | | --wal-replica-recovery-tail-skip-limit | Number of consecutive missing WAL files before stopping replica recovery. | 128 | | --replica-gen0-load-concurrency | Limit on the number of Gen0 files loaded concurrently when the replica starts. | 16 | | --replica-max-buffer-size | Maximum replica buffer size. Requires a unit suffix or percentage. Used by query nodes to store WAL files replicated from ingest nodes. | 50% of available memory, capped at 16GB | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --shard-count | Target number of shards per compaction window. | 1 | | --compactor-input-size-budget | Maximum total input bytes across all active compaction jobs. Acts as an admission control budget for the compactor scheduler. | 50% of system memory at startup | | --compactor-import-batch-size | Largest batch of staged data that one bulk import merge reads at once. Still capped at one quarter of --compactor-input-size-budget . Available in v3.12.1 and later. | 4gb | | --final-compaction-age | Age threshold for final compaction. When all L1-L3 run sets in a window are older than this, a final compaction merges everything into L4. | 4h (time-disjoint layout); 72h (legacy layout) | | --compactor-cleanup-cooldown | Cooldown after checkpoint publish before replaced files can be cleaned up. | 10min | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --compactor-dispatch-target | Where compactions run: local (the primary only), remote (other compact -mode nodes only), or all (both). Beta. | local | | --compactor-dispatch-ack-timeout | How long a remote output-group dispatch can sit unacknowledged before the worker is declared unresponsive and its work redispatches elsewhere. | 5s | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --compactor-sweep-interval | Orphan-sweep cadence. Set to off to disable the scheduled sweep. | 7d | | --compactor-sweep-mode | armed enqueues confirmed orphans for deletion; dry-run only records them in the audit trail. | armed | | --compactor-sweep-grace | Objects modified within this window before a pass’s capture instant are never deletion candidates. Values below 6h are raised to 6h . | 24h | | --compactor-sweep-auto-resume | Resume an unfinished scheduled sweep pass after a restart instead of waiting a full interval. | true | | --compactor-sweep-audit-retention | Retention for the sweep’s audit-trail objects, expired at the start of each pass. | 7d | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --recover-deferred-snapshots-in-flight-limit | Number of snapshots an ingest node can have in flight before deferred snapshot recovery pauses. Valid values are 1 to 4096 , validated at startup. Applies to recovery only–compaction scheduling keeps the built-in limit of three. | 3 | | Level | Role | Default tail target | Default file size | Promotion trigger | | --- | --- | --- | --- | --- | | Level | Role | Default tail target | Default file size | Promotion trigger | | L1 | Ingest landing zone | 600MB | 25MB | 3 run sets | | L2 | First promotion tier | 1.2GB | 40MB | 3 run sets | | L3 | Second promotion tier | 2.5GB | 75MB | 4 run sets | | L4 | Terminal (fully compacted) | 50GB | 125MB | N/A | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --l1-tail-target-size | L1 tail run set target size. | 600MB | | --l1-target-file-size | L1 target file size. | 25MB | | --l1-promotion-count | Number of L1 run sets that triggers promotion to L2. | 3 | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --l2-tail-target-size | L2 tail run set target size. | 1.2GB | | --l2-target-file-size | L2 target file size. | 40MB | | --l2-promotion-count | Number of L2 run sets that triggers promotion to L3. | 3 | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --l3-tail-target-size | L3 tail run set target size. | 2.5GB | | --l3-target-file-size | L3 target file size. | 75MB | | --l3-promotion-count | Number of L3 run sets that triggers promotion to L4. | 4 | | Option | Description | Default | | --- | --- | --- | | Option | Description | Default | | --l4-tail-target-size | L4 tail run set target size. | 50GB | | --l4-target-file-size | L4 target file size. | 125MB | | Option | Description | | --- | --- | | Option | Description | | --cluster-id | (Required) Cluster identifier. | | --object-store | (Required) Object storage type ( file , s3 , gcs , azure ). | | --data-dir | Location of data files for a local ( file ) object store. | | --bucket | Object store bucket name (for s3 , gcs , azure ). | | --dry-run | Preview mode–list files that would be deleted without making changes. | | --yes | Skip the confirmation prompt. | | --ignore-running | Proceed even if nodes appear to be running. Warning: may cause data inconsistency if nodes are actively writing. | | Old name | New name | | --- | --- | | Old name | New name | | --pt-compactor-cleanup-cooldown | --compactor-cleanup-cooldown | | --pt-compactor-input-size-budget | --compactor-input-size-budget | | --pt-disable-data-file-cache | --disable-file-cache (shared with the Parquet-based engine) | | --pt-disable-hybrid-query | --disable-hybrid-query | | --pt-enable-auto-dvc | --enable-auto-dvc | | --pt-enable-retention | --enable-retention | | --pt-engine-path-prefix | --engine-path-prefix | | --pt-file-cache-evict-after | --file-cache-evict-after | | --pt-file-cache-recency | --file-cache-recency (shared with the Parquet-based engine) | | --pt-file-cache-size | --file-cache-size (shared with the Parquet-based engine) | | --pt-final-compaction-age | --final-compaction-age | | --pt-gen0-max-bytes-per-file | --gen0-max-file-size | | --pt-gen0-max-rows-per-file | --gen0-max-rows-per-file | | --pt-l1-promotion-count | --l1-promotion-count | | --pt-l1-tail-target-bytes | --l1-tail-target-size | | --pt-l1-target-file-bytes | --l1-target-file-size | | --pt-l2-promotion-count | --l2-promotion-count | | --pt-l2-tail-target-bytes | --l2-tail-target-size | | --pt-l2-target-file-bytes | --l2-target-file-size | | --pt-l3-promotion-count | --l3-promotion-count | | --pt-l3-tail-target-bytes | --l3-tail-target-size | | --pt-l3-target-file-bytes | --l3-target-file-size | | --pt-l4-tail-target-bytes | --l4-tail-target-size | | --pt-l4-target-file-bytes | --l4-target-file-size | | --pt-max-columns | --max-total-columns | | --pt-max-concurrent-snapshots | --max-concurrent-snapshots | | --pt-merge-threshold-size | --merge-threshold-size | | --pt-replica-gen0-load-concurrency | --replica-gen0-load-concurrency | | --pt-replica-max-buffer-size | --replica-max-buffer-size | | --pt-row-delete-min-age | --row-delete-min-age | | --pt-shard-count | --shard-count | | --pt-snapshot-duration | --snapshot-duration | | --pt-snapshot-size | --snapshot-size | | --pt-upgrade-poll-interval | --upgrade-poll-interval | | --pt-wal-flush-concurrency | --wal-flush-concurrency | | --pt-wal-flush-interval | --wal-flush-interval (shared with the Parquet-based engine) | | --pt-wal-max-buffer-size | --wal-buffer-size | | --pt-wal-replica-queue-size | --wal-replica-queue-length | | --pt-wal-replica-recovery-concurrency | --wal-replica-recovery-concurrency | | --pt-wal-replica-recovery-tail-skip-limit | --wal-replica-recovery-tail-skip-limit | | --pt-wal-replica-steady-concurrency | --wal-replica-steady-concurrency | | --pt-wal-replication-interval | --replication-interval (shared with the Parquet-based engine) | | --pt-wal-snapshots-to-keep | --wal-snapshots-to-keep | --- # Usage telemetry Source: https://docs.influxdata.com/influxdb3/enterprise/reference/telemetry/ InfluxData collects information, or *telemetry data*, about the usage of InfluxDB 3 Enterprise to help improve the product. Learn what data InfluxDB 3 Enterprise collects and sends to InfluxData, how it’s used, and how you can opt out. ## What data is collected InfluxDB 3 Enterprise collects the following telemetry data: ### System metrics * **CPU utilization**: Process-specific CPU usage * **Memory usage**: Process memory consumption in MB * **Cores**: Number of CPU cores in use * **OS**: Operating system information * **Version**: InfluxDB 3 Enterprise version * **Uptime**: Server uptime in seconds ### Write metrics * **Write requests**: Number of write operations * **Write lines**: Number of lines written * **Write bytes**: Amount of data written in MB ### Query metrics * **Query requests**: Number of query operations ### Storage metrics * **Parquet file count**: Number of Parquet files * **Parquet file size**: Total size of Parquet files in MB * **Parquet row count**: Total number of rows in Parquet files ### Processing engine metrics * **WAL triggers**: Write-Ahead Log trigger counts * **Schedule triggers**: Scheduled processing trigger counts * **Request triggers**: Request-based processing trigger counts ### Instance information * **Instance ID**: Unique identifier for the server instance * **Cluster UUID**: Unique identifier for the cluster * **Storage type**: Type of object storage being used * **Invocation**: How the server was started * **Product type**: “Enterprise” ## Collection frequency * **System metrics** (CPU, memory): Collected every 60 seconds * **Write and query metrics**: Collected per operation, rolled up every 60 seconds * **Storage and processing engine metrics**: Collected at snapshot time (when available) * **Instance information**: Static data collected once Telemetry data is transmitted once per hour. ## Disable telemetry To “opt-out” of collecting and sending InfluxDB 3 Enterprise telemetry data, include the `--disable-telemetry-upload` flag or set the `INFLUXDB3_DISABLE_TELEMETRY_UPLOAD` environment variable when starting InfluxDB 3 Enterprise. **Default:** `false` | influxdb3 flag | Environment variable | |----------------------------|------------------------------------| |`--disable-telemetry-upload`|`INFLUXDB3_DISABLE_TELEMETRY_UPLOAD`| #### Command line flag ```sh influxdb3 serve --disable-telemetry-upload ``` #### Environment variable ```sh export INFLUXDB3_DISABLE_TELEMETRY_UPLOAD=true ``` When telemetry is disabled, no usage data is collected or transmitted. ## Data handling The telemetry data is used by InfluxData to understand product usage patterns, improve product performance and reliability, prioritize feature development, and identify/resolve issues. No personally identifiable information (PII) is collected. ## Privacy and security All telemetry data is transmitted securely via HTTPS. No database contents, queries, or user data is collected; only operational metrics and system information is transmitted. All data collection follows InfluxData’s privacy policy. [telemetry](/influxdb3/enterprise/tags/telemetry/)[monitoring](/influxdb3/enterprise/tags/monitoring/)[metrics](/influxdb3/enterprise/tags/metrics/)[observability](/influxdb3/enterprise/tags/observability/) | influxdb3 flag | Environment variable | | --- | --- | | influxdb3 flag | Environment variable | | --disable-telemetry-upload | INFLUXDB3_DISABLE_TELEMETRY_UPLOAD | --- # InfluxDB 3 Enterprise release notes Source: https://docs.influxdata.com/influxdb3/enterprise/release-notes/ > [!Important] > #### The upgraded storage engine is now the default (3.11+) > > New clusters default to the upgraded storage engine (no opt-in flag > required). Existing Parquet clusters stay on Parquet until you run the > storage engine upgrade with `--upgrade-pacha-tree` (environment variable`INFLUXDB3_UPGRADE_PACHA_TREE`). The previous flag, `--use-pacha-tree`, > still works but is deprecated. > [!Note] > #### InfluxDB 3 Core and Enterprise relationship > > InfluxDB 3 Enterprise is a superset of InfluxDB 3 Core. > All updates to Core are automatically included in Enterprise. > The Enterprise sections below only list updates exclusive to Enterprise. ## v3.11.9 ### Core No adjustments in this release. Core remains on v3.11.8. ### Enterprise #### Bug fixes * **WAL plugins received values under the wrong column names**: This release brings the v3.12.2 fix to Enterprise 3.11. Rows passed to a WAL trigger’s plugin now use the column names of their own record batch. Previously, when a flush held batches with the same columns in a different order, later batches used the first batch’s column names. ## v3.12.2 ### Core #### Features * **`influxdb3 debug catalog repair`**: This offline command, added in v3.11.6, is now also available on 3.12.2. It repairs a catalog that fails to load because of a duplicate table name or a reused table or database ID. Stop every node before you run it with `--execute`. * **Processing engine Python updated to 3.13.18**: Picks up the security fixes released since 3.13.15. #### Bug fixes * **Queries that exceed memory limits returned `500`**: The HTTP query endpoints now return `429 Too Many Requests` when a query exhausts the DataFusion memory pool. Previously, they returned `500`. For CSV, JSON, and JSON Lines output, this applies when the query fails before it returns any rows. An error after the first rows still ends the HTTP `200` response early. Some Core queries still return `500`, such as a query on a table that runs out of memory while deduplicating data. Enterprise already mapped resource-exhaustion errors to `429`. * **Catalog deadlock when checking a deleted table**: Checking whether a table is deleted now takes the catalog lock once. Previously, a catalog change that arrived between its two reads could deadlock the catalog. * **WAL plugins received values under the wrong column names**: Rows passed to a WAL trigger’s plugin now use the column names of their own record batch. Previously, when a flush held batches with the same columns in a different order, later batches used the first batch’s column names. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Faster [file index](/influxdb3/enterprise/admin/file-index/) loading (Parquet engine)**: Loading a table’s file index, including during a storage engine upgrade, now sorts each posting list once. Previously, every generation re-sorted each list it touched, so a table with many generations loaded slowly and used more memory. * **Faster reads of compaction metadata (Parquet engine)**: Compaction and snapshot metadata files are now parsed in one pass, including during a storage engine upgrade. Previously, each was parsed twice. * **Table index cache starts once the server is ready (Parquet engine)**: On ingest nodes, the table index cache now starts its background work when the server reports ready, instead of at the beginning of startup. * **Fewer file downloads per query (upgraded storage engine)**: A query now downloads only the run set files whose key range covers the queried field family. Previously, it also downloaded files that held only the table’s other field families. * **Faster merging of overlapping files (upgraded storage engine)**: When a query merges rows from several files, each output column is now built in one vectorized step instead of row by row. #### Bug fixes * **Storage engine upgrade completed with unconverted Parquet files**: If a Parquet file fails every conversion attempt, the ingest node now reports `failed` in `system.upgrade_parquet_node`. The `last_message` column identifies the file and its error. The cluster upgrade also fails. To resume the upgrade, follow the [retry procedure](/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/#retry-after-an-ingest-conversion-failure). Previously, the node dropped the file and reported its upgrade as complete. * **Storage engine upgrade stalled after an ingest node was removed**: The upgrade now drops a removed ingest node from its roster once its Parquet files are converted or absorbed. Previously, a removed ingest node held the cluster at `upgrading` forever. * **Overwrites lost after a query node restart (upgraded storage engine)**: After a restart, a query node now reads gen0 files in the order they were written, so the latest write to a series and timestamp wins. Previously, a query could return the value from before an overwrite until compaction. * **Query node could miss writes from a stopping ingest node (upgraded storage engine)**: A query node now keeps following an ingest node that is [`stopping`](/influxdb3/enterprise/admin/node-lifecycle/) and fetches its remaining WAL files after it acknowledges the stop. Previously, the query node could stop fetching before the node flushed its last writes. * **Trailing WAL fetch could hang after an ingest node stopped (upgraded storage engine)**: A query node’s fetch of a stopped ingest node’s remaining WAL files now completes. Previously, it could wait forever on files its workers had already fetched. * **Nodes stopped applying compaction updates after falling behind (Parquet engine)**: A node that refreshes its view of compacted data no longer reads the detail of each generation the compaction removed. Previously, a node more than [`--compaction-cleanup-wait`](/influxdb3/enterprise/reference/config-options/#compaction-cleanup-wait) behind the compactor failed every refresh with `generation detail not found` until it restarted. * **Retention stopped after Gen1 file cleanup deleted a snapshot (Parquet engine)**: The table index cache now skips a snapshot that [Gen1 file cleanup](/influxdb3/enterprise/admin/gen1-file-cleanup/) already deleted. Previously, the missing snapshot failed the cache initialization, and retention didn’t run until the next restart. * **Restarted node listed as stopped by its previous process**: A process that is shutting down no longer marks its node stopped once another process has registered under the same `--node-id`. Previously, a slow shutdown could mark the running replacement as stopped, and on the upgraded storage engine query nodes stopped following its WAL. * **Schema changes lost when upgrading a Core node to Enterprise**: [Upgrading from Core](/influxdb3/enterprise/admin/upgrade-from-core/) now promotes the node’s live catalog. Previously, if the Core node had first run a release before 3.10, the upgrade promoted a stale catalog checkpoint, so schema changes made since then were missing. * Other bug fixes and performance improvements ## v3.11.8 ### Core #### Features * **Processing engine Python updated to 3.13.18**: Picks up the security fixes released since 3.13.15. #### Bug fixes * **Queries that exceed memory limits returned `500`**: The HTTP query endpoints now map DataFusion memory-pool exhaustion to `429 Too Many Requests` instead of `500`. If the error occurs during query execution, `pretty` and `parquet` output return `429`. For CSV, JSON, and JSON Lines output, execution errors still end the HTTP `200` response early, even before the first rows. Some Core queries still return `500`, such as a query on a table that runs out of memory while deduplicating data. Enterprise already mapped resource-exhaustion errors to `429`. * **Catalog deadlock when checking a deleted table**: Checking whether a table is deleted now takes the catalog lock once. Previously, a catalog change that arrived between its two reads could deadlock the catalog. * **WAL plugins received values under the wrong column names**: Rows passed to a WAL trigger’s plugin now use the column names of their own record batch. Previously, when a flush held batches with the same columns in a different order, later batches used the first batch’s column names. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Faster [file index](/influxdb3/enterprise/admin/file-index/) loading (Parquet engine)**: Loading a table’s file index, including during a storage engine upgrade, now sorts each posting list once. Previously, every generation re-sorted each list it touched, so a table with many generations loaded slowly and used more memory. * **Faster reads of compaction metadata (Parquet engine)**: Compaction and snapshot metadata files are now parsed in one pass, including during a storage engine upgrade. Previously, each was parsed twice. * **Table index cache starts once the server is ready (Parquet engine)**: On ingest nodes, the table index cache now starts its background work when the server reports ready, instead of at the beginning of startup. #### Bug fixes * **Storage engine upgrade completed with unconverted Parquet files**: If a Parquet file fails every conversion attempt, the ingest node now reports `failed` in `system.upgrade_parquet_node`. The `last_message` column identifies the file and its error. The cluster upgrade also fails. To resume the upgrade, follow the [retry procedure](/influxdb3/enterprise/reference/cli/influxdb3/manage/retry-upgrade-to-pacha-tree/#retry-after-an-ingest-conversion-failure). Previously, the node dropped the file and reported its upgrade as complete. * **Storage engine upgrade stalled after an ingest node was removed**: The upgrade now drops a removed ingest node from its roster once its Parquet files are converted or absorbed. Previously, a removed ingest node held the cluster at `upgrading` forever. * **Overwrites lost after a query node restart (upgraded storage engine)**: After a restart, a query node now reads gen0 files in the order they were written, so the latest write to a series and timestamp wins. Previously, a query could return the value from before an overwrite until compaction. * **Nodes stopped applying compaction updates after falling behind (Parquet engine)**: A node that refreshes its view of compacted data no longer reads the detail of each generation the compaction removed. Previously, a node more than [`--compaction-cleanup-wait`](/influxdb3/enterprise/reference/config-options/#compaction-cleanup-wait) behind the compactor failed every refresh with `generation detail not found` until it restarted. * **Retention stopped after Gen1 file cleanup deleted a snapshot (Parquet engine)**: The table index cache now skips a snapshot that [Gen1 file cleanup](/influxdb3/enterprise/admin/gen1-file-cleanup/) already deleted. Previously, the missing snapshot failed the cache initialization, and retention didn’t run until the next restart. * **Restarted node listed as stopped by its previous process**: A process that is shutting down no longer marks its node stopped once another process has registered under the same `--node-id`. Previously, a slow shutdown could mark the running replacement as stopped, and on the upgraded storage engine query nodes stopped following its WAL. * Other bug fixes and performance improvements ## v3.12.1 ### Core #### Features * **Faster startup after large file removals**: At startup, the files that snapshots and checkpoints mark as removed are now dropped from each table’s file list in one pass. Previously, every removed file scanned the table’s whole file list, so startup slowed sharply after many files were removed from a large table. #### Bug fixes * **Catalog failed to load after upgrading to v3.12.0**: The server now loads a catalog in which a table has more than one field family with the same automatically generated name. Core releases before v3.10.0 and Enterprise releases before v3.9.0 could write such names when fields were added to an existing table. v3.12.0 rejected them, so a node with an affected catalog failed to start. * **Acknowledged writes lost after a forced snapshot and a restart**: WAL flushes now run one at a time. Previously, a forced Parquet snapshot, triggered by write buffer memory or [`--force-snapshot-max-age`](/influxdb3/enterprise/reference/config-options/#force-snapshot-max-age), could run while an earlier WAL file was still being written, and recorded that file as covered before its rows reached the buffer. If the node restarted before the next snapshot, it skipped that file and the acknowledged writes in it were lost. * **Overwrites lost across snapshots**: When two Parquet files hold the same time window because it was persisted across two snapshots, a point in the later file now wins over a point with the same series and timestamp in the earlier file. Previously, the earlier file could win, so a query returned the value from before the overwrite. * **Failed streaming queries returned `200`**: A query in CSV, JSON, or JSON Lines format that fails before it returns any rows, such as a sort that exceeds the memory limit, now returns the error status and message. Previously, the server sent `200` and then dropped the connection, so clients saw an empty body and a transport error. An error after the first rows still ends the response early. * **Writes near the maximum timestamp**: A line with a timestamp later than `2262-04-11T23:39:59.999999999Z` is now rejected with a `400` error. Previously, a timestamp that close to the maximum could panic the WAL flush when the server forced a snapshot. * **`system.parquet_files` listed files under the wrong tables**: A query on `system.parquet_files` without a `table_name` filter now lists only the tables in the current database. Previously, it could list the current database’s files under the names of tables in other databases that had the same table ID. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Larger bulk import batches (upgraded storage engine)**: The new [`--compactor-import-batch-size`](/influxdb3/enterprise/reference/storage-engine-config-options/#compactor) option sets the largest batch of staged data that one bulk import merge reads at once. The default is `4gb` and is configurable up to one quarter of `--compactor-input-size-budget`. This brings bulk import speeds to 1.5TB/hr on a standard 24xl, 10x faster than the 3.11 line. * **Faster ingest node startup with many snapshots (upgraded storage engine)**: An ingest node no longer reads every snapshot manifest in object storage at startup. Previously, it read them all, so startup time grew with the number of manifests. The first start on v3.12.1 still reads every manifest once. `system.pt_ingest_files` now lists the files from the newest manifest plus the snapshots taken since the node started. * **Faster compaction I/O (Parquet engine)**: The compactor now uploads a finished output file while it encodes the next one, and checks up to 8 input files at a time. Previously, it did both one file at a time. * **WAL flush wait metric (Parquet engine)**: The new `influxdb3_parquet_wal_flush_lock_wait_seconds` histogram reports how long a WAL flush waited for an earlier flush to finish. The `kind` label is `timed` for the regular flush interval and `forced` for a flush that forces a snapshot. #### Bug fixes * **Missing rows after changing a table’s [file index](/influxdb3/enterprise/admin/file-index/) columns (Parquet engine)**: The file index now records which columns each compacted generation indexed. A query that filters on a column that only some generations indexed scans the other generations instead of skipping them. Previously, after you added or removed file index columns, such a query could return incomplete results from compacted data. * **Overwrites reverted to the older value (upgraded storage engine)**: When the ingest buffer merges new writes into data it already merged for the same time window, a later write now wins over an earlier write to the same series and timestamp. Previously, the earlier value could replace the later one. * **Storage engine upgrade failed with more than one compact node**: During an upgrade to the upgraded storage engine, only the compact node that holds the compactor lease now drives the upgrade. Previously, every compact node drove it at the same time, their progress updates conflicted, and the upgrade could be marked as failed. * **Compactor replanned forever against a missing data file (upgraded storage engine)**: A run set whose data file is confirmed gone from object storage is now skipped after three failed compaction attempts, so planning moves on to other work. Previously, the compactor planned the same compaction on every pass, and it failed every time. Skipped run sets are counted in `influxdb3_compactor_skipped_run_sets` with `reason="auto_data_file_not_found"`. Compactions of new snapshots and imports aren’t covered. * **Bulk import could stall object storage requests (upgraded storage engine)**: The server no longer leaves object storage requests unfinished while it checks the status of uploaded imports. Previously, during a credential refresh, one of those requests could hold the object store client’s credential lock and never release it, so every other request on the node waited behind it: the server stayed up but stopped reading from and writing to object storage. * **Nodes without query mode built an unused file index (Parquet engine)**: Only nodes that run in `query` mode, including `all`, now load the compacted data file index at startup. Previously, compact and process nodes also loaded it, which used startup time and memory for an index they never read. A node that skips the index logs `compacted-data file index not loaded`. * **Ingest node refused to restart after an unclean stop (Parquet engine)**: An ingest node that stopped without a clean shutdown now starts again. Previously, if a snapshot covered all of its WAL or a forced snapshot ran during startup, the node could fail every start with an error that it was `still running in another process`, though no other process existed. * Other bug fixes and performance improvements ## v3.11.7 ### Core #### Bug fixes * **Catalog failed to load after upgrading to v3.11.6**: The server now loads a catalog in which a table has more than one field family with the same automatically generated name. Core releases before v3.10.0 and Enterprise releases before v3.9.0 could write such names when fields were added to an existing table. v3.11.6 rejected them, so a node with an affected catalog failed to start. * **Acknowledged writes lost after a forced snapshot and a restart**: WAL flushes now run one at a time. Previously, a forced Parquet snapshot, triggered by write buffer memory or [`--force-snapshot-max-age`](/influxdb3/enterprise/reference/config-options/#force-snapshot-max-age), could run while an earlier WAL file was still being written, and recorded that file as covered before its rows reached the buffer. If the node restarted before the next snapshot, it skipped that file and the acknowledged writes in it were lost. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **WAL flush wait metric (Parquet engine)**: The new `influxdb3_parquet_wal_flush_lock_wait_seconds` histogram reports how long a WAL flush waited for an earlier flush to finish. The `kind` label is `timed` for the regular flush interval and `forced` for a flush that forces a snapshot. #### Bug fixes * **Two processes with the same `--node-id` (upgraded storage engine)**: When two ingest-capable processes run with the same `--node-id`, the one that loses a snapshot write now logs `another process is writing under this node id` and exits, so only one writer survives. * **Ingest node refused to restart after an unclean stop (Parquet engine)**: An ingest node that stopped without a clean shutdown now starts again. Previously, if a snapshot covered all of its WAL or a forced snapshot ran during startup, the node could fail every start with an error that it was `still running in another process`, though no other process existed. ## v3.11.6 ### Core #### Features * **`influxdb3 debug catalog repair`**: This new offline command repairs a catalog that fails to load because two tables in a database hold the same name, or because a table or database ID was used twice. Without `--execute`, the command is a dry run that reports what it would change. Stop every node before you run it with `--execute`. For a duplicate name, the repair renames the earlier table to `-orphaned-`, so data under both tables stays reachable. It backs up the previous catalog snapshot first. * **Faster startup with many snapshots**: Snapshot manifests are parsed in parallel at startup, on up to half of the available cores. On Enterprise, this applies to the Parquet engine. * **Faster startup after large file removals**: At startup, the files that snapshots and checkpoints mark as removed are now dropped from each table’s file list in one pass. Previously, every removed file scanned the table’s whole file list, so startup slowed sharply after many files were removed from a large table. Writing a snapshot checkpoint has the same improvement. On Enterprise, this applies to the Parquet engine. #### Bug fixes * **Catalog recorded a name or ID twice**: The catalog now rejects a new resource whose name or ID is already in use. Previously, it could record a second table under an existing name, which could panic a later restart. A catalog that holds such records, including one that loaded on v3.11.5, now fails to load with an `already exists` error. To repair it, use `influxdb3 debug catalog repair`. * **Panic on an oversized distinct value cache `max_age`**: A distinct value cache with a `max_age` of more than about 349 years now keeps its values with no age limit. Previously, eviction panicked and stopped for every distinct value cache on the node until a restart, and queries against that cache failed. * **One query on a large Parquet file could run a node out of memory**: A query now fetches each Parquet file once per scan. Previously, every partition of the scan fetched and buffered its own copy of the whole file, so one query against a 4.4 GB file could use about 70 GB. To restore the old behavior, set `--datafusion-config iox.share_cached_parquet_loader_fetches:false`. * **Repeated downloads under Parquet cache pressure**: The Parquet cache no longer evicts files that are still downloading, and no longer starts a second download of a file that is already being fetched. Previously, a cache under sustained pressure could download the same file over and over. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Compaction cycle summaries (Parquet engine)**: After each compaction cycle, the compactor logs a `compaction cycle finished` line with the outcome, the duration, the number of plans that were planned, succeeded, and failed, and the number of files compacted and produced. The line is logged at `INFO` when every plan succeeds and at `WARN` otherwise. * **Stuck compaction warning only for growing backlogs (Parquet engine)**: The `compaction not planned` report added in v3.11.5 is now logged at `WARN` only when the table’s backlog of uncompacted files grew since the last report. A backlog that isn’t growing, such as a table that no longer receives writes, is reported at `DEBUG`. * **Startup phase details**: Startup phases now log what they loaded, such as file counts and bytes, and the server logs a `startup phase summary` line with per-phase durations when it becomes ready. * **Faster compaction I/O (Parquet engine)**: The compactor now uploads a finished output file while it encodes the next one, and checks up to 8 input files at a time. Previously, it did both one file at a time. #### Bug fixes * **Missing rows from `IN` filters against a [file index](/influxdb3/enterprise/admin/file-index/) (Parquet engine)**: A filter with an `IN` list of four or more values on a column the file index doesn’t cover, such as a field, no longer skips every compacted file. Previously, such a query returned no rows from compacted data. An equality or `IN` filter could also skip every file after you narrowed a table’s file index columns. * **Missing rows after changing a table’s file index columns (Parquet engine)**: The file index now records which columns each compacted generation indexed. A query that filters on a column that only some generations indexed scans the other generations instead of skipping them. Previously, after you added or removed file index columns, such a query could return incomplete results from compacted data. * **Backfilled windows skipped by compaction (Parquet engine)**: The default for `--compaction-max-num-files-per-plan` is now `10000`. Previously, the default was `500`, and a compaction window with more input files than the limit was skipped on every cycle, so a table with a large backfill kept accumulating small files. A value you set explicitly still applies. * **Storage engine upgrade failed with more than one compact node**: During an upgrade to the upgraded storage engine, only the compact node that holds the compactor lease now drives the upgrade. Previously, every compact node drove it at the same time, their progress updates conflicted, and the upgrade could be marked as failed. * **Large compactions delayed new data (upgraded storage engine)**: One large background compaction is now held to about half of the compactor’s task slots and memory budget, so compaction of newly written snapshots keeps running beside it. Previously, a large compaction could take every slot, and snapshot compaction waited until it finished. * **Row deletes could remove the rows they were meant to keep (upgraded storage engine)**: A rewrite that applies a row delete keeps the files the delete doesn’t touch by their original ID. Previously, it also queued those files for deletion, so after the cleanup cooldown the rows the delete was meant to preserve were gone and later reads of the run set failed with `Object not found`. On the first start on v3.11.6, row-delete entries already in the persisted delete queue are dropped instead of replayed, and the objects they name stay in object storage. * **Rows lost when clearing the Parquet WAL after switching to the upgraded storage engine**: When a node starts on the upgraded storage engine with unflushed Parquet WAL, clearing that backlog no longer re-runs snapshots the previous process recorded but never persisted. Previously, a re-run could overwrite a manifest, and its Parquet file, so the rows it held were listed by no manifest. * **Backup and restore could hang (upgraded storage engine)**: The backup and restore copy loops now keep their in-flight copies running while they write progress. Previously, a copy that held the object store client’s credential-refresh lock was left unpolled during a progress write, which blocked every request on that client and the backup or restore never finished. * **Future-dated data displaced current data in the cache (upgraded storage engine)**: Cache preloading no longer warms time windows that start more than one window length in the future. Previously, every future window was warm, so data timestamped far in the future could use the preload budget before the current window loaded. * **Login response time revealed valid usernames**: A username and password login now takes the same time whether or not the username exists. Previously, a login for an unknown username returned faster, which revealed whether a username existed. This affects deployments with `--user-auth-type basic`. * **Queries that exceed memory limits returned 500**: With `pretty` or `parquet` output, the HTTP query endpoints now return `429 Too Many Requests` when a query fails because the DataFusion memory pool or the data file cache budget is exhausted. Previously, a memory pool failure returned `500`. Other output formats still return `200` and end the response early. Flight already returned `RESOURCE_EXHAUSTED`. * Other bug fixes and performance improvements #### Breaking changes * **Data file cache is a hard limit (upgraded storage engine)**: [`--file-cache-size`](/influxdb3/enterprise/reference/config-options/#file-cache-size) now also counts data file bytes held by running queries and compactions. When the budget is used up, the query that needs more fails with a `file access cache budget exhausted` error instead of the node using memory beyond the budget. The new `influxdb3_file_cache_admission_rejected_total` metric counts fetches refused by the budget. ## v3.12.0 InfluxDB 3.12 makes bulk import faster and adds distributed compaction (beta), per-database schema enforcement, and generally available user authentication and role-based access control to InfluxDB 3 Enterprise. Before you upgrade, review [Upgrade InfluxDB](/influxdb3/enterprise/admin/upgrade/). > [!Important] > #### Back up your catalog before you upgrade > > InfluxDB 3.12 adds a catalog record that v3.11.x can’t read. > After the upgrade, you can’t roll back to v3.11.x unless you have a catalog backup. > > The catalog records the new feature level automatically. > On a single node, this happens the first time you start v3.12. > In a cluster, it happens once every running node runs v3.12. > It happens whether or not you use the new features. > This applies to Core and Enterprise. > > After that, v3.11.x refuses to load the catalog and returns an error like the following: > > ``` > this node's feature level (core=, enterprise=) is below the cluster's committed level (core=, enterprise=); upgrade required > ``` > > Before you upgrade: > > 1. Back up everything under `{prefix}/catalog/`, including the catalog snapshot and logs under `catalog/v3/`. > 2. Keep the backup until you’re sure you won’t roll back. > > To roll back to v3.11.x, restore the backup. > For details, see [Before you upgrade](/influxdb3/enterprise/admin/upgrade/#before-you-upgrade). ### Core #### Features * **Add columns to an existing table**: `influxdb3 update table` now takes `--tags` (tag names) and `--fields` (`name:type`), and the new `PATCH /api/v3/configure/table` endpoint does the same over HTTP. You can only add columns. Adding a column that already exists with the same type does nothing, and a different type returns an error. * **Trigger defaults in the create trigger API**: `trigger_settings` and `disabled` are now optional when you create a trigger through `POST /api/v3/configure/processing_engine_trigger`. They default to `run_async=false`, `error_behavior=log`, and `disabled=false`. Previously, a request that omitted either field returned 400. * **Regular expressions as `IN` lists**: More anchored regular expressions, such as `^host-(01|02)$` or `^val[1-4]$`, are now rewritten as an `IN` list (or `NOT IN` for `!~`), up to 1,000 values, in SQL and InfluxQL. * **Narrower InfluxQL scans with `LIMIT`, `OFFSET`, and `fill(N)`**: These queries now read only the columns they reference. Previously, they read every column in the table, so queries on wide tables were slower than the same query without these clauses. * **Faster startup with many tables**: Replaying the catalog no longer copies a database’s whole schema for every table it creates or alters, so startup time no longer grows quadratically with the number of tables in a database. * **Faster startup with many snapshots**: Snapshot manifests are parsed in parallel at startup, on up to half of the available cores. On Enterprise, this applies to the Parquet engine. #### Bug fixes * **InfluxQL `SPREAD` with negative values**: `SPREAD` now returns the difference between the maximum and minimum when every value is negative. Previously, it returned the negated minimum. * **Packaged service failed to start on SELinux hosts**: On RHEL-family hosts with SELinux enforcing, the `systemd` unit from the `.deb` and `.rpm` packages exited with status `226`. No configuration change is needed. * **One query on a large Parquet file could run a node out of memory**: A query now fetches each Parquet file once per scan. Previously, every partition of the scan fetched and buffered its own copy of the whole file, so one query against a 4.4 GB file could use about 70 GB. To restore the old behavior, set `--datafusion-config iox.share_cached_parquet_loader_fetches:false`. * **Repeated downloads under Parquet cache pressure**: The Parquet cache no longer evicts files that are still downloading, and no longer starts a second download of a file that is already being fetched. Previously, a cache under sustained pressure could download the same file over and over. * Other bug fixes and performance improvements #### Breaking changes * **Finite default for `--max-concurrent-queries`**: The [`--max-concurrent-queries`](/influxdb3/enterprise/reference/config-options/#max-concurrent-queries) default is now the larger of 50 and 4 times the node’s query parallelism. Previously, the default was effectively unlimited. Queries over the limit wait for a slot instead of failing. An explicit value below 16, or below the node’s query parallelism, logs a startup warning. * **WAL buffer limit enforced**: [`--wal-max-buffered-writes`](/influxdb3/enterprise/reference/config-options/#wal-max-buffered-writes) (default `100000`) was never enforced. When the WAL buffer is full, writes now return `429 Too Many Requests`, so retry them after a short delay. On Enterprise, this applies to the Parquet engine. * **HTTP and gRPC request metrics split by protocol**: `http_requests*` metrics now count only HTTP requests, and `grpc_requests*` metrics count only gRPC requests. Previously, both counted every request, so dashboards that sum the two families will show lower values. The `path` and `method_path` labels now use route templates, such as `/api/v3/engine/:path`. Unrecognized paths are reported as `other`. Panels that filter on a literal path that includes an ID return no data. * **DataFusion 52.5**: The query engine is upgraded from DataFusion 51.0 to 52.5. `pow()` and `power()` with integer arguments now return a float. * **New system table columns**: `system.databases` adds a `schema_mode` column at the end. `system.nodes` adds `conn_info`, and `system.tables` adds `retention_period_ns`, both before existing columns, so the column order of `SELECT *` on those tables changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Distributed compaction (beta, upgraded storage engine)**: Compaction jobs can now run on every compact node in the cluster, not only on the node that holds the compactor lease. Set `--compactor-dispatch-target` to the same value on every compact node: * `local` (default): The lease holder runs all compaction, as in v3.11. * `all`: The lease holder and every other compact node run compaction. * `remote`: Only the other compact nodes run compaction. With no other compact node available, compaction waits. A compact node receives work only if it sets `--internode-bind-addr`. Use compact nodes of the same size. The lease holder plans for the smallest `--compactor-input-size-budget` and `--compactor-max-concurrent-merges` among them. If a node fails, its jobs are retried on another node. New `influxdb3_compactor_*` metrics report worker connections, dispatch activity, and which node holds the lease (`influxdb3_compactor_is_primary`). * **Processing Engine on dedicated process nodes**: Process nodes (`--mode process` or `--mode all`) now run triggers for the whole cluster: * Every process node follows the WAL of every ingest node through object storage, so WAL triggers fire on a process node for writes sent to any ingest node. Previously, a WAL trigger fired only on the node that received the write. A new process node starts from the present and doesn’t fire on existing WAL. * The trigger’s `--node-spec` now selects which process nodes’ schedulers own the trigger. Each owning scheduler spreads runs across itself and the other running process nodes that set `--internode-bind-addr`. A node that doesn’t have the plugin file in its `--plugin-dir` declines the run, and another node runs it. Keep plugin directories and Python packages the same on every process node. * Each process node saves its scheduler state to object storage every second and at shutdown. After a restart, it replays schedule ticks that came due while it was down, resumes following the WAL where it stopped, and keeps pending retries. Make sure your plugins handle a repeated run. Triggers that were disabled or deleted while the node was down stay that way. * New options: `--trigger-retry-max-attempts` (default `5`) sets the attempts before a failing invocation is discarded when the trigger uses `--error-behavior retry`, `--trigger-work-silence-timeout` (default `60s`) sets how long a scheduler waits on a silent node before giving up the run, and `--processing-engine-restart-state-snapshot-interval` (default `1s`) sets how often scheduler state is saved. * **Schema enforcement**: Create a database with `influxdb3 create database --schema-mode explicit` (or `"schema_mode": "explicit"` in the API) to reject writes to tables and columns you haven’t declared. Declare tables with `influxdb3 create table` and add columns with `influxdb3 update table --tags --fields`. Every write path enforces it, including the Processing Engine and bulk import, on both storage engines. The default, `implicit`, keeps schema-on-write. You can’t change a database’s schema mode after you create it. After you add columns through one node, other nodes can reject writes to them until their next catalog sync (`--catalog-sync-interval`, default `1s`). Core rejects `--schema-mode explicit` with a 400 error. * **Bulk import from object storage (upgraded storage engine)**: `influxdb3 import from-object-store` (and `POST /api/v3/enterprise/import`) imports Parquet files that are already in object storage, without streaming them through the client. The source is a prefix in the cluster’s own object store or an `s3://bucket/prefix` URL. The server reads the source with its own object store credentials and tries an S3 server-side copy first (`--import-attempt-server-side-copy`, default `true`), falling back to streaming. * **Faster bulk import uploads (upgraded storage engine)**: Uploaded files are converted into slices of up to 128 MiB that compaction processes in parallel, so large imports use more of the compactor’s cores. * **Database-scoped import permissions**: Creating an import now requires write permission on the target database instead of an admin token. Listing imports requires the `describe` action on at least one database and returns only the import jobs for databases the token can describe. For example, grant `db::describe,write` for an import job. * **Write timestamp bounds (upgraded storage engine)**: The new `--write-timestamp-max-past` and `--write-timestamp-max-future` options reject lines whose explicit timestamp is further from the request time than the given duration. Lines without a timestamp are accepted. Both are off unless set, apply to every database, and are read by each ingest node, so set the same values on every ingest node. Rejections are counted in `influxdb3_write_timestamp_rejections_total`. * **Faster `ORDER BY time ... LIMIT` queries (upgraded storage engine)**: SQL queries sorted by `time` with a `LIMIT` now read time windows in sort order and stop once they have enough rows. Previously, they scanned and sorted every window in the queried time range. * **User authentication and role-based access control (GA)**: Username and password sign-in, single sign-on (OAuth), and role-based access control are now generally available. They remain opt-in: `--user-auth-type` defaults to `none`, and custom role authoring stays off (`--rbac-authoring-disabled` defaults to `true`). Existing token workflows are unchanged. See [Manage users](/influxdb3/enterprise/admin/security/manage-users/). * **Integrated Explorer 1.11**: The Explorer UI embedded in `influxdb3 serve --mode ...,webui` is updated to 1.11. * Explorer connects to its own cluster automatically, with no URL or token to enter: at once with `--without-auth`, or after you sign in when user authentication is on. With token authentication alone, Explorer still asks for a URL and token. * With user authentication on and no users yet, Explorer opens a setup page that creates the first administrator and shows the operator token once. With OAuth sign-in only, the page shows the `influxdb3 manage init-admin --oauth-id` command to run instead. * The new `--webui-public-uri` option sets the public URL for single sign-on. Register `/auth/callback` as the redirect URI with your identity provider. * Sessions are stored on the server and last 120 days. Changing `--webui-session-secret` signs everyone out. * When you turn on user authentication on a deployment that used the v3.11 UI, Explorer offers once per browser to move saved dashboards, queries, and settings into the signed-in account. Saved API keys and tokens aren’t moved. * Adds Area, Single Stat, Gauge, and Pie visualizations, and `EXPLAIN ANALYZE` in the SQL editor. * **Orphaned file cleanup (upgraded storage engine)**: The primary compactor now finds and deletes compacted files in object storage that nothing references, left behind by crashes or failed jobs. The first pass starts a few minutes after the compactor first starts on v3.12, then passes repeat every 7 days. Objects modified in the last 24 hours are never deleted, and passes don’t run during a restore. To only record candidates, set `--compactor-sweep-mode dry-run`. To turn off cleanup, set `--compactor-sweep-interval off`. The offline, read-only `influxdb3 debug sweep list`, `influxdb3 debug sweep report`, and `influxdb3 debug verify-references` commands inspect cleanup passes and check that every file a checkpoint references exists. * **New metrics (upgraded storage engine)**: * Per-ingest-node compaction progress: `influxdb3_compactor_uncompacted_snapshots` and `influxdb3_compactor_last_compacted_snapshot_sequence`, labeled by `node_id`. * Data file cache: `influxdb3_file_cache_admission_rejected_total` counts fetches refused by the cache budget, and `influxdb3_file_cache_fetched_bytes_total` counts bytes fetched from object storage. * Ingest time spread: `influxdb3_snapshot_windows`, `influxdb3_snapshot_windows_seen_24h`, and `influxdb3_snapshot_rows_by_window_age_total` show how many time windows each snapshot touches and how old incoming rows are. * **New Processing Engine metrics**: `influxdb3_processing_engine_scheduler_*` gauges for pending, running, retrying, and auto-disabling invocations, `influxdb3_processing_engine_wal_*_watermark` and `influxdb3_processing_engine_timer_*_watermark_unix_seconds` for how far each process node has followed the WAL and the schedule clock, and `influxdb3_processing_engine_restart_state_checkpoints` for saved scheduler state. These apply to both storage engines. #### Bug fixes * **Missing rows from `IN` filters against a [file index](/influxdb3/enterprise/admin/file-index/) (Parquet engine)**: A filter with an `IN` list of four or more values on a column the file index doesn’t cover, such as a field, no longer skips every compacted file. Previously, such a query returned no rows from compacted data. An equality or `IN` filter could also skip every file after you narrowed a table’s file index columns. * **Login response time revealed valid usernames**: A username and password login now takes the same time whether or not the username exists. Previously, a login for an unknown username returned faster, which revealed whether a username existed. This affects deployments with `--user-auth-type basic`. * **Removing a node could delete acknowledged writes (upgraded storage engine)**: Removing a node that still has snapshots the compactor deferred now returns `409 Conflict` and names the deferred snapshots. Previously, removal deleted those files, losing writes that had been acknowledged. * **Two processes with the same `--node-id` (upgraded storage engine)**: When two ingest-capable processes run with the same `--node-id`, the one that loses a snapshot write now logs `another process is writing under this node id` and exits, so only one writer survives. * **Two primary compactors on S3 (upgraded storage engine)**: Each compactor lease renewal now writes a unique value. Previously, a renewal wrote identical content with an unchanged ETag, so a standby compactor could take the lease immediately after a renewal while the primary kept running. * **Retention kept deleting after it was cleared (upgraded storage engine)**: After you remove the last retention period, the compactor stops applying the old cutoffs. Previously, it kept deleting backfilled and late-arriving data by the old cutoffs until it restarted. * **Restore on GCS and compactor deletes during a restore (upgraded storage engine)**: A restore on Google Cloud Storage now holds its lease past the first renewal, so a second restore started at the same time returns `409 Conflict`. The compactor now holds its deletes while a restore runs. Previously, it could delete files the restore had just copied back. * **No compaction on new clusters with `--shard-count` above 1 (upgraded storage engine)**: A new cluster started with `--shard-count` greater than `1` now compacts. Previously, it never compacted, and query nodes eventually ran out of memory. * **Growing ingest file inventory (upgraded storage engine)**: `system.pt_ingest_files` and ingest node memory now track only live files. Previously, they grew with every file written for the life of the process. * **Repeated object storage reads for new ingest nodes (upgraded storage engine)**: Query nodes no longer fetch an ingest node’s recent files from object storage on every query when that ingest node joined after the query node started. * **Future-dated data displaced current data in the cache (upgraded storage engine)**: Cache preloading no longer warms time windows that start more than one window length in the future. Previously, every future window was warm, so data timestamped far in the future could use the preload budget before the current window loaded. * **`/ready` briefly reported not ready**: Expected object storage conflicts, such as a lost conditional write, no longer mark the node as not ready. * **Package install failures reported as success**: The package install API and `influxdb3 install package` now return an error with the end of the `pip` output when `pip` fails. Previously, they returned success, and the missing package surfaced later as a `ModuleNotFoundError`. * **Row deletes could remove the rows they were meant to keep (upgraded storage engine)**: A rewrite that applies a row delete keeps the files the delete doesn’t touch by their original ID. Previously, it also queued those files for deletion, so after the cleanup cooldown the rows the delete was meant to preserve were gone and later reads of the run set failed with `Object not found`. On the first start after upgrading from an affected v3.11.x, stale row-delete entries in the persisted delete queue are dropped instead of replayed. * **Backup and restore could hang (upgraded storage engine)**: The backup copy loop now keeps its in-flight copies running while it writes progress. Previously, a copy that held the object store client’s credential-refresh lock was left unpolled during a progress write, which blocked every request on that client and the backup or restore never finished. * **Rows lost when clearing the Parquet WAL after switching to the upgraded storage engine**: When a node starts on the upgraded storage engine with unflushed Parquet WAL, clearing that backlog no longer re-runs snapshots the previous process recorded but never persisted. Previously, a re-run could overwrite a manifest, and its Parquet file, so the rows it held were listed by no manifest. * **Queries that exceed memory limits returned 500**: The HTTP query endpoints now return `429 Too Many Requests` when a query fails because the data file cache budget or the DataFusion memory pool is exhausted. Previously, they returned `500`. Flight already returned `RESOURCE_EXHAUSTED` for the same errors. * Other bug fixes and performance improvements #### Breaking changes * **Data file cache is a hard limit (upgraded storage engine)**: [`--file-cache-size`](/influxdb3/enterprise/reference/config-options/#file-cache-size) now also counts data file bytes held by running queries. When the budget is used up, the query that needs more fails with `429 Too Many Requests` and a `file access cache budget exhausted` error instead of the node using memory beyond the budget. Query nodes that were killed for running out of memory under this load now stay up and fail individual queries. * **Nodes without query mode refuse data queries (Parquet engine)**: Nodes that don’t run the `query` mode now return `405 Method Not Allowed` for data queries. System table queries still work. Previously, these nodes answered data queries, sometimes with partial results. The upgraded storage engine already behaved this way. * **`--node-spec` and WAL triggers on multiple process nodes**: `--node-spec` no longer pins where a trigger runs; it selects the process nodes whose schedulers own the trigger. With the default, `all`, every process node owns the trigger and now follows every ingest node’s WAL, so a WAL trigger runs once per process node for each WAL flush. To run a WAL trigger once per flush in a cluster with more than one process node, set `--node-spec` to a single node. * **Request triggers return 503 when their queue is full**: A request trigger with 60 pending invocations now returns `503 Service Unavailable` with `trigger queue is full`. Previously, the request waited for a free slot. * **Username and password sessions must be renewed**: Access tokens issued to users who sign in with a username and password now must carry the cluster’s catalog UUID as the `aud` claim. Access tokens issued before v3.12 are rejected. Refresh the token or sign in again. API tokens aren’t affected. * **`--compaction-max-num-files-per-plan` has no effect (Parquet engine)**: The compactor no longer limits plans by input file count. Setting the option logs a deprecation warning. * **Metric changes**: The unused `influxdb3_memory_pool_evictions`, `influxdb3_memory_pool_rejections`, `influxdb3_memory_pool_eviction_bytes`, `influxdb3_memory_pool_eviction_size_bytes`, and `influxdb3_memory_pool_eviction_duration_seconds` metrics are removed. `influxdb3_compaction_plans_skipped` (Parquet engine) now reports `reason="memory_exhaustion"` with no `phase` label, instead of `reason="file_limit"`. #### Known issues The following issues affect the Processing Engine. For setup guidance, see [Configure a standalone process node](/influxdb3/enterprise/admin/processing-engine/). * **Queued WAL trigger runs can fail**: If a WAL trigger’s queued runs fall behind and an ingest node deletes a WAL file before its run starts, the run fails with `WAL object ... is no longer available`. The error appears only in the process node’s log. * **WAL triggers can stop after a long outage**: If the process node is down longer than the ingest nodes keep their WAL files, its WAL triggers don’t process new writes after it restarts, and the node logs nothing. To detect it, check that `influxdb3_processing_engine_wal_processed_watermark` in the process node’s `/metrics` output increases. Workaround: restart the process node again. * **Docker Desktop containers exit with code 133**: On a new Docker Desktop installation, a node with a bind-mounted data directory can exit with code 133 because of a conflict in the Processing Engine’s restart state. * **`influxdb3 show plugins` is empty on a process node**: The command returns no plugins even when plugin files exist, and querying `system.plugin_files` on the process node returns `400 table 'public.system.plugin_files' not found`. Workaround: list the files in the plugin directory. * **`system.processing_engine_triggers` doesn’t show `--node-spec`**: You can’t query which node owns a trigger. Workaround: check the `node_id` column in `system.processing_engine_logs` or the process node’s log. * **Plugins can’t read string columns from `system.nodes`**: The string columns use the `Utf8View` type, which the plugin API can’t convert (`unsupported Arrow type for Python conversion: Utf8View`), and `CAST(... AS VARCHAR)` doesn’t help. Workaround: query with `database="_internal"` and convert each string column with `arrow_cast(, 'Utf8')`. * **Configuration template shows the wrong virtual environment path**: The `.conf` template names the virtual environment `/venv`. The actual path is `/.venv`. * **Some options are missing from `influxdb3 serve --help`**: `--trigger-work-silence-timeout`, `--processing-engine-restart-state-snapshot-interval`, `--plugin-repo`, and the internode options appear only in `influxdb3 serve --help-all`. ### Known issues The query-error note below also describes v3.12.1 behavior. #### Core * **Failed package installs return success (Core)**: In Core, the package install API can return `200` and `influxdb3 install package` can report success when `pip` exits with an error. Enterprise returns an error. Workaround: after you install a package, run a trigger that imports it to confirm the install. * **Memory-limit query errors depend on the edition, output format, and version**: In v3.12.0, memory-limit errors in `pretty` or `parquet` output return `500` on Core and `429 Too Many Requests` on Enterprise. CSV, JSON, and JSON Lines responses can instead return `200` and end early. In v3.12.1, a memory-limit error before the first rows returns `500` on Core or `429` on Enterprise in all output formats. An error after streaming starts still ends the response early. Workaround: handle query errors and incomplete responses, and narrow a query that exhausts memory before you retry. * **Creating a table over HTTP requires `fields`**: `POST /api/v3/configure/table` returns `400` with `missing field `fields` `when the request body has no `fields` key. Workaround: send `"fields": []` for a table with no fields. See [Create a table](/influxdb3/enterprise/admin/tables/create/). #### Enterprise * **Tables with no fields can’t be queried (upgraded storage engine)**: A query on a table that has tags but no fields returns `500` with `field_family_projections is required`. Workaround: define at least one field when you create a table. * **`PUT /api/v3/configure/table` doesn’t add columns**: `PUT` ignores `tags` and `fields` in the request body and still returns `200`. Workaround: use `PATCH /api/v3/configure/table` to add columns. See [Update a table](/influxdb3/enterprise/admin/tables/update/). * **Bulk import catalog rejections return 500**: An import that adds an undeclared column to an explicit schema, or infers a type incompatible with an existing column, can return `500` with `Could not modify catalog` and create no import job. Compatible new columns can be added in an implicit database. Column-mapping validation errors can return `400`. Workaround: fix the column mapping or declare the missing columns, then import again. Retrying the same rejected request doesn’t resolve the catalog error. * **Generic Parquet imports don’t infer tags from the target table**: For generic Parquet files, the importer infers an unmapped string column as a field, even when the target table declares that column as a tag. The import then fails. IOx Parquet files preserve their tag metadata. Workaround: for generic Parquet files, map every string tag column, for example `--column host=tag` or `"column_metadata": {"column_mapping": {"host": "tag"}}`. See [Import data](/influxdb3/enterprise/admin/import-data/). * **Bulk import jobs can report a time range of 0**: If a source Parquet file has no usable time-column minimum and maximum statistics, the import job can report `min_timestamp_ns` and `max_timestamp_ns` as `0`. A job can complete with that metadata. An error while reading the statistics instead returns `400`. * **Queued imports can run after their database is soft-deleted**: An import job that is queued when you soft-delete its database can still run and complete. Workaround: wait for a database’s import jobs to finish before you delete the database. ## v3.11.5 ### Core #### Features * **Age-based snapshot trigger**: The new [`--force-snapshot-max-age`](/influxdb3/enterprise/reference/config-options/#force-snapshot-max-age) option (default `1h`) forces a snapshot once the oldest WAL data that isn’t in a snapshot reaches that age. Previously, snapshots triggered only on WAL file count or buffer memory, so a node receiving few or no writes could hold hours of data in memory and replay all of it on its next restart. #### Bug fixes * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Parquet engine WAL and snapshot metrics**: Parquet engine nodes that ingest data now report WAL files and bytes not yet covered by a persisted snapshot (`influxdb3_parquet_wal_pending_files`, `influxdb3_parquet_wal_pending_bytes`), buffered bytes and the threshold that forces a snapshot (`influxdb3_parquet_write_buffer_bytes`, `influxdb3_parquet_write_buffer_memory_threshold_bytes`), the time of the last successful snapshot (`influxdb3_parquet_snapshot_last_success_timestamp_seconds`), and failed snapshot attempts by `error_code` (`influxdb3_parquet_snapshot_failures_total`). * **Stalled storage engine upgrade reporting**: `system.upgrade_parquet_node` now flags a node whose upgrade has stopped making progress (`stalled`, `stalled_since`), and reports when it last progressed (`last_progress_at`), how many times the compactor restarted during the upgrade (`restarts`), and conversion and import progress. Previously, an upgrade that had stopped making progress looked the same as one still in progress. * **Compaction completion logging (Parquet engine)**: The compactor logs each completed compaction at `INFO` level, so the logs show whether compaction is running. * **Stuck compaction warning (Parquet engine)**: The compactor logs a `compaction not planned` warning when a table has enough uncompacted files for a compaction but none has been planned for 15 minutes, and repeats it every 15 minutes while the condition holds. #### Bug fixes * **Missing rows from filtered queries on tables without tags (upgraded storage engine)**: A query with a `WHERE` clause, including one with only a time predicate, now returns compacted data from a table that has no tag columns. Previously, once such a table’s data compacted, a filtered query could return no rows, while the same query without a `WHERE` clause returned every row. * **Compaction stopped while work waited for memory (upgraded storage engine)**: Compaction work waiting for memory is no longer aborted as stuck before it starts. Previously, when more work was scheduled than memory could admit, the waiting work was aborted and retried until its compaction failed, after which the compactor stopped planning new work, including snapshots. * **Missing rows from tag queries after a compactor restart (Parquet engine)**: Removing a generation now removes only that generation’s [file index](/influxdb3/enterprise/admin/file-index/) entries. Previously, a compact-only node numbered its files from zero again after a restart, so removing an older generation also removed the index entries for a live file with the same number, and queries that filter on an indexed tag value skipped that file and returned incomplete results. * **Startup stalled on large compaction metadata**: Compaction summary, compaction detail, and generation detail reads larger than 128 MiB are now fetched as 16 MiB ranged reads, so a slow or failed transfer retries one range. Previously, a single whole-object GET restarted a multi-gigabyte download from byte 0, so a node loading Parquet engine compacted data at startup, including during a storage engine upgrade, could stall without an error. * **Stopped nodes blocked an At-Home license node from starting**: The [At-Home license](/influxdb3/enterprise/admin/license/) single-node check now ignores nodes in the `stopped` or `removing` state. Previously, any other node registered in the catalog prevented startup. * **Ingest-only nodes built an unused file index**: Nodes that run only in ingest mode (`--mode ingest`) no longer build the Parquet engine’s compacted data file index at startup, including during a storage engine upgrade, which reduces their startup memory. Ingest-only nodes never use the index. * **Unneeded file index decoding during a storage engine upgrade**: The upgrade now reads only the file list from each generation’s detail object when it discovers files to import. Previously, it also decoded the file index, which added CPU and memory cost to compactor startup on tables with large indexes. * **Spurious snapshot sequence error on first startup (upgraded storage engine)**: A new node starting with an empty object store no longer logs a `snapshot sequence seeds disagree` error. * Other bug fixes and performance improvements ## v3.11.4 ### Core #### Bug fixes * **Overwrites in the write buffer resolved arbitrarily**: Writes that repeat a series key and timestamp within one write buffer chunk now resolve to the last acknowledged write, in query results and in the persisted Parquet file. Previously the duplicate rows tied on both deduplication sort keys and an unstable sort picked an arbitrary survivor, so a superseded version could win. An overwrite that arrives while a snapshot is persisting also now wins over the frozen rows it replaces. * **Snapshot manifest skipped for an empty snapshot**: Every snapshot sequence number now lands a manifest, including a snapshot with nothing to persist. Previously the manifest was skipped, leaving a permanent hole in the sequence. Consumers that walk the sequence by exact key cannot tell a skipped snapshot from a manifest that failed to persist, so a single hole stalled Enterprise compaction on that node and blocked its read replicas. * **Stale plugin source after re-enabling a trigger**: Stopping a trigger now drops its plugin from the worker’s in-memory cache, so re-enabling it re-reads the code. Since v3.11.0 the cache was insert-only, so disabling and re-enabling a trigger resumed the cached copy and never contacted the plugin repository again, which is the documented way to pick up a new version of a `gh:` plugin. * **Plugin source retained after a database was deleted**: Deleting a database now evicts its triggers’ cached plugin source. Previously it stayed resident until the server restarted. * **Processing engine Python updated to 3.13.15**: Picks up the security fixes in that release. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Startup phase logging**: The server logs the start and completion of each startup phase, so a node killed during startup shows which phase it was in. #### Bug fixes * **Privilege escalation through role assignment**: Creating, updating, and deleting a role, and assigning roles to a user, now require admin authorization. Previously a non-admin subject holding `User:Update`, or `User:Create` with roles in the request, could grant permissions beyond its own. This affects the [user authentication preview](/influxdb3/enterprise/admin/security/manage-users/), which is off by default (`--user-auth-type none`); deployments that have not enabled it are unaffected. * **Out-of-memory clearing the WAL backlog during a storage engine upgrade**: The upgrade now snapshots during WAL replay whenever the buffer reaches a quarter of the query executor’s memory pool, instead of buffering the whole backlog and persisting it in one pass. Each snapshot removes the WAL files it covers, so a restart resumes from the last one rather than replaying from the beginning. * **Out-of-memory importing a large source during a storage engine upgrade**: Bulk import now admits a job by its estimated working set rather than its compressed size, runs a job larger than the compactor’s input-size budget on its own, and refuses a source that cannot fit at all with an error naming the budget it would need. Previously an oversized source was retried until the node ran out of memory, with no attempt recorded. * **File index left behind when a generation expires**: Retention now deletes a generation’s detail object along with its Parquet files. Previously the detail, which holds the file index describing those files, was left unreferenced in object storage. On a table with high-cardinality index columns the detail is the larger of the two, so retention shrank the data and accumulated the indexes for the life of the cluster. * **Compactor replans forever against a missing index file (upgraded storage engine)**: A run set whose index object is confirmed gone from object storage is now skipped after three failed reads on separate scheduler passes, so planning moves on to other work. Previously every planner that read the index failed, abandoned the plan, and recreated it on the next tick indefinitely. The skip list is in memory only: a restart clears it, and a repaired index is picked up again. * **Empty snapshot manifests were never deleted**: Gen1 cleanup now deletes a consumed manifest that references no files. Previously it returned early on such a manifest without deleting it, and because cleanup retains the oldest entries in each batch, undeletable manifests could eventually fill every batch and starve cleanup. * **Stale plugin source after a catalog restore**: Restoring a catalog now evicts the worker’s cached plugin source, so restarted triggers re-read their code. Previously a trigger whose restored definition matched the cached one resumed the pre-restore source without contacting the plugin repository. * **Spurious memory reservation warning at startup**: The warning threshold is now above the sum of the shipped percentage defaults, which come to 90 percent of detected memory. Previously a server running the default configuration warned about itself. * Other bug fixes and performance improvements ## v3.10.6 ### Core #### Features * **`--shutdown-timeout` graceful shutdown bound**: This new option (default `30s`) caps how long the server waits for active connections to drain during shutdown before forcibly closing them. Set it to `0s` to skip the drain. #### Bug fixes * **Crash loop migrating a catalog written by Core**: Catalog snapshots deserialize again when the node spec is missing from trigger, last value cache, and distinct value cache entries, defaulting to all nodes. Core never writes that field, so the unified deserializer rejected Core-written checkpoint files and the v2-to-v3 catalog migration crash-looped on upgrade. * **Snapshot manifest skipped for an empty snapshot**: Every snapshot sequence number now lands a manifest, including a snapshot with nothing to persist. Previously the manifest was skipped, leaving a permanent hole in the sequence. Consumers that walk the sequence by exact key cannot tell a skipped snapshot from a manifest that failed to persist, so a single hole stalled Enterprise compaction on that node and blocked its read replicas. * **Unnecessary node shutdown after a retried WAL write**: Each WAL file now carries a nonce in its object metadata, so a node recognizes its own write and shuts down only for a file written by another node. Previously, a conditional PUT that was applied but returned a 500 was retried, and the node read the resulting 412 as a second process holding the same `--node-id` and shut down, though the file was durable and correct. * **Panic on oversized duration values**: Time arithmetic now saturates at the minimum or maximum representable timestamp instead of panicking when a duration is too large to represent. Previously, an oversized user-supplied duration produced a panic and could put the server into a startup panic loop. * **Processing engine Python updated to 3.13.15**: Picks up the security fixes in that release. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Startup phase logging**: The server logs the start and completion of each startup phase, so a node killed during startup shows which phase it was in. #### Bug fixes * **Privilege escalation through role assignment**: Creating, updating, and deleting a role, and assigning roles to a user, now require admin authorization. Previously a non-admin subject holding `User:Update`, or `User:Create` with roles in the request, could grant permissions beyond its own. This affects the [user authentication preview](/influxdb3/enterprise/admin/security/manage-users/), which is off by default (`--without-user-auth true`); deployments that have not enabled it are unaffected. * **File index left behind when a generation expires**: Retention now deletes a generation’s detail object along with its Parquet files. Previously the detail, which holds the file index describing those files, was left unreferenced in object storage. On a table with high-cardinality index columns the detail is the larger of the two, so retention shrank the data and accumulated the indexes for the life of the cluster. * **Slow compaction catch-up behind a lagging node**: The compactor now accumulates the rerun signal across every node in a cycle. Previously it kept only the last node’s value, so a caught-up node could end the cycle after a single snapshot and leave a lagging node advancing one snapshot per check interval until restart. * Other bug fixes and performance improvements ## v3.9.13 ### Core #### Bug fixes * **Snapshot manifest skipped for an empty snapshot**: Every snapshot sequence number now lands a manifest, including a snapshot with nothing to persist. Previously the manifest was skipped, leaving a permanent hole in the sequence. Consumers that walk the sequence by exact key cannot tell a skipped snapshot from a manifest that failed to persist, so a single hole stalled Enterprise compaction on that node and blocked its read replicas. * **Unnecessary node shutdown after a retried WAL write**: Each WAL file now carries a nonce in its object metadata, so a node recognizes its own write and shuts down only for a file written by another node. Previously, a conditional PUT that was applied but returned a 500 was retried, and the node read the resulting 412 as a second process holding the same `--node-id` and shut down, though the file was durable and correct. * **Restarted downloads of large manifests and checkpoints at startup**: Snapshot manifest and checkpoint reads larger than 128 MiB are now fetched as 16 MiB ranged reads, so a slow or failed transfer retries one range. Previously, a single whole-object GET restarted a multi-gigabyte download from byte 0. * **Panic on oversized duration values**: Time arithmetic now saturates at the minimum or maximum representable timestamp instead of panicking when a duration is too large to represent. Previously, an oversized user-supplied duration produced a panic and could put the server into a startup panic loop. * **Processing engine Python updated to 3.13.15**: Picks up the security fixes in that release. * Other bug fixes and performance improvements ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Startup phase logging**: The server logs the start and completion of each startup phase, so a node killed during startup shows which phase it was in. #### Bug fixes * **File index left behind when a generation expires**: Retention now deletes a generation’s detail object along with its Parquet files. Previously the detail, which holds the file index describing those files, was left unreferenced in object storage. On a table with high-cardinality index columns the detail is the larger of the two, so retention shrank the data and accumulated the indexes for the life of the cluster. * **Slow compaction catch-up behind a lagging node**: The compactor now accumulates the rerun signal across every node in a cycle. Previously it kept only the last node’s value, so a caught-up node could end the cycle after a single snapshot and leave a lagging node advancing one snapshot per check interval until restart. * Other bug fixes and performance improvements ## v3.11.3 ### Core No adjustments in this release. Core remains on v3.11.2. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Read run-set indexes written by later releases**: The compactor now reads the v3 run-set index format that later releases write, so you can roll back to v3.11.3 from a release that writes v3 indexes without leaving unreadable indexes behind. #### Bug fixes * **Missing rows from an `OR` predicate against a [file index](/influxdb3/enterprise/admin/file-index/)**: File pruning now applies to an `OR` only when both sides resolve against the index, and otherwise falls back to scanning all files. Previously, when one side referenced a column that the index doesn’t cover, the query kept only the files that matched the resolved side, so rows that matched only through the uncovered side were dropped from the results. * **Deferred snapshot recovery makes no progress on a busy cluster**: The new[`--recover-deferred-snapshots-in-flight-limit`](/influxdb3/enterprise/reference/storage-engine-config-options/#deferred-snapshot-recovery)option raises the per-node in-flight snapshot limit that gates`--recover-deferred-snapshots`. The limit was previously fixed at three, so on a cluster whose ingest nodes snapshot faster than compaction completes batches, recovery never ran. The default is unchanged. * Other bug fixes and performance improvements ## v3.11.2 ### Core #### Bug fixes * **Rows missing from queries during snapshot persistence**: Each persist job now drops only the buffer chunk whose Parquet file it published. Previously, the first job of a snapshot to finish evicted every sibling chunk, so whole time ranges vanished from query results until their persist job finished. * **Partial schema changes from a rejected line**: A line protocol line is now validated in full before the catalog is mutated, so a rejected line contributes nothing to it. Previously, with `accept_partial=true`, columns created earlier in the line were committed anyway, and a line rejected after its tags were processed merged them into the next valid line’s series key. * **Unnecessary node shutdown after a retried WAL write**: Each WAL file now carries a nonce in its object metadata, so a node recognizes its own write and shuts down only for a file written by another node. Previously, a conditional PUT that was applied but returned a 500 was retried, and the node read the resulting 412 as a second process holding the same `--node-id` and shut down, though the file was durable and correct. * **Panic querying `system.parquet_files` for an unknown table**: Filtering on a `table_name` that is not a current table, or querying after its database is dropped, now returns zero rows. Previously, both panicked the request thread and returned a truncated HTTP 200. * **Duplicate rows from a concurrent snapshot handoff**: Buffer chunks and persisted Parquet files are now read under a single lock. Previously, a read landing between the two saw a new Parquet file alongside the buffer chunks it was written from and returned those rows twice. Served queries masked this through deduplication, but paths that read record batches directly, such as the Processing Engine, did not. * **Restarted downloads of large manifests and checkpoints at startup**: Snapshot manifest and checkpoint reads larger than 128 MiB are now fetched as 16 MiB ranged reads, so a slow or failed transfer retries one range. Previously, a single whole-object GET restarted a multi-gigabyte download from byte 0. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Retry storage engine upgrade**: The new `influxdb3 manage retry-upgrade-to-pacha-tree` command resets a [storage engine upgrade](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet), so it resumes the next time the compactor node starts with `--upgrade-pacha-tree`. Sources whose table, database, or objects are gone become skips, and anything still readable returns to the queue. Nothing is deleted, so the command is safe to run again. #### Bug fixes * **Compaction falls behind on a large snapshot backlog**: Snapshot compaction now caps outstanding plans relative to the compactor’s core count, batches snapshots in ingest-time order, and fetches each plan’s gen0 files once per worker. Previously, a deep backlog could accumulate thousands of outstanding plans, recompact the same time windows once per node, and delay the first plan by over 12 minutes. * **Compactor primacy livelock during a migration**: Primary lease reads and writes now bypass the process-wide object store concurrency limit. Previously, a migration’s bulk import could starve lease renewal, so the holder repeatedly demoted at its 30-second lease TTL and reacquired, pinning the migration indefinitely while object storage itself was healthy. * **Multi-window migration imports make no progress**: The migration baseline now clips each staged file’s contribution to its window’s time range, and migration imports the newest windows first. Previously, a file crossing a window boundary caused publish validation to refuse each consolidation, so the scheduler replanned the same work forever. * **Migration fails on an unreadable source**: A source Parquet file confirmed missing, or whose table or database has been dropped, is now recorded as a skip instead of failing the whole upgrade, and `system.upgrade_parquet` reports it as `skipped_source_missing` or `skipped_table_dropped`. Only confirmed-missing sources are skipped, so a transient `NotFound` stays retryable and a grouped job still converts its readable companions. * **Compaction and migration fail on stale gen1 references**: `leftover_gen1_files` references are now reconciled against the deletions that ingesters record in their snapshots, so a reference is dropped when its Parquet file is deleted regardless of retention configuration. Previously, references were pruned only against the current retention cutoff, so they accumulated indefinitely on a table with no retention period, and a storage engine upgrade then failed enumerating them. * **`Internal error` querying a last value or distinct value cache**: A query node now registers caches that were created on another node during startup, regardless of the cache’s node spec. Previously, a cache present in the catalog but missing from a node’s in-memory provider failed every query against it with `last cache crate is invalid` until the node was restarted. * **Duplicate rows from a replica snapshot handoff**: Each buffer on a node now has its chunks and Parquet files read under a single lock, and each replica advances the gen1 chunk order offset by its own chunk count. Previously, a snapshot handoff landing between the two read passes returned those rows twice, and on clusters with more than three replicas the old offset let local chunks reuse an order a replica had taken. * **Slow compaction catch-up behind a lagging node**: The compactor now accumulates the rerun signal across every node in a cycle. Previously it kept only the last node’s value, so a caught-up node could end the cycle after a single snapshot and leave a lagging node advancing one snapshot per check interval until restart. * **Catalog checkpoint persistence failures for large payloads**: Catalog checkpoint and month-rollover writes now fall back to a multipart upload for oversized payloads. Previously, both used a single PUT, so a checkpoint merging a month of snapshots could exceed object storage’s 5 GiB single-PUT limit and fail, silently losing the boot-time fast path. * **Warning flood during startup snapshot restore**: `removed_files` entries whose target database or table is no longer present are now reported in one aggregated warning per restore. Previously, each miss logged its own warning, which buried real startup errors and slowed restore. * Other bug fixes and performance improvements #### Breaking changes * **`--l1-hot-tail-target-size` is functional again**: The option (default `250mb`) caps live L1 tail rewrites during snapshot compaction, and a larger tail is handed to L1 consolidation to seal. It was previously accepted and ignored. * **`--l1-consolidation-min-age` and `--row-delete-max-jobs-per-tick` are now ignored**: L1 consolidation has no age gate, and row-delete work shares the weighted delete slot. Both options are still accepted. * **`--l1-consolidation-target-size` is now the established-L1 size boundary**: Consolidation consumes run sets at or below it, and promotion requires at least two L1 run sets larger than it. `--l1-consolidation-min-run-sets` and `--compactor-max-source-run-sets-per-promotion` now apply only to the legacy layout. * **`--replica-snapshot-manifest-load-concurrency` applies to both storage engines**: It now bounds the boot-time snapshot manifest load on the Parquet engine as well as the upgraded engine. When unset, each engine derives its own default. * **Compactor status log fields changed**: The `service_memory_*` fields are gone, and `pending_input_mb` is renamed to `charged_input_mb`. ## v3.11.1 ### Core #### Bug fixes * **Panic on oversized duration values**: Time arithmetic now saturates at the minimum or maximum representable timestamp instead of panicking when a duration is too large to represent. Previously, an oversized user-supplied duration produced a panic and could put the server into a startup panic loop. * **Duplicate admin token registration**: The catalog now rejects registering a token whose hashed value already exists. Previously, the server registered the duplicate under a new ID. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Remove migrated Parquet data after a storage engine upgrade**: The new `influxdb3 manage cleanup-parquet` command and `/api/v3/enterprise/upgrade/parquet_cleanup` API permanently delete the pre-upgrade Parquet data and compactor metadata from object storage after a [storage engine upgrade](/influxdb3/enterprise/reference/internals/storage-engine/#upgrade-from-parquet) completes. Use `--dry-run` to report what a cleanup would delete, and how much space it would reclaim, without deleting anything; `--wait` to poll until the cleanup completes; and `--status-only` to check the status of the current or most recent cleanup. The cleanup runs on the compactor node, resumes automatically after a restart, and reports progress in the new `system.upgrade_parquet_cleanup` system table. After a cleanup deletes data, [downgrading to Parquet](/influxdb3/enterprise/reference/internals/storage-engine/#downgrade-to-parquet) is no longer possible. #### Bug fixes * **Data missing from queries during a storage engine upgrade**: While a cluster migrates from Parquet to the upgraded storage engine, hybrid queries now serve the migrating node’s own not-yet-compacted gen1 data, and data imported during the migration is queryable in all-in-one mode. Previously, on a single-node cluster, rows that the Parquet compactor had not yet folded into compacted data were missing from query results for the duration of the migration. * **Compactor crash loop after primary re-election**: The catalog’s subscription registry now reclaims a subscription slot whose receiver was dropped and skips closed subscribers when broadcasting updates, and compactor teardown now completes before the next term re-subscribes. Previously, a compactor primary re-election within one process could panic on the stale slot and crash-loop the node, and a lingering stale slot could fail an unrelated catalog update. * **Compactor retention sweep stalls (upgraded storage engine)**: The retention sweep now checks each run set’s key span in memory, and remembers negative results, instead of re-reading every run-set index from object storage on every scheduler tick. Previously, one table with a short retention period alongside a large historical dataset could freeze compaction dispatch, completion handling, and status reporting for hours while the sweep serially re-read every index. * **Query node boot loop with a large snapshot backlog (upgraded storage engine)**: At startup, a query node now fetches peer snapshot manifests concurrently, bounded by `--replica-gen0-load-concurrency` (default `16`), and walks at most 10,000 manifests per node. If the cap is reached, the server logs an error noting that data in the skipped older snapshots stays unqueryable until compaction covers it. Previously, the walk fetched the entire backlog serially with no bound, so a large backlog behind a stalled compactor could run for hours, exhaust memory, and restart the node from zero in a permanent boot loop. * **Incorrect query results from out-of-order series scans (upgraded storage engine)**: A window scan that advertises series-key ordering now always merge-sorts its inputs. Previously, files could be chained together out of order while the query plan claimed sorted output, and downstream sort-merge operations silently produced incorrect results. * Other bug fixes and performance improvements ## v3.11.0 ### Core #### Features * **Async trigger concurrency limit**: The new `--async-trigger-concurrency-limit`option caps the number of concurrent invocations for an asynchronous processing engine trigger. The default is unlimited. * **Processing engine retry behavior for asynchronous triggers**: When`trigger_settings.run_async = true`, a failed trigger invocation now retries up to 5 times before being discarded. Previously, failed invocations retried indefinitely. * **WAL triggers skip empty flushes**: A WAL trigger no longer runs when the WAL flush it would process is empty. Previously, the trigger ran on every flush, including empty ones. * **Disabled trigger state persists across restarts**: When`trigger_settings.error_behavior = disable` disables a trigger, the disabled state now survives a server restart. Previously, a disabled trigger re-enabled itself on restart. * **`--disable-package-management`**: Use this option to prevent the server from creating or modifying a Python virtual environment or invoking `pip`. Package-install API calls are rejected while it’s set. * **Virtual `event_time` column on `system.processing_engine_logs`**: Queries that reference the former `event_time` column continue to work after the column was renamed to `time`. * **`--shutdown-timeout` graceful shutdown bound**: This new option (default`30s`) caps how long the server waits for active connections to drain during shutdown before forcibly closing them. Set it to `0s` to skip the drain. #### Bug fixes * **cgroup-aware resource sizing**: Containerized deployments now size memory-based defaults against the container’s cgroup limits instead of the host machine’s total resources. #### Breaking changes * **Size options require an explicit unit**: Options that accept a size value (for example, `--exec-mem-pool-size` and `--file-cache-size`) now reject a bare number. Append a unit suffix (`b`, `kb`, `mb`, `gb`, `tb`) or, where noted, a percentage (for example, `20%`). This avoids a silent change in meaning — historically, a bare number meant megabytes for some options and bytes for others. * **Three memory and cache options renamed, with a deprecated alias**:`--exec-mem-pool-bytes` is now `--exec-mem-pool-size`,`--parquet-mem-cache-size` is now `--file-cache-size`, and`--force-snapshot-mem-threshold` is now `--force-snapshot-mem-size`(`--parquet-mem-cache-query-path-duration` is now `--file-cache-recency`, and `--disable-parquet-mem-cache` is now `--disable-file-cache`). The deprecated old names, and their environment variables, still work and still accept a bare number as megabytes — their pre-3.11 meaning — but log a startup deprecation warning. * **`--max-http-request-size` keeps its name and its pre-3.11 meaning**: A bare number is still accepted as bytes, but now logs a startup warning. Prefer an explicit unit suffix, for example `10mb`. * **`--query-log-size` renamed to `--query-log-max-entries`**, with the old name and its environment variable kept as a deprecated, backward-compatible alias. * **Other options and environment variables renamed, with a deprecated alias**: legacy names still work and log a startup deprecation warning; if both the old and new name are set with different values, the new name wins. | Old | New | |----------------------------------------------------------|-----------------------------------------| |`--disable-parquet-mem-cache`, `--disable-data-file-cache`| `--disable-file-cache` | | `--wal-max-write-buffer-size` | `--wal-max-buffered-writes` | | `--wal-snapshot-size` | `--wal-files-per-snapshot` | | `INFLUXDB3_DB_DIR` | `INFLUXDB3_DATA_DIR` | | `INFLUXDB3_NODE_IDENTIFIER_PREFIX` | `INFLUXDB3_NODE_ID` | | `INFLUXDB3_NODE_IDENTIFIER_FROM_ENV` | `INFLUXDB3_NODE_ID_FROM_ENV` | | `INFLUXDB3_NUM_WAL_FILES_TO_KEEP` |`INFLUXDB3_SNAPSHOTTED_WAL_FILES_TO_KEEP`| | `INFLUXDB3_START_WITHOUT_AUTH` | `INFLUXDB3_WITHOUT_AUTH` | | `INFLUXDB3_TCP_LISTINER_FILE_PATH` (misspelling) | `INFLUXDB3_TCP_LISTENER_FILE_PATH` | | `INFLUXDB3_TELEMETRY_DISABLE_UPLOAD` | `INFLUXDB3_DISABLE_TELEMETRY_UPLOAD` | These renames apply to both Core and Enterprise. For the separate Enterprise-only `INFLUXDB3_ENTERPRISE_*` → `INFLUXDB3_*` environment variable renames, see Enterprise breaking changes below. * **Duplicate tag keys are rejected at write time**: A point that repeats a tag key (for example, `m,t=a,t=a f=1i`) is now rejected with a clear error, the same way duplicate field keys are. * **`/api/v2/write` returns 503 instead of 400 when the node is stopped**: Clients that key retry logic off the status code now correctly treat this response as retryable. * **`--hard-delete-default-duration` is confirmed to have no effect**: This option has never affected hard-delete behavior in any release — the server always uses its built-in default duration. The option is still accepted so existing configurations keep starting, but the server now logs a startup warning recommending you remove it. * **`--object-store-cache-endpoint` removed**: This option was parsed but consumed by dead code and never had an effect. The flag now fails to parse at startup, and the `INFLUXDB3_OBJECT_STORE_CACHE_ENDPOINT` /`OBJECT_STORE_CACHE_ENDPOINT` environment variables are ignored. Remove it from your configuration. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **The upgraded storage engine is now the default**: New clusters are created on the upgraded storage engine with no opt-in flag required. The engine is resolved from the catalog’s persisted storage mode, not from a flag: * A brand new cluster is stamped for the upgraded storage engine at catalog creation. There’s no user-facing opt-out. * An existing Parquet cluster keeps running Parquet after the binary upgrade. The persisted mode wins on load. * To migrate an existing Parquet cluster, pass `--upgrade-pacha-tree`(environment variable `INFLUXDB3_UPGRADE_PACHA_TREE`). This moves the catalog into a hybrid Parquet-and-upgraded-storage-engine mode and starts the migration. A catalog in the upgraded or migrating mode always runs the upgraded storage engine from then on. `--use-pacha-tree` still works and keeps its historical migration-starting semantics, but is deprecated and logs a warning at startup. Core remains Parquet-only. * **Compaction throughput improvements (upgraded storage engine)**: Time-disjoint two-level compaction and overlapping-L1 leading-edge parallelism are now the default. Previously, all leading-edge ingest funneled through a single hot-tail L1 run set. This capped leading-edge throughput at one in-flight compaction job. In 3.11, several concurrent L1 compaction jobs can serve the leading edge and any heavily-written range. This decouples ingest throughput from compactor reconciliation. New windows and shards use the time-disjoint layout. Existing checkpoints keep their recorded layout until you upgrade them. * **Integrated Explorer**: The InfluxDB 3 Web UI (Explorer) now ships inside the Enterprise binary as a WebAssembly guest, hosted in-process behind a WASI sandbox. Enable it with `--mode all,webui`. The Web UI is **not** included in plain `--mode all`, and a session secret is mandatory when `webui` mode is enabled: ``` mkdir -p ./plugins influxdb3 serve \ --cluster-id cluster0 \ --node-id node0 \ --mode all,webui \ --plugin-dir ./plugins \ --webui-session-secret "$(openssl rand -base64 24)" ``` You configure a connection to your local server (for example,`http://localhost:8181`), the same way you connect to a database server from the standalone Docker Explorer. It keeps a SQLite database that’s automatically synchronized to object storage per cluster. AI chat is included and can point at any OpenAI-compatible endpoint with `--webui-openai-base-url`. For setup steps, see[Use the integrated InfluxDB 3 Explorer UI](/influxdb3/enterprise/visualize-data/explorer/). * **Thread defaults scale with your license on the upgraded storage engine**: On clusters running the upgraded storage engine (the default for new clusters), `--num-io-threads` and the DataFusion thread pool each default to your licensed core count, instead of the flat defaults used on Parquet-engine clusters. `--num-cores` validation runs after the server resolves the active storage engine, since the upgraded engine can license more cores than a Parquet-engine cluster does. A thread count set above the licensed core count is capped with a startup warning instead of rejected. * **Incremental backups**: Backup is no longer full-only. Each incremental backup names a parent, and restoring an incremental walks the manifest chain to produce a full restore. Deleting an incremental also deletes every child that depends on it. ``` influxdb3 create backup --name base --token $ADMIN_TOKEN influxdb3 create backup --name inc-1 --incremental --parent base --token $ADMIN_TOKEN influxdb3 create restore --backup inc-1 --token $ADMIN_TOKEN ``` * **Restore is now a point-in-time rollback**: Restore was previously additive. It’s now destructive: it truncates the WAL above the backup’s watermark so a restart doesn’t replay and resurrect post-backup data. * **In-place restore without a restart**: Restore can now apply to a running cluster. Supporting work includes gen0 buffer eviction, fencing peer writes on ingest-mode nodes during the restore, and evicting the query-node replica buffer so queriers stop serving the pre-restore view. * **Query performance (upgraded storage engine)**: `PachaTreeWindowExec` and`PachaTreeBufferExec` are now the default plan shape, giving visibility through `EXPLAIN` and `EXPLAIN ANALYZE`. Field family pruning skips loading field families that filters guarantee can’t affect the result. Scan predicate pushdown now extends across single-field-family scans, the union path, dedup and gap-leaf scans, and time predicates. * **Bulk import: remote sources and concurrency**: `influxdb3 import upload`now accepts an object store URL as a source, in addition to a local directory, and a `--concurrency` option to control how many files import at once (default 8). ``` influxdb3 import upload --database mydb --table events \ s3://my-bucket/exports/ \ --source-opt aws_region=us-west-2 \ --concurrency 16 ``` * **Five new system tables (upgraded storage engine)**: `system.pt_shards`,`system.pt_compaction_files`, `system.pt_storage_snapshots`,`system.pt_storage_checkpoints`, and `system.pt_storage_run_set_indexes`.`pt_ingest_wal` and `pt_ingest_files` gained `node_id` and `node_name`columns. * **`--user-auth-type` replaces `--without-user-auth`**: Configure the user authentication preview with a comma-separated list of `basic` and/or`oauth`, or `none` (the default). `--without-user-auth` is deprecated and hidden, but still takes precedence when explicitly set. * **Query group CLI commands (not yet operational)**: `influxdb3 create query_group`, `influxdb3 show query_groups`, `influxdb3 update query_group`, and `influxdb3 delete query_group` store query group definitions in the catalog, but the server doesn’t yet use those definitions to affect query routing, data placement, or replication. Creating a query group currently has no effect on query node behavior. #### Bug fixes * **Bulk import data loss window (checkpoint v13)**: Import watermarks are now embedded in the checkpoint itself, preventing data loss during a crash window. * **In-place restore stability**: Release-candidate validation found that live in-place restore could wedge queriers until restart, and that writing after a restore could trigger ID-recycling corruption. Both are fixed in this release. #### Breaking changes * **Upgraded storage engine options dropped the `pt-` prefix, with no aliases**: An old `--pt-*` flag now causes a startup error, and legacy`INFLUXDB3_PT_*` and `INFLUXDB3_ENTERPRISE_PT_*` environment variables are ignored — startup logs a warning for each one that’s still set. Review any saved command line, systemd unit, Docker Compose file, Helm chart, or packaged conf file before upgrading. Any `--pt-*` option not listed below simply drops the `pt-` prefix, and its environment variable follows the same pattern (`INFLUXDB3_PT_SNAPSHOT_SIZE`becomes `INFLUXDB3_SNAPSHOT_SIZE`). The following options changed beyond dropping the `pt-` prefix: | Old name | New name | |------------------------------|----------------------------| | `--pt-max-columns` | `--max-total-columns` | |`--pt-gen0-max-bytes-per-file`| `--gen0-max-file-size` | |`--pt-wal-replica-queue-size` |`--wal-replica-queue-length`| | `--pt-wal-max-buffer-size` | `--wal-buffer-size` | Every remaining upgraded-storage-engine byte-size flag ending in `-bytes`now ends in `-size` (the L1 through L4 tail and target-file flags). A few `--pt-*` flags map onto flag names shared with the Parquet engine, rather than getting their own new name — `--pt-wal-flush-interval` is now`--wal-flush-interval`, `--pt-wal-replication-interval` is now`--replication-interval`, `--pt-file-cache-size` is now `--file-cache-size`,`--pt-file-cache-recency` is now `--file-cache-recency`, and`--pt-disable-data-file-cache` is now `--disable-file-cache`.`--file-cache-size` is a total budget shared across both engines: during a storage engine upgrade with hybrid query enabled, the budget splits 50/50 between the two engine caches; otherwise the active engine gets the full budget. The upgraded storage engine’s flags no longer require `--use-pacha-tree`: in 3.10.x a `--pt-*` flag without `--use-pacha-tree` was a parse error, but with the upgraded storage engine as the default that constraint is gone. For the complete old-to-new name table, see[Migrate from `pt-` option names](/influxdb3/enterprise/performance-preview/configure/#migrate-from-pt-option-names). * **`INFLUXDB3_ENTERPRISE_*` environment variables become plain`INFLUXDB3_*`**: 35 Enterprise-specific environment variables dropped the`ENTERPRISE_` segment — for example, `INFLUXDB3_ENTERPRISE_CLUSTER_ID` is now `INFLUXDB3_CLUSTER_ID`, and `INFLUXDB3_ENTERPRISE_MODE` is now`INFLUXDB3_MODE`. The legacy `ENTERPRISE_` names remain supported as deprecated aliases; the server logs a deprecation warning at startup when it detects one, and if both names are set with different values, the new name wins. * **`--wait-for-running-ingestor` renamed to `--wait-for-running-ingester`**(the old name was a misspelling). The old option, and its`INFLUXDB3_WAIT_FOR_RUNNING_INGESTOR` /`INFLUXDB3_ENTERPRISE_WAIT_FOR_RUNNING_INGESTOR` environment variables, remain as deprecated, backward-compatible aliases. * **Metrics lost their per-database `db` label**: This breaks any dashboard or alert that groups by database. Affected series include`influxdb3_write_lines`, `influxdb3_write_lines_rejected`,`influxdb3_write_bytes`, `influxdb3_compactions`,`influxdb3_last_values_cache_query_duration`, and the upgraded-storage-engine`influxdb3_ingest_*` family (`lp_bytes_received`, `pre_snapshot_buffer_*`,`pre_wal_flush_buffer_*`, `write_concurrency`). * **`influxdb3 stop node` waits by default**: Previously, the server marked the node stopping and returned immediately, but the CLI claimed the node “has been stopped.” The CLI now polls until the node reads `stopped`.`--timeout` bounds the wait (default 5m), and `--no-wait` restores the old fire-and-forget behavior. On timeout, it prints the last observed state and recovery guidance, then exits non-zero. * **Parquet-only catalog limits are inert on the upgraded storage engine**: If`--num-database-limit`, `--num-table-limit`, or`--num-total-columns-per-table-limit` is set on a cluster running the upgraded storage engine, startup now warns that they’re ignored instead of hard-failing, since existing manifests carry them across the storage engine upgrade. Only the per-table column limit has an upgraded-engine counterpart (`--max-total-columns`). * **Four inert Enterprise options removed**: `--database-split-level`,`--table-split-level`, `--max-compact-destination`, and`--pt-enable-row-deletes` were parsed but never read; they are removed. The flags now fail to parse at startup and their environment variables are ignored. Remove them from your configuration. ## v3.10.5 ### Core #### Bug fixes * **Oversized buffer chunk persistence**: When a buffer chunk splits because a single string or tag column exceeds 2 GiB, each resulting chunk now persists to its own Parquet file. Previously, the split chunks all wrote to the same path. ### Enterprise All Core updates are included in Enterprise. ## v3.9.11 ### Core #### Bug fixes * **Oversized buffer chunk persistence**: When a buffer chunk splits because a single string or tag column exceeds 2 GiB, each resulting chunk now persists to its own Parquet file. Previously, the split chunks all wrote to the same path. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * Other bug fixes and performance improvements ## v3.10.4 ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Skip loading the compacted data file index**: The new `--compacted-data-skip-file-index` option (`INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX` environment variable, default `false`) loads compacted data without materializing the file index, letting nodes start when the index has grown too large to fit in host memory. Queries remain correct, but will run slower in exchange for a lower memory footprint. #### Bug fixes * **Skipped gen1 files in recompaction plans**: Gen1 files pulled into a recompaction plan that was skipped for exceeding the file limit are now carried into later plans’ leftover lists. Previously, when the recompaction loop produced multiple plans for one table in a single cycle, those files were dropped from the final persisted compaction detail; the files remained in object storage, but no query path would serve them. * Other bug fixes and performance improvements ## v3.9.10 ### Core Maintenance release: v3.9.10 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Skipped gen1 files in recompaction plans**: Gen1 files pulled into a recompaction plan that was skipped for exceeding the file limit are now carried into later plans’ leftover lists. Previously, when the recompaction loop produced multiple plans for one table in a single cycle, those files were dropped from the final persisted compaction detail; the files remained in object storage, but no query path would serve them. * Other bug fixes and performance improvements ## v3.9.9 ### Core #### Bug fixes * **Object store errors at startup**: When the catalog checkpoint existence check fails at startup, the underlying object store error is now logged and included in the reported error. Previously, the process exited with an opaque status code and no indication of the cause. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Skip loading the compacted data file index**: The new `--compacted-data-skip-file-index` option (`INFLUXDB3_ENTERPRISE_COMPACTED_DATA_SKIP_FILE_INDEX` environment variable, default `false`) loads compacted data without materializing the file index, letting nodes start when the index has grown too large to fit in host memory. Queries remain correct, but will run slower in exchange for a lower memory footprint. #### Bug fixes * Other bug fixes and performance improvements ## v3.10.3 ### Core #### Bug fixes * **Duplicate tag key rejection**: Writes that repeat a tag key (for example, `m,t=a,t=a f=1i`) are now rejected with a clear error, the same way duplicate field keys are rejected. Previously, a point with a repeated tag key was accepted into the WAL and later caused a panic during snapshotting that crash-looped the node on WAL replay. * **Processing engine trigger cancellation**: Disabling or deleting a trigger now cancels its in-flight plugin run in Core, extending the Enterprise fix from v3.10.2. Previously, a synchronous scheduled trigger whose plugin run was still executing could block trigger `disable` and `delete --force` operations until the run finished. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Compacted data load concurrency limit**: The new `--compacted-data-load-concurrency-limit` option (`INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT` environment variable, default `20`) bounds concurrent object store reads when a node loads compacted data at startup. Previously, nodes with large compaction indexes issued unbounded concurrent reads at startup, which could saturate the network, cause object store timeouts, and starve other subsystems of object store connections. #### Bug fixes * **Corrupt peer WAL and snapshot handling**: A durably corrupt WAL or snapshot file from a peer node is now logged, counted, and skipped so replication continues with later files. Previously, a corrupt peer WAL file stalled replication from that node—or prevented server startup—and a corrupt snapshot manifest silently halted snapshot replication from that peer. Transient errors, such as network failures, still retry as before. * Other bug fixes and performance improvements ## v3.9.8 ### Core #### Bug fixes * **Duplicate tag key rejection**: Writes that repeat a tag key (for example, `m,t=a,t=a f=1i`) are now rejected with a clear error, the same way duplicate field keys are rejected. Previously, a point with a repeated tag key was accepted into the WAL and later caused a panic during snapshotting that crash-looped the node on WAL replay. * **Processing engine trigger cancellation**: Disabling or deleting a trigger now cancels its in-flight plugin run in Core, extending the Enterprise fix from v3.9.7. Previously, a synchronous scheduled trigger (the default, created without `--run-asynchronous`) whose plugin run was still executing could block trigger `disable`, `delete --force`, and database deletion until the run finished. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Compacted data load concurrency limit**: The new `--compacted-data-load-concurrency-limit` option (`INFLUXDB3_ENTERPRISE_COMPACTED_DATA_LOAD_CONCURRENCY_LIMIT` environment variable, default `20`) bounds concurrent object store reads when a node loads compacted data at startup. Previously, nodes with large compaction indexes issued unbounded concurrent reads at startup, which could saturate the network, cause object store timeouts, and starve other subsystems of object store connections. #### Bug fixes * Other bug fixes and performance improvements ## v3.10.2 ### Core Maintenance release: v3.10.2 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Processing engine trigger cancellation**: Disabling or deleting a trigger now cancels its in-flight plugin run. Previously, a synchronous scheduled trigger whose plugin run was still executing could block trigger `disable` and `delete --force` operations until the run finished. * Other bug fixes and performance improvements ## v3.9.7 ### Core Maintenance release: v3.9.7 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Processing engine trigger cancellation**: Disabling or deleting a trigger now cancels its in-flight plugin run promptly. Previously, a synchronous scheduled trigger (the default, created without `--run-asynchronous`) whose plugin run was still executing could block trigger `disable`, `delete --force`, and even unrelated `create` operations until the run finished. * Other bug fixes and performance improvements ## v3.10.1 ### Core #### Bug fixes * **Snapshot manifest persistence**: Snapshot manifests are now persisted using multipart uploads, preventing errors when writing large manifests to object storage. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Compacted generation deduplication**: Overlapping compacted generations are now co-partitioned so the querier correctly deduplicates them. * **Performance upgrade preview file access**: A canceled file fetch no longer cascades cancellation to other waiters with the storage engine upgrade (`--use-pacha-tree`). * Other bug fixes and performance improvements ## v3.9.6 ### Core Maintenance release: v3.9.6 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Compacted generation deduplication**: Overlapping compacted generations are now co-partitioned so the querier correctly deduplicates them. * Other bug fixes and performance improvements ## v3.9.5 ### Core #### Bug fixes * **Snapshot manifest persistence**: Snapshot manifests are now persisted using multipart uploads, preventing errors when writing large manifests to object storage. ### Enterprise All Core updates are included in Enterprise. ## v3.10.0 > [!Important] > #### Upgrading to InfluxDB 3.10 is a one-way migration > > The first time you start InfluxDB 3.10, it automatically upgrades the on-disk > catalog format from v2 to v3. After migration, 3.9.x and older > binaries are unable to read the new catalog, and fail to start on the same > cluster data. > > Before upgrading, back up your current catalog. The paths depend on the version > you’re upgrading from: > > * **3.4.0 or later**: `{prefix}/catalog/v2/logs/` and `{prefix}/catalog/v2/snapshot` > * **Before 3.4.0**: `{prefix}/catalogs/` and `{prefix}/_catalog_checkpoint` > > Restoring these objects is the only way to roll back to 3.9.x. > > On a cluster running 3.4.0 or later, `{prefix}/catalogs/` and`{prefix}/_catalog_checkpoint` may still be present as leftovers from an earlier > catalog format. They aren’t current and aren’t a valid rollback source. > > If your cluster uses the upgraded storage engine (the default for new clusters, or after running the storage engine upgrade with `--upgrade-pacha-tree`), data written in the new `.pt` file format is also unreadable by 3.9.x. ### Core #### Features * **Catalog format upgrade (catalog v2 → v3)**: InfluxDB 3.10 automatically migrates the on-disk catalog to v3 format on first startup. The v3 catalog uses a compact binary record format (\~5–6x smaller than v2). Migration is automatic, idempotent, and crash-safe. **Back up your current catalog before upgrading — the migration is one-way and 3.9.x binaries cannot read a v3 catalog.** If you’re upgrading from 3.4.0 or later, back up `{prefix}/catalog/v2/logs/` and `{prefix}/catalog/v2/snapshot`. If you’re upgrading from a version before 3.4.0, back up `{prefix}/catalogs/` and `{prefix}/_catalog_checkpoint`. * **`influxdb3 debug catalog` command**: Inspect catalog state offline directly from object storage — no running server required. Subcommands: `list`, `snapshot`, `sequence`. Available in both Core and Enterprise. * **`--max-concurrent-queries` flag**: Limit the number of queries that run concurrently. The limit can also be updated at runtime via `POST /api/v3/configure/query_concurrency_limit`. * **Processing engine: cross-database queries**: Plugins can now read data from any database using the optional `database=` keyword argument on `influxdb3_local.query()`. * **Processing engine: trigger lockdown flags**: Two new serve flags restrict plugin behavior. `--restrict-plugin-triggers-to` limits triggers to one or more of `wal`, `schedule`, or `request`. `--plugin-dir-only` (Enterprise) blocks plugin installation from any source other than the configured plugin directory. * **Observability: always-on heap profiling**: Heap profiling is now enabled at startup with negligible overhead (\~\<1% CPU). Access profiles at the existing pprof endpoint. To disable, set `MALLOC_CONF=prof:false` before starting the server. * **Observability: per-request query traces**: Query tracing is now opt-in per request rather than enabled for all queries. This reduces trace volume for high-throughput deployments. See the monitoring documentation for how to enable tracing on individual requests. * **Embedded Python updated to 3.13.14**: The Processing engine’s embedded Python is updated to 3.13.14, which includes upstream security fixes. #### Bug fixes * **`/api/v2/write` returns 403 for unauthorized tokens**: A valid token that lacks write permission on the target database now receives `403 Forbidden` instead of `401 Unauthorized`. Update client-side retry logic if it differentiates on these status codes. * **Line-protocol parse errors return 400**: Malformed line protocol sent to the v1 `/write` or v2 `/api/v2/write` endpoints now returns `400 Bad Request` instead of `500 Internal Server Error`. * **Invalid queries return HTTP 4xx**: A syntactically invalid query now returns an appropriate 4xx response rather than a 5xx error. * **Query log records `query_text` on terminal phases**: The query log now includes the `query_text` field for queries that have reached a terminal phase. #### Breaking changes * **Catalog format upgrade (catalog v2 → v3) is one-way**: The first startup of InfluxDB 3.10 migrates the catalog to v3. After migration, 3.9.x binaries cannot start against the same object store. Back up your current catalog before upgrading: `{prefix}/catalog/v2/logs/` and `{prefix}/catalog/v2/snapshot` if you’re upgrading from 3.4.0 or later, or `{prefix}/catalogs/` and `{prefix}/_catalog_checkpoint` if you’re upgrading from an earlier version. * **`influxdb3 write` output changed**: The write command now prints a throughput report on success instead of printing `success`. Scripts that parse the previous output should use `--quiet` (`-q`) to suppress all output. * **`/api/v2/write` returns 403 instead of 401**: See bug fixes above. Clients that treat 401 and 403 differently must be updated. * **Line-protocol parse errors return 400 instead of 500**: See bug fixes above. * **Heap profiling is always on**: The \~\<1% CPU overhead is present by default. Opt out with `MALLOC_CONF=prof:false`. * **Query traces are now per-request opt-in**: Observability pipelines that expect a trace for every query will see far fewer traces. Update your pipeline to request traces explicitly per query. ### Enterprise All Core updates are included in Enterprise. The following updates are exclusive to Enterprise. #### Features * **Wide-tag support**: Tag IDs have been widened from u8 to u16. This raises the practical limit to thousands of tables and millions of columns per database. Available with the storage engine upgrade (`--use-pacha-tree`). * **Row-level deletion**: Delete rows by time range and tag predicates using `influxdb3 delete rows` and `influxdb3 cancel row-delete`. Deletion is asynchronous — requests persist to object storage and the compactor applies them when rewriting run sets. Requires `--use-pacha-tree`. Monitor pending deletes with the `system.row_deletes` system table and 9 new `influxdb3_compactor_row_delete_*` metrics. * **Runtime query-concurrency limit**: Adjust the maximum number of concurrent queries at runtime via the `/api/v3/configure/query_concurrency_limit` API — `GET` to read the current limit, `PUT` to set it, and `DELETE` to reset it to the startup default. * **`GET /ready` endpoint**: Returns `200 OK` when the server can reach object storage, or `503 Service Unavailable` when it cannot. Use this endpoint for readiness probes in load balancers and orchestration systems. * **Backup and restore**: Create and manage full backups of Enterprise data with `influxdb3 create backup`, `influxdb3 status backup`, `influxdb3 show backups`, `influxdb3 delete backup`, and `influxdb3 cancel backup`. Initiate restore operations with `influxdb3 create restore`, `influxdb3 status restore`, `influxdb3 show restores`, and `influxdb3 cancel restore`. Backup and restore require `--use-pacha-tree` and a compactor node with an admin token. `create backup` refuses to overwrite an existing backup. Only one restore runs at a time across the cluster. After a restore completes, restart the node(s) for the in-memory view to update. API: `POST|GET|DELETE /api/v3/enterprise/backup[/{name}]` and `/api/v3/enterprise/restore[/{id}]`. * **Bulk import**: Import generic (non-IOx) Parquet files into Enterprise with `influxdb3 import upload`. Map Parquet columns to InfluxDB types (`i64`, `u64`, `f64`, `bool`, `string`, `time`, `tag`) using `--column` flags. Unmapped columns become fields. List in-progress and completed import jobs with `influxdb3 import list`. The target database and table must exist before importing. * **User auth and RBAC preview**: Multi-user authentication is now available as a preview feature. It is off by default (`--without-user-auth true`). When enabled, users authenticate with username and password to receive JWTs. Optional OAuth/OIDC is supported. Three built-in roles are available: Admin, Auditor, and Member. New CLI commands: `influxdb3 auth login`, `influxdb3 auth logout` (removes local credentials; does not revoke the signed JWT), `influxdb3 auth reset-password`, `influxdb3 create user`, `influxdb3 show users`, `influxdb3 update user`, `influxdb3 update user-roles`, `influxdb3 delete user`, `influxdb3 user require-password-reset`. New API endpoints: * `POST /api/v3/configure/user` — configure the initial user and create the operator token (also used by `influxdb3 manage init-admin`) * `POST /api/v3/authorize` — authenticate and obtain tokens * `POST /api/v3/authorize/refresh` — refresh an access token using a refresh token * `POST /api/v3/authorize/reset-password` — reset password using current credentials * `GET /api/v3/users`, `POST /api/v3/users` — list or create users (Enterprise) * `GET /api/v3/users/{id}`, `PATCH /api/v3/users/{id}`, `DELETE /api/v3/users/{id}` — get, update, or delete a user * `POST /api/v3/users/{id}/require-password-reset` — force password reset on next login * `GET /api/v3/users/{id}/roles`, `PUT /api/v3/users/{id}/roles` — read or replace a user’s roles * `GET /api/v3/roles` — list available roles * `GET /api/v3/auth/oauth/config` — discover OAuth configuration for device-code login New serve flags: `--without-user-auth`, `--jwt-key-id`, `--jwt-private-key`, `--jwt-issuer`, `--jwt-default-ttl-seconds`, `--oauth-issuer`, `--oauth-audience`, `--oauth-client-id`, `--oauth-scopes`, and `--rbac-authoring-disabled`. JWT keys must be PKCS#1 format (`openssl genrsa -traditional`). PKCS#8 format silently fails. * **`influxdb3 manage` command group**: A new `manage` subcommand groups offline administrative operations: `influxdb3 manage init-admin`, `influxdb3 manage add-admin-token`, and `influxdb3 manage downgrade-to-parquet`. The `downgrade-to-parquet` command has moved from the top level to this group (the old spelling still works but prints a deprecation warning). * **`influxdb3 remove node` command**: Remove a stopped node from the catalog. The compactor drains the node’s data before removal completes. * **Service-level logs**: Structured query and storage logging is now available for observability. Configure log output format and levels using new `serve` flags. * **Processing engine: internode gRPC for plugin writes**: Plugin writes from non-ingester nodes now route over internode gRPC rather than HTTP. This improves reliability in multi-node clusters. Requires `--internode-bind-addr` and `--conn-info` pointing at the gRPC port. * **Licensing: object-store portability**: Enterprise licenses are no longer bound to the object-store configuration (type, bucket, endpoint, region). Validation now enforces only JWT signature, expiry, and licensed core count. You can move to a different bucket or store with the same license. When moving to an empty store, copy `{cluster-id}/commercial_license` from the old store or restart with `--license-file`. * **Observability: 36 new compactor metrics**: 36 new `influxdb3_compactor_*` Prometheus metrics are now emitted. The primary health signal is `influxdb3_compactor_snapshot_lag_seconds`. A new `influxdb3_compaction_sequence_number` gauge tracks Parquet engine lag. * **`influxdb3 debug object-store-check` command**: Validate S3-compatible backend semantics before putting a store into production. Checks that the backend correctly implements the operations that InfluxDB relies on. #### Bug fixes * **Compaction stability**: Several compaction bugs are fixed, including: compaction incorrectly setting `ingest_time` (causing deduplication and row delete bugs), compactor deadlock and write amplification, stopped compactor nodes blocking storage engine upgrades, and compactor orphaning gen1 files. * **Tag case preserved during storage engine upgrades**: Tag names now preserve their original case when upgrading from Parquet to the new storage engine. * **Bulk import memory usage reduced**: Peak memory during multi-file bulk import operations is significantly reduced. * **Last cache delete deadlock fixed**: Deleting a last-value cache entry no longer causes a deadlock. * **Row delete: aborted requests no longer processed**: Row delete requests that were aborted are no longer picked up by the compactor. * **Table and database soft-delete name collision fixed**: Deleting a table or database and recreating it with the same name now works correctly. * **TLS CA flag cleanup**: The `serve` command no longer accepts `--tls-ca` — it was non-functional there. Client commands (such as `query` and `write`) still accept `--tls-ca` to trust a custom or self-signed CA, and the flag is now consistently bound to the `INFLUXDB3_TLS_CA` environment variable across commands that were previously missing the binding. The `cancel row-delete` command now also accepts TLS options. #### Breaking changes * **`influxdb3 row-delete` → `influxdb3 delete rows` and `influxdb3 cancel row-delete`**: The old `row-delete` top-level command is removed. Update scripts to use the new `delete rows` and `cancel row-delete` subcommands. * **`--conn-info` must point to the internode gRPC port for plugin writes**: In multi-node deployments, `--conn-info` must now reference the internode gRPC port (not the HTTP port) for plugin writes to reach the ingester. Update your cluster configuration before upgrading. * **PT compactor stale-job timeout changed from 5 minutes to 1 hour**: Compactor jobs that appear stuck take up to 1 hour to be retried (previously 5 minutes). This reduces false-positive preemption on slow storage backends. * **`--help-full` removed**: The `--help-full` flag is no longer available. Update any scripts that invoke `influxdb3 --help-full`. * **`--package-manager` flag deprecated**: The `uv` package manager has been removed. `pip` is always used for plugin package installation. The `--package-manager` flag still starts the server but prints a deprecation warning. Remove it from your startup configuration. * **`--pt-partition-count` renamed to `--pt-shard-count`**: The flag has no alias. Update any startup scripts that pass `--pt-partition-count` before upgrading to 3.10. * **System table columns renamed**: The following columns in storage engine system tables are renamed. Update any dashboards or queries that reference the old names: * `partition_id` → `shard_id` * `partition_start_time` → `shard_start_time` ### Known issues * **Row delete ghost rows**: After a row delete reports as “completed,” rows in the un-compacted ingest tail can survive and remain visible in queries. Workaround: re-issue the delete request after the affected data has been compacted and verify row counts. * **`system.row_deletes` returns HTTP 500 for predicate-less `--all-time` deletes**: Querying the `system.row_deletes` system table after a delete issued with `--all-time` and no tag predicate may return HTTP 500. Workaround: use `GET /api/v3/row_delete_requests` instead. * **Multi-shard data loss with `--use-pacha-tree`**: When the `--use-pacha-tree` storage engine is enabled, running with more than one shard (`--pt-shard-count > 1`) can cause data loss and a bootstrap deadlock. Workaround: keep `--pt-shard-count` at `1`. * **Backup does not capture row-delete state**: Backup (beta) doesn’t currently pick up row-delete state files in object storage, so row deletes may persist across a restore. * **Built-in roles grant narrower access than their descriptions suggest**: With the user authentication preview enabled, the Auditor and Member roles enforce less access than their role descriptions imply. Auditor users can list databases but cannot query data or read users or roles. Member users can read and write data but cannot list users or roles. Workaround: use an Admin-role user or an admin token for user and role management. ## v3.9.3 ### Core Maintenance release: v3.9.3 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Query chunk deduplication**: Fixed an issue where the same file could reach the query path from both the compactor and the ingester, causing affected queries to abort. * **Large file uploads during compaction**: Index files written during compaction now use adaptive uploads, preventing errors when writing large files to object storage. * Other bug fixes and performance improvements ## v3.9.2 ### Core Maintenance release: v3.9.2 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Bug fixes * **Gen1 file deduplication in compactor**: Fixed an issue where stale snapshot markers after `CompactionSummary` recovery could leave duplicate gen1 file entries and cause recompaction to abort. * **Empty series key handling**: Fixed compaction for tables with no tags (empty series key). * **Catalog token hash lookup**: Fixed a case where a failed `add_token` insert could leave a stale entry in the token hash lookup map. The lookup is now only updated after the underlying repository insert succeeds. * Other bug fixes and performance improvements ## v3.9.1 ### Core Maintenance release: v3.9.1 Core includes only build and dependency updates—no user-facing changes. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific updates: #### Features * **Configurable compactor snapshot loading**: The number of snapshots the Parquet compactor loads at startup is now externally configurable, making it easier to tune recovery behavior for large deployments. #### Bug Fixes and Performance Improvements * **Performance Improvements**: This release features faster multi-source query merges and improved retention scheduling with the new Performance Update Preview. * **Bug Fixes**: New updates fix issues where duplicate rows could be returned, Gen0 pruning safety, invalid status codes, and more. ## v3.9.0 ### Core #### Features * **DataFusion upgrade**: Upgraded the embedded DataFusion query engine for more efficient query execution. * **Python runtime upgrade**: Updated the bundled Python runtime for processing engine plugins with the latest security and bug fixes. * **Product identity in HTTP responses**: Metrics, HTTP response headers, and metadata now distinguish between Core and Enterprise builds. * **Database lifecycle hardening**: Background resources such as processing engine triggers are now cleanly decommissioned when a database is removed. #### Bug fixes * Additional bug fixes and performance improvements. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **Performance upgrade preview (beta)**: Preview major storage layer upgrades with the `--use-pacha-tree` flag. Includes a new columnar file format (`.pt` files), automatic Parquet migration with hybrid query mode, column families for efficient wide-table I/O, and bounded compaction. See [Performance upgrade preview](/influxdb3/enterprise/performance-preview/). The performance upgrade preview is a beta feature for staging and test environments only. Do not use for production workloads. * **Bulk data export**: Export compacted data as Parquet files for use with external tools. Use the new `influxdb3 export` subcommands to list databases, tables, and compacted time windows, then export selected data. See [Export to Parquet](/influxdb3/enterprise/performance-preview/#export-to-parquet). * **Automatic distinct value caching**: Enable automatic DVC creation for`SHOW TAG VALUES` queries and the `tag_values()` SQL function with`--pt-enable-auto-dvc`. Max cardinality and refresh intervals are configurable. * **Downgrade from performance preview**: Use`influxdb3 downgrade-to-parquet` to revert from the performance preview back to standard Parquet storage. Only data that existed before the upgrade (original Parquet files) is preserved. See [Downgrade to Parquet](/influxdb3/enterprise/performance-preview/#downgrade-to-parquet). * **Non-interactive delete confirmation**: Use the `--yes` (`-y`) flag with delete commands to skip interactive confirmation prompts in automated and headless environments. * **1MB default string field limit**: The maximum string field size defaults to 1MB (previously 64KB) to support v1 migration workloads. Writes exceeding 1MB are rejected with a validation error. #### Bug fixes * **Compaction stability**: Multiple fixes to compaction scheduling, priority handling, and resource management for improved stability in multi-node clusters. * Additional bug fixes and performance improvements. ## v3.8.4 ### Core No adjustments in this release. Core remains on v3.8.3. ### Enterprise #### Security * **Read and write tokens can no longer delete databases**: Authorization now evaluates both the HTTP method and the request path. Previously, tokens with read or write access to a database could also issue delete requests. #### Bug fixes * **Stale compactor blocking startup**: Fixed an issue where stopped (stale) compactor entries in the catalog prevented new compactor nodes from starting. Enterprise now only considers currently running compactor nodes for conflict checks. * **WAL replay**: Fixed an issue where combined-mode deployments silently ignored the `--wal-replay-concurrency-limit` flag and always used serial replay (concurrency of 1). The flag is now respected. * Other bug fixes and performance improvements. ## v3.8.3 ### Core #### Bug fixes * **WAL Buffer**: Fix an edge case that could potentially cause the WAL buffer to overflow ## v3.8.2 ### Core #### Features * **TLS: Skip certificate verification in CLI subcommands**: Use the new `--tls-no-verify` flag with any CLI subcommand to skip TLS certificate verification when connecting to a server. Useful for testing environments with self-signed certificates. * **Environment variable prefix standardization**: InfluxDB 3 specific environment variables use the `INFLUXDB3_` prefix for consistency. Legacy variable names continue to work (deprecated) for backward compatibility. > [!Important] > `INFLUXDB3_LOG_FILTER` is currently ignored. To set the log filter, use `LOG_FILTER` or the `--log-filter` flag. * **Parquet output format for `show` subcommands**: You can now save query results from the `show` subcommand directly to a Parquet file. * **SQL: `tag_values()` table function**: Query distinct tag values using the new `tag_values()` SQL table function. * **InfluxQL: `SHOW TAG VALUES` improvements**: In Enterprise deployments with auto-DVC enabled, `SHOW TAG VALUES` queries now use the Distinct Value Cache (DVC) automatically for improved performance. The `WHERE` clause is also now supported in `SHOW TAG VALUES` queries backed by the DVC, including compound predicates using `AND` and `OR`. * **InfluxQL: `SHOW RETENTION POLICIES` returns duration**: The `duration` column in `SHOW RETENTION POLICIES` results now returns the configured retention period in InfluxDB v1-compatible format (for example, `168h0m0s`) instead of returning an empty value. * **Ceph S3 backend support**: Use `--aws-s3-custom-backend ceph` with `influxdb3 serve` to connect to Ceph S3-compatible object storage. This enables ETag quote stripping required for conditional PUT operations with Ceph. * **`_internal` database default retention**: The `_internal` system database now defaults to a 7-day retention period (previously infinite). Only admin tokens can modify retention on the `_internal` database. * **Snapshot checkpointing for faster startup**: Use the new [`--checkpoint-interval`](/influxdb3/enterprise/reference/config-options/#checkpoint-interval) serve option to periodically consolidate snapshots into monthly checkpoints. On startup, the server loads one to two checkpoints per calendar month instead of thousands of individual snapshots, reducing startup time for long-running servers. #### Bug fixes * **Sparse write handling for LVC, DVC, and Processing Engine**: Fixed incorrect behavior when processing sparse writes (writes that include only some fields from a table with multiple field families). * **`influxdb3-launcher`: SSL certificate path on RHEL systems**: Fixed an issue where the `SSL_CERT_FILE` environment variable was not correctly set on affected RHEL-based systems when using the `influxdb3-launcher` script. * Additional bug fixes and performance improvements. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **Data-only deletion for databases and tables**: Delete only the stored data from a database or table while preserving catalog entries, schema, and associated resources (tokens, triggers, caches, and processing engine configurations). #### Bug fixes * **Compaction stability**: Several fixes to compaction scheduling and processing to improve stability and correctness in multi-node clusters. * **TableIndexCache initialization**: Fixed a concurrency bug that could cause incorrect behavior during `TableIndexCache` initialization. * **Snapshot checkpointing**: Fixed an issue where snapshot checkpoint cleanup was not running as a background task. ## v3.8.0 ### Core #### Features * **Linux Service Management**: Run InfluxDB 3 as a managed system service on Linux ([#27026](https://github.com/influxdata/influxdb/pull/27026)): * Use `influxdb3-launcher` script to initialize the service * Deploy with systemd on modern Linux distributions * Deploy with SysV init on legacy systems * Customize service behavior with configuration files #### Bug fixes * **CLI**: View only active databases and tables when running `SHOW RETENTION` * **Database operations**: Receive an error when attempting to delete tables from an already-deleted database * **Retention Policy**: Receive an error when attempting to modify retention settings on deleted databases #### Security * **Processing Engine**: Run processing engine plugins with Python 3.13.11, which includes security and bug fixes ([#27014](https://github.com/influxdata/influxdb/pull/27014)) ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Bug fixes * **Table Limits**: Delete tables without affecting your table limit quota * **Retention Policy**: Receive an error when attempting to modify retention settings on deleted tables ## v3.7.0 ### Core #### Features * **HTTP API Enhancements**: * All HTTP responses now include a `cluster-uuid` header containing the catalog UUID, enabling clients to identify specific cluster instances programmatically * HTTP API now supports multi-member gzip payloads enabling batch operations * **CLI Commands**: * The new `influxdb3 show retention` command displays effective retention periods for each table, showing whether retention is set at the database-level or table-level with human-readable formatting (for example, “7d”, “24h”) #### Bug fixes * **Authorization**: Fixed multi-database permission handling to properly authorize queries across multiple databases. * **General Improvements**: Several key bug fixes and performance improvements. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: * **General Improvements**: Several key bug fixes and performance improvements. ## v3.6.0 ### Core #### Features * **Quick-Start Developer Experience**: * `influxdb3` now supports running without arguments for instant database startup, automatically generating IDs and storage flags values based on your system’s setup. * **Processing Engine**: * Plugins now support multiple files instead of single-file limitations. * When creating a trigger, you can upload a plugin directly from your local machine using the `--upload` flag. * Existing plugin files can now be updated at runtime without recreating triggers. * New `system.plugin_files` table and `show plugins` CLI command now provide visibility into all loaded plugin files. * Custom plugin repositories are now supported via `--plugin-repo` CLI flag. * Python package installation can now be disabled with `--package-manager disabled` for locked-down environments. * Plugin file path validation now prevents directory traversal attacks by blocking relative and absolute path patterns. #### Bug fixes * **Write API**: Fixed abbreviated precision values (`ns`, `ms`, `us`, `s`) to work correctly with the `/api/v3/write_lp` endpoint. Previously, only full precision names (`nanosecond`, `microsecond`, `millisecond`, `second`) worked. * **Token management**: Token display now works correctly for hard-deleted databases ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Operational improvements * **Storage engine**: improvements to the Docker-based license service development environment * **Catalog consistency**: Node management fixes for catalog edge cases * Other enhancements and performance improvements ## v3.5.0 ### Core #### Features * **Custom Plugin Repository**: * Use the `--plugin-repo` option with `influxdb3 serve` to specify custom plugin repositories. This enables loading plugins from personal repos or disabling remote repo access. #### Bug fixes * **Database reliability**: * Table index updates now complete atomically before creating new indices, preventing race conditions that could corrupt database state ([#26838](https://github.com/influxdata/influxdb/pull/26838)) * Delete operations are now idempotent, preventing errors during object store cleanup ([#26839](https://github.com/influxdata/influxdb/pull/26839)) * **Write path**: * Write operations to soft-deleted databases are now rejected, preventing data loss ([#26722](https://github.com/influxdata/influxdb/pull/26722)) * **Runtime stability**: * Fixed a compatibility issue that could cause deadlocks for concurrent operations ([#26804](https://github.com/influxdata/influxdb/pull/26804)) * Other bug fixes and performance improvements #### Security & Misc * Sensitive environment variable values are now hidden in CLI output and log messages ([#26837](https://github.com/influxdata/influxdb/pull/26837)) ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **Cache optimization**: * Last Value Cache (LVC) and Distinct Value Cache (DVC) now populate on creation and only on query nodes, reducing resource usage on ingest nodes. #### Bug fixes * **Object store reliability**: * Object store operations now use retryable mechanisms with better error handling #### Operational improvements * **Compaction optimizations**: * Compaction producer now waits 10 seconds before starting cycles, reducing resource contention during startup * Enhanced scheduling algorithms distribute compaction work more efficiently across available resources * **System tables**: * System tables now provide consistent data across different node modes (ingest, query, compact), enabling better monitoring in multi-node deployments ## v3.4.2 ### Core #### Bug fixes * **Database reliability**: * TableIndexCache initialization and ObjectStore improvements * Persister doesn’t need a TableIndexCache #### HTTP API changes * **v2 write API**: Standardized `/api/v2/write` error response format to match other InfluxDB editions. Error responses now use the consistent format: `{"code": "", "message": ""}` ([#26787](https://github.com/influxdata/influxdb/pull/26787)) ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **Storage engine**: Pass in root CA and disable TLS verify for object store * **Support**: Add support for manually stopping a node #### Bug fixes * **Bug fix**: Generation detail path calculation panic * **Database reliability**: Pass TableIndexCache through to PersistedFiles #### Operational improvements * **Compaction optimizations**: * Compaction cleaner now waits for 1 hour by default (previously 10 minutes) * Compaction producer now waits for 10 seconds before starting compaction cycle * **Catalog synchronization**: Background catalog update is synchronized every 1 second (previously 10 seconds) * **Logging improvements**: Added clear logging to indicate what sequence is persisted on producer side and what is consumed by the consumer side ## v3.4.1 ### Core #### Bug Fixes * Upgrading from 3.3.0 to 3.4.x no longer causes possible catalog migration issues ([#26756](https://github.com/influxdata/influxdb/pull/26756)) ## v3.4.0 ### Core #### Features * **Token Provisioning**: * Generate admin tokens offline and use them when starting the database if tokens do not already exist. This is meant for automated deployments and containerized environments. ([#26734](https://github.com/influxdata/influxdb/pull/26734)) * **Azure Endpoint**: * Use the `--azure-endpoint` option with `influxdb3 serve` to specify the Azure Blob Storage endpoint for object store connections. ([#26687](https://github.com/influxdata/influxdb/pull/26687)) * **No\_Sync via CLI**: * Use the `--no-sync` option with `influxdb3 write` to skip waiting for WAL persistence on write and immediately return a response to the write request. ([#26703](https://github.com/influxdata/influxdb/pull/26703)) #### Bug Fixes * Validate tag and field names when creating tables ([#26641](https://github.com/influxdata/influxdb/pull/26641)) * Using GROUP BY twice on the same column no longer causes incorrect data ([#26732](https://github.com/influxdata/influxdb/pull/26732)) #### Operational and security improvements * Introduce a new `v2` catalog path structure: * `catalog/v2/logs/` directory for log files (instead of `catalogs/`) * `catalog/v2/snapshot` file for checkpoint/snapshot files (instead of `_catalog_checkpoint`) * Reduce verbosity of the TableIndexCache log. ([#26709](https://github.com/influxdata/influxdb/pull/26709)) * WAL replay concurrency limit defaults to number of CPU cores, preventing possible OOMs. ([#26715](https://github.com/influxdata/influxdb/pull/26715)) * Remove unsafe signal\_handler code. ([#26685](https://github.com/influxdata/influxdb/pull/26685)) * Upgrade Python version to 3.13.7-20250818. ([#26686](https://github.com/influxdata/influxdb/pull/26686), [#26700](https://github.com/influxdata/influxdb/pull/26700)) * Tags with `/` in the name no longer break the primary key. ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **Token Provisioning**: * Generate *resource* and *admin* tokens offline and use them when starting the database. * Select a home or trial license without using an interactive terminal. Use `--license-type` [home | trial | commercial] option to the `influxdb3 serve` command to automate the selection of the license type. #### Bug Fixes * Don’t initialize the Processing Engine when the specified `--mode` does not require it. * Don’t panic when `INFLUXDB3_PLUGIN_DIR` is set in containers without the Processing Engine enabled. ## v3.3.0 ### Core #### Features * **Database management**: * Add `influxdb_schema` system table for database schema management ([#26640](https://github.com/influxdata/influxdb/pull/26640)) * Add `system.processing_engine_trigger_arguments` table for trigger configuration management ([#26604](https://github.com/influxdata/influxdb/pull/26604)) * Add write path logging to capture database name and client IP address for failed writes. The IP address is fetched from `x-forwarded-for` header if available, `x-real-ip` if available, or remote address as reported by TlsStream/AddrStream ([#26616](https://github.com/influxdata/influxdb/pull/26616)) * **Storage engine**: Introduce `TableIndexCache` for efficient automatic cleanup of expired gen1 Parquet files based on retention policies and hard deletes. Includes new background loop for applying data retention policies with configurable intervals and comprehensive purge operations for tables and retention period expired data ([#26636](https://github.com/influxdata/influxdb/pull/26636)) * **Authentication and security**: Add admin token recovery server that allows regenerating lost admin tokens without existing authentication. Includes new `--admin-token-recovery-http-bind` option for running recovery server on separate port, with automatic shutdown after successful token regeneration ([#26594](https://github.com/influxdata/influxdb/pull/26594)) * **Build process**: Allow passing git hash via environment variable in build process ([#26618](https://github.com/influxdata/influxdb/pull/26618)) #### Bug Fixes * **Database reliability**: * Fix URL-encoded table name handling failures ([#26586](https://github.com/influxdata/influxdb/pull/26586)) * Allow hard deletion of existing soft-deleted schema ([#26574](https://github.com/influxdata/influxdb/pull/26574)) * **Authentication**: Fix AWS S3 API error handling when tokens are expired ([#1013](https://github.com/influxdata/influxdb/pull/1013)) * **Query processing**: Set nanosecond precision as default for V1 query API CSV output ([#26577](https://github.com/influxdata/influxdb/pull/26577)) * **CLI reliability**: * Mark `--object-store` CLI argument as explicitly required ([#26575](https://github.com/influxdata/influxdb/pull/26575)) * Add help text for the new update subcommand ([#26569](https://github.com/influxdata/influxdb/pull/26569)) ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **License management**: * Improve licensing suggestions for Core users * Update license information handling * **Database management**: * Enhance `TableIndexCache` with advanced features beyond Core’s basic cleanup: persistent snapshots, object store integration, merge operations for distributed environments, and recovery capabilities for multi-node clusters * Add `TableIndexSnapshot`, `TableIndex`, and `TableIndices` types for distributed table index management * **Support**: Include contact information in trial error messages * **Telemetry**: Send onboarding telemetry before licensing setup #### Bug Fixes * **Compaction stability**: * Fix compactor re-compaction issues on max generation data overwrite * Fix compactor to treat “all” mode as “ingest” mode * **Database reliability**: * Add missing system tables to compact mode * **Storage integrity**: Update Parquet file paths to use 20 digits of 0-padding * **General fixes**: * Only load processing engine in correct server modes * Remove load generator alias clash ## v3.2.1 ### Core #### Features * **Enhanced database lifecycle management**: * Allow updating the hard deletion date for already-deleted databases and tables, providing flexibility in managing data retention and compliance requirements * Include `hard_deletion_date` column in `_internal` system tables (`databases` and `tables`) for better visibility into data lifecycle and audit trails #### Bug Fixes * **CLI improvements**: * Added help text for the new `update` subcommand for database and table update features ([#26569](https://github.com/influxdata/influxdb/pull/26569)) * `--object-store` and storage configuration parameters are required for the `serve` command ([#26575](https://github.com/influxdata/influxdb/pull/26575)) * **Query processing**: Fixed V1-compatible `/query` HTTP API endpoint to correctly default to nanosecond precision (`ns`) for CSV output, ensuring backward compatibility with InfluxDB 1.x clients and preventing data precision loss ([#26577](https://github.com/influxdata/influxdb/pull/26577)) * **Database reliability**: Fixed issue preventing hard deletion of soft-deleted databases and tables, enabling complete data removal for compliance and storage management needs ([#26574](https://github.com/influxdata/influxdb/pull/26574)) ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **License management improvements**: New `influxdb3 show license` command displays detailed license information including type, expiration date, and resource limits, making it easier to monitor license status and compliance #### Bug Fixes * **API stability**: Fixed HTTP API trigger specification to use the correct `"request:REQUEST_PATH"` syntax, ensuring proper request-based trigger configuration for processing engine workflows ## v3.2.0 **Core**: revision 1ca3168bee **Enterprise**: revision 1ca3168bee ### Core #### Features * **Hard delete for databases and tables**: Permanently delete databases and tables, enabling complete removal of data structures for compliance and storage management ([#26553](https://github.com/influxdata/influxdb/pull/26553)) * **AWS credentials auto-reload**: Support dynamic reloading of ephemeral AWS credentials from files, improving security and reliability when using AWS services ([#26537](https://github.com/influxdata/influxdb/pull/26537)) * **Database retention period support**: Add retention period support for databases via CLI commands (`create database` and `update database` commands) and HTTP APIs ([#26520](https://github.com/influxdata/influxdb/pull/26520)): * New CLI command: `update database --retention-period` * **Configurable lookback duration**: Users can specify lookback duration for PersistedFiles buffer, providing better control over query performance ([#26528](https://github.com/influxdata/influxdb/pull/26528)) * **WAL replay concurrency control**: Add concurrency limits for WAL (Write-Ahead Log) replay to improve startup performance and resource management ([#26483](https://github.com/influxdata/influxdb/pull/26483)) * **Enhanced write path**: Separate write path executor with unbounded memory for improved write performance ([#26455](https://github.com/influxdata/influxdb/pull/26455)) #### Bug Fixes * **WAL corruption handling**: Handle corrupt WAL files during replay without panic, improving data recovery and system resilience ([#26556](https://github.com/influxdata/influxdb/pull/26556)) * **Database naming validation**: Disallow underscores in database names when created via API to ensure consistency ([#26507](https://github.com/influxdata/influxdb/pull/26507)) * **Object store cleanup**: Automatic intermediate directory cleanup for file object store, preventing storage bloat ([#26480](https://github.com/influxdata/influxdb/pull/26480)) #### Additional Updates * Track generation 1 duration in catalog for better performance monitoring ([#26508](https://github.com/influxdata/influxdb/pull/26508)) * Add retention period support to the catalog ([#26479](https://github.com/influxdata/influxdb/pull/26479)) * Update help text for improved user experience ([#26509](https://github.com/influxdata/influxdb/pull/26509)) ### Enterprise All Core updates are included in Enterprise. Additional Enterprise-specific features and fixes: #### Features * **License management improvements**: * New `influxdb3 show license` command to display current license information * **Table-level retention period support**: Add retention period support for individual tables in addition to database-level retention, providing granular data lifecycle management * New CLI commands: `create table --retention-period` and `update table --retention-period` * Set or clear table-specific retention periods independent of database settings * **Compaction improvements**: * Address compactor restart issues for better reliability * Track compacted generation durations in catalog for monitoring * Disable Parquet cache for ingest mode to optimize memory usage #### Bug Fixes * **Query optimization**: Correctly partition query chunks into generations for improved performance * **Data integrity**: Don’t delete generation 1 files as part of compaction process * **License handling**: Trim whitespace from license file contents after reading to prevent validation issues ## v3.1.0 **Core**: revision `482dd8aac580c04f37e8713a8fffae89ae8bc264` **Enterprise**: revision `2cb23cf32b67f9f0d0803e31b356813a1a151b00` ### Core #### Token and Security Updates * Named admin tokens can now be created, with configurable expirations * `health`, `ping`, and `metrics` endpoints can now be opted out of authorization * `Basic $TOKEN` is now supported for all APIs * Additional info available when creating a new token * Additional info available when starting InfuxDB using `--without-auth` #### Additional Updates * New catalog metrics available for count operations * New object store metrics available for transfer latencies and transfer sizes * New query duration metrics available for Last Value caches * `/ping` API now contains versioning headers * Other performance improvements #### Fixes * New tags are now backfilled with NULL instead of empty strings * Bitcode deserialization error fixed * Series key metadata not persisting to Parquet is now fixed * Other general fixes and corrections ### Enterprise #### Token and Security Updates * Resource tokens now use resource names in `show tokens` * Tokens can now be granted `CREATE` permission for creating databases #### Additional Updates * Last value caches reload on restart * Distinct value caches reload on restart * Other performance improvements * Replaces remaining “INFLUXDB\_IOX” Dockerfile environment variables with the following: * `ENV INFLUXDB3_OBJECT_STORE=file` * `ENV INFLUXDB3_DB_DIR=/var/lib/influxdb3` #### Fixes * Improvements and fixes for license validations * False positive fixed for catalog error on shutdown * UX improvements for error and onboarding messages * Other general fixes and corrections ## v3.0.3 **Core**: revision 384c457ef5f0d5ca4981b22855e411d8cac2688e **Enterprise**: revision 34f4d28295132b9efafebf654e9f6decd1a13caf ### Core #### Fixes * Prevent operator token, `_admin`, from being deleted. ### Enterprise #### Fixes * Fix object store info digest that is output during onboarding. * Fix issues with false positive catalog error on shutdown. * Fix licensing validation issues. * Other fixes and performance improvements. ## v3.0.2 **Core**: revision d80d6cd60049c7b266794a48c97b1b6438ac5da9 **Enterprise**: revision e9d7e03c2290d0c3e44d26e3eeb60aaf12099f29 ### Core #### Security updates * Generate testing TLS certificates on the fly. * Set the TLS CA via the INFLUXDB3\_TLS\_CA environment variable. * Enforce a minimum TLS version for enhanced security. * Allow CORS requests from browsers. #### General updates * Support the `--format json` option in the token creation output. * Remove the Last Values Cache size limitation to improve performance and flexibility. * Incorporate additional performance improvements. #### Fixes * Fix a counting bug in the distinct cache. * Fix how the distinct cache handles rows with null values. * Fix handling of `group by` tag columns that use escape quotes. * Sort the IOx table schema consistently in the `SHOW TABLES` command. ### Enterprise #### Updates * Introduce a command and system table to list cluster nodes. * Support multiple custom permission argument matches. * Improve overall performance. #### Fixes * Initialize the object store only once. * Prevent the Home license server from crashing on restart. * Enforce the `--num-cores` thread allocation limit. ## v3.0.1 **Core**: revision d7c071e0c4959beebc7a1a433daf8916abd51214 **Enterprise**: revision 96e4aad870b44709e149160d523b4319ea91b54c ### Core #### Updates * TLS CA can now be set with an environment variable: `INFLUXDB3_TLS_CA` * Other general performance improvements #### Fixes * The `--tags` argument is now optional for creating a table, and additionally now requires at least one tag *if* specified ### Enterprise #### Updates * Catalog limits for databases, tables, and columns are now configurable using `influxdb3 serve` options: * `--num-database-limit` * `--num-table-limit` * `--num-total-columns-per-table-limit` * Improvements to licensing prompts for clarity * Other general performance improvements #### Fixes * **Home** license thread count log errors ## v3.0.0 ### Core #### Breaking Changes * **Parquet cache configuration**: Replaced `--parquet-mem-cache-size-mb` option with `--parquet-mem-cache-size`. The new option accepts values in megabytes (as an integer) or as a percentage of total available memory (for example, `20%`). The default value changed from `1000` MB to `20%` of total available memory. The environment variable `INFLUXDB3_PARQUET_MEM_CACHE_SIZE_MB` was replaced with `INFLUXDB3_PARQUET_MEM_CACHE_SIZE`. ([#26023](https://github.com/influxdata/influxdb/pull/26023)) * **Memory settings updates**: * Force snapshot memory threshold now defaults to `50%` of available memory * DataFusion execution memory pool now defaults to `20%` of available memory #### General Updates * Performance and reliability improvements. ### Enterprise #### Token Support * Authorization is now turned on by default. * Token support for database level permissions are now available. * Token support for system level queries are now available. #### General Updates * You can now use Commercial, Trial, and At-Home licenses. ## v3.0.0-0.beta.3 **Core**: revision f881c5844bec93a85242f26357a1ef3ebf419dd3 **Enterprise**: revision 6bef9e700a59c0973b0cefdc6baf11583933e262 ### Core #### General Improvements * InfluxDB 3 now supports graceful shutdowns when sending the interrupt signal to the service. #### Bug fixes * Empty batches in JSON format results are now handled properly * The Processing Engine now properly extracts data from DictionaryArrays ### Enterprise ##### Multi-node improvements * Query nodes now automatically detect new ingest nodes #### Bug fixes * Several fixes for compaction planning and processing * The Processing Engine now properly extracts data from DictionaryArrays ## v3.0.0-0.beta.2 **Core**: revision 033e1176d8c322b763b4aefb24686121b1b24f7c **Enterprise**: revision e530fcd498c593cffec2b56d4f5194afc717d898 This update brings several backend performance improvements to both Core and Enterprise in preparation for additional new features over the next several weeks. ## v3.0.0-0.beta.1 ### Core #### Features ##### Query and storage enhancements * New ability to stream response data for CSV and JSON queries, similar to how JSONL streaming works * Parquet files are now cached on the query path, improving performance * Query buffer is incrementally cleared when snapshotting, lowering memory spikes ##### Processing engine improvements * New Trigger Types: * *Scheduled*: Run Python plugins on custom, time-defined basis * *Request*: Call Python plugins via HTTP requests * New in-memory cache for storing data temporarily; cached data can be stored for a single trigger or across all triggers * Integration with virtual environments and install packages: * Specify Python virtual environment via CLI or `VIRTUAL_ENV` variable * Install packages or a `requirements.txt` * Python plugins are now implemented through triggers only. Simply create a trigger that references your Python plugin code file directly * Snapshots are now persisted in parallel, improving performance by running jobs simultaneously, rather than sequentially * Write to logs from within the Processing Engine ##### Database and CLI improvements * You can now specify the precision on your timestamps for writes using the `--precision` flag. Includes nano/micro/milli/seconds (ns/us/ms/s) * Added a new `show` system subcommand to display system tables with different options via SQL (default limit: 100) * Clearer table creation error messages ##### Bug fixes * If a database was created and the service was killed before any data was written, the database would not be retained * A last cache with specific “value” columns could not be queried * Running CTRL-C no longer stopped an InfluxDB process, due to a Python trigger * A previous build had broken JSON queries for RecordBatches * There was an issue with the distinct cache that caused panics #### Parameter changes For Core and Enterprise, there are parameter changes for simplicity: | Old Parameter |New Parameter| |-------------------------------|-------------| |`--writer-id`
`--host-id`| `--node-id` | ### Enterprise features #### Cluster management * Nodes are now associated with *clusters*, simplifying compaction, read replication, and processing * Node specs are now available for simpler management of cache creations #### Mode types * Set `ingest`, `query`, `compact`, and `process` individually per node ### Enterprise parameter changes For Enterprise, additional parameters for the `serve` command have been consolidated for simplicity: | Old Parameter | New Parameter | |------------------------------------------------------|---------------------------------------| |`--read-from-node-ids`
`--compact-from-node-ids`| `--cluster-id` | | `--run-compactions`
`--mode=compactor` |`--mode=compact`
`--mode=compact`| In addition to the above changes, `--cluster-id` is now a required parameter for all new instances. #### Related * [Get started with InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/) | Old | New | | --- | --- | | Old | New | | --disable-parquet-mem-cache , --disable-data-file-cache | --disable-file-cache | | --wal-max-write-buffer-size | --wal-max-buffered-writes | | --wal-snapshot-size | --wal-files-per-snapshot | | INFLUXDB3_DB_DIR | INFLUXDB3_DATA_DIR | | INFLUXDB3_NODE_IDENTIFIER_PREFIX | INFLUXDB3_NODE_ID | | INFLUXDB3_NODE_IDENTIFIER_FROM_ENV | INFLUXDB3_NODE_ID_FROM_ENV | | INFLUXDB3_NUM_WAL_FILES_TO_KEEP | INFLUXDB3_SNAPSHOTTED_WAL_FILES_TO_KEEP | | INFLUXDB3_START_WITHOUT_AUTH | INFLUXDB3_WITHOUT_AUTH | | INFLUXDB3_TCP_LISTINER_FILE_PATH (misspelling) | INFLUXDB3_TCP_LISTENER_FILE_PATH | | INFLUXDB3_TELEMETRY_DISABLE_UPLOAD | INFLUXDB3_DISABLE_TELEMETRY_UPLOAD | | Old name | New name | | --- | --- | | Old name | New name | | --pt-max-columns | --max-total-columns | | --pt-gen0-max-bytes-per-file | --gen0-max-file-size | | --pt-wal-replica-queue-size | --wal-replica-queue-length | | --pt-wal-max-buffer-size | --wal-buffer-size | | Old Parameter | New Parameter | | --- | --- | | Old Parameter | New Parameter | | --writer-id --host-id | --node-id | | Old Parameter | New Parameter | | --- | --- | | Old Parameter | New Parameter | | --read-from-node-ids --compact-from-node-ids | --cluster-id | | --run-compactions --mode=compactor | --mode=compact --mode=compact | --- # Visualize data Source: https://docs.influxdata.com/influxdb3/enterprise/visualize-data/ Use visualization tools like Grafana, Superset, and others to visualize time series data queried from InfluxDB 3 Enterprise. ### [InfluxDB 3 Explorer](/influxdb3/enterprise/visualize-data/explorer/) Serve the InfluxDB 3 Explorer web UI directly from your InfluxDB 3 Enterprise server. Starting with v3.11, Explorer ships inside the Enterprise binary as a WebAssembly (WASM) guest, so you don’t run a separate container. ### [Chronograf](/influxdb3/enterprise/visualize-data/chronograf/) Chronograf is a data visualization and dashboarding tool designed to visualize data in InfluxDB 1.x. Learn how to use Chronograf with InfluxDB 3 Enterprise. ### [Grafana](/influxdb3/enterprise/visualize-data/grafana/) Install and run [Grafana](https://grafana.com/) to query and visualize data from InfluxDB 3 Enterprise. ### [Power BI](/influxdb3/enterprise/visualize-data/powerbi/) Use Microsoft Power BI Desktop with the InfluxDB 3 custom connector to query and visualize data from InfluxDB 3 Enterprise. #### Related * [Query data in InfluxDB 3 Enterprise](/influxdb3/enterprise/query-data/) --- # Use Chronograf to visualize data Source: https://docs.influxdata.com/influxdb3/enterprise/visualize-data/chronograf/ * InfluxQL only [Chronograf](/chronograf/v1/) is a data visualization and dashboarding tool designed to visualize data in InfluxDB 1.x using the **InfluxQL** query language. This page walks through how to use Chronograf with **InfluxDB 3 Enterprise**. ## Prerequisites * [Download and install Chronograf](/chronograf/v1/introduction/installation/#download-and-install) * A running InfluxDB 3 Enterprise cluster with: * A [database](/influxdb3/enterprise/admin/databases/) to query * A [database token](/influxdb3/enterprise/admin/tokens/)with read permissions ## Enable InfluxDB 3 support To connect Chronograf to InfluxDB 3 Enterprise, start Chronograf with InfluxDB 3 support enabled using one of the following methods: [CLI flag](#cli-flag)[Environment variable](#environment-variable) ```sh chronograf --influxdb-v3-support-enabled ``` ```sh export INFLUXDB_V3_SUPPORT_ENABLED=true chronograf ``` ## Create an InfluxDB connection 1. Open Chronograf and click **Configuration** (wrench icon) in the navigation menu. 2. Click **Add Connection**. ![Chronograf connections landing page](/img/chronograf/1-6-connection-landing-page.png) 3. In the **Server Type** dropdown, select **InfluxDB 3 Enterprise**. Chronograf Server Type dropdown 4. Enter your InfluxDB 3 Enterprise connection credentials: * **Connection URL:** URL of your InfluxDB 3 Enterprise cluster ``` http://localhost:8181 ``` * **Connection Name:** A name that uniquely identifies this connection configuration * **Database Token:** InfluxDB [database token](/influxdb3/enterprise/admin/tokens/) with read permissions on the database you want to query * **Telegraf Database Name:** InfluxDB [database](/influxdb3/enterprise/admin/databases/)Chronograf uses to populate parts of the application, including the Host List page (default is `telegraf`) Chronograf InfluxDB 3 Enterprise connection configuration 5. Click **Add Connection**. 6. Select the dashboards you would like to create, and then click **Next**. 7. To configure a Kapacitor connection, provide the necessary credentials, and then click **Continue**. Otherwise, click **Skip**. 8. Click **Finish**. ### Configure connection via CLI You can also configure the connection when starting Chronograf: Replace the following: * `DATABASE_NAME`: [Database](/influxdb3/enterprise/admin/databases/) name * `DATABASE_TOKEN`: [database token](/influxdb3/enterprise/admin/tokens/)with read permissions ```sh chronograf --influxdb-v3-support-enabled \ --influxdb-type=influx-v3-enterprise \ --influxdb-url=http://localhost:8181 \ --influxdb-token=DATABASE_TOKEN \ --influxdb-default-db=DATABASE_NAME ``` For a complete list of configuration options, see [InfluxDB 3 connection options](/chronograf/v1/administration/config-options/#influxdb-3-connection-options). ## Query data in the Data Explorer 1. In Chronograf, click **Explore** in the left navigation bar. 2. Build and submit InfluxQL queries. > [!Note] > #### Schema information in the Data Explorer > > InfluxDB 3 Enterprise supports InfluxQL metaqueries, so schema information > is available in the Data Explorer to help build queries. > You can also use [fully qualified measurements](/influxdb3/enterprise/reference/influxql/select/#fully-qualified-measurement)in the `FROM` clause. For example: > > ``` > -- Fully qualified measurement > SELECT * FROM "db-name"."rp-name"."measurement-name" > > -- Fully qualified measurement shorthand (use the default retention policy) > SELECT * FROM "db-name".."measurement-name" > ``` > > For more information about available InfluxQL functionality, see[InfluxQL feature support](/influxdb3/enterprise/reference/influxql/feature-support/). ## Important notes * [Database view is disabled](#database-view-is-disabled) * [No administrative functionality](#no-administrative-functionality) * [Annotations and variables](#annotations-and-variables) ### Database view is disabled When connected to InfluxDB 3 Enterprise, the database view in Chronograf is disabled. ### No administrative functionality Chronograf cannot be used for administrative tasks in InfluxDB 3 Enterprise. For example, you **cannot** do the following: * Define databases * Modify retention policies * Add users * Kill queries When connected to an InfluxDB 3 Enterprise database, functionality in the **InfluxDB Admin** section of Chronograf is disabled. To complete administrative tasks, use the[`influxdb3` CLI](/influxdb3/enterprise/reference/cli/influxdb3/). ### Annotations and variables Annotations and dashboard variables work with InfluxDB 3 Enterprise when a `chronograf` database exists and is accessible with the same database token. When setting up variables with dynamic tag values, the backend query limits the scope of the record search with a time condition. By default, this is `time > now() - 7d`. Tags from records older than this limit are ignored. To change this setting, use the `--influxdb-v3-time-condition` flag or `INFLUXDB_V3_TIME_CONDITION` environment variable. #### Related * [Chronograf documentation](/chronograf/v1/) * [Query data with InfluxQL](/influxdb3/enterprise/query-data/influxql/) --- # Use the integrated InfluxDB 3 Explorer UI Source: https://docs.influxdata.com/influxdb3/enterprise/visualize-data/explorer/ * InfluxDB 3 Enterprise v3.11+ Starting with InfluxDB 3 Enterprise v3.11, the[InfluxDB 3 Explorer](/influxdb3/explorer/) web UI ships inside the Enterprise binary as a [WebAssembly](https://webassembly.org/) (WASM) guest that the server hosts in-process behind a[WASI](https://wasi.dev/) sandbox. You don’t need to run the Explorer Docker container alongside InfluxDB. Explorer isn’t enabled by default. To enable it and configure sign-in, sessions, and SSO, see[Use the integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/). When it’s enabled, the server serves Explorer at the root path of its regular HTTP address and port–for example, [http://localhost:8181/](http://localhost:8181/). * [Check your version](#check-your-version) * [Connect Explorer to your server](#connect-explorer-to-your-server) * [Enable AI chat](#enable-ai-chat) * [Choose between integrated and containerized Explorer](#choose-between-integrated-and-containerized-explorer) ## Check your version Explorer availability depends on the InfluxDB 3 server version and edition, not on the Explorer version alone. Use either of the following checks. To check the version of a local binary: ```bash influxdb3 --version ``` To check the version and edition of a running server, send a `GET` request to the `/ping` endpoint: ```sh curl --get "http://localhost:8181/ping" \ --header "Authorization: Bearer AUTH_TOKEN" ``` The response headers include `x-influxdb-version` and `x-influxdb-build`.`x-influxdb-build` reports `Core` or `Enterprise`, so one request answers both the version question and the edition question. Use `GET`; a `HEAD` request returns `404`. ## Connect Explorer to your server After the server starts, configure a connection to your server in Explorer the same way you configure one in the standalone Docker Explorer–for example,`http://localhost:8181`. For the connection fields and the steps to create a connection, see[Get started with InfluxDB 3 Explorer](/influxdb3/explorer/get-started/). Choose the token for the connection based on what you need Explorer to do: * A [resource token](/influxdb3/enterprise/admin/tokens/resource/) is enough to query and write data within the permissions you grant it. * To manage databases, tokens, and other resources from Explorer, use an[admin token](/influxdb3/enterprise/admin/tokens/admin/). ## Enable AI chat Explorer includes an AI chat feature that supports OpenAI, Anthropic, and Gemini. Each user enters their own AI provider API key in Explorer’s settings. You don’t configure an API key on the server, and no server option turns the feature on. [`--webui-openai-base-url`](/influxdb3/enterprise/reference/config-options/#webui-openai-base-url)changes only where Explorer sends OpenAI requests. Set it to route OpenAI traffic to an OpenAI-compatible endpoint, such as a self-hosted model or a gateway: ```bash influxdb3 serve \ --cluster-id cluster0 \ --node-id node0 \ --mode all,webui \ --webui-session-secret "$WEBUI_SESSION_SECRET" \ --webui-openai-base-url "https://your-openai-compatible-endpoint" ``` The option doesn’t affect Anthropic or Gemini requests. Chat prompts, and any query results included with them, go to the AI provider the user selects. Choose providers and endpoints that your data handling policies allow. ## Choose between integrated and containerized Explorer | | Integrated (WASM) | Docker container | |--------------------------|-------------------------------------|------------------------------------| | Requires | InfluxDB 3 Enterprise v3.11+ | Docker | | Runs | In the InfluxDB server process | As a separate container | | Enabled by | `--mode all,webui` |`docker run influxdata/influxdb3-ui`| | Application data |SQLite synchronized to object storage| SQLite in a mounted volume | |Works with InfluxDB 3 Core| No | Yes | The container isn’t replaced by the integrated UI. It’s required for Core and for Enterprise earlier than v3.11, and it still works with v3.11 and later. Use the container when you run Core, when you run an Enterprise release earlier than v3.11, or when you want Explorer to run separately from the database server–for example, on an operator workstation. See [Install and run InfluxDB 3 Explorer with Docker](/influxdb3/explorer/install/docker/). #### Related * [Use the integrated Explorer UI](/influxdb3/enterprise/admin/explorer-ui/) * [InfluxDB 3 Explorer documentation](/influxdb3/explorer/) * [Install InfluxDB 3 Explorer](/influxdb3/explorer/install/) | | Integrated (WASM) | Docker container | | --- | --- | --- | | | Integrated (WASM) | Docker container | | Requires | InfluxDB 3 Enterprise v3.11+ | Docker | | Runs | In the InfluxDB server process | As a separate container | | Enabled by | --mode all,webui | docker run influxdata/influxdb3-ui | | Application data | SQLite synchronized to object storage | SQLite in a mounted volume | | Works with InfluxDB 3 Core | No | Yes | --- # Use Grafana to visualize data Source: https://docs.influxdata.com/influxdb3/enterprise/visualize-data/grafana/ Use [Grafana](https://grafana.com/) or [Grafana Cloud](https://grafana.com/products/cloud/)to query and visualize data from InfluxDB 3 Enterprise. > [!Note] > #### Identify your InfluxDB version > > If you are unsure which InfluxDB product you are using, use our interactive version detector to help identify it: > > [Detect my InfluxDB version](#) > *(Interactive feature in HTML: Opens version detector modal)* > > > [Grafana] enables you to query, visualize, alert on, and explore your metrics, > logs, and traces wherever they are stored. > [Grafana] provides you with tools to turn your time-series database (TSDB) > data into insightful graphs and visualizations. > > > > – [Grafana documentation](https://grafana.com/docs/grafana/latest/introduction/) > > * [Install Grafana or log in to Grafana Cloud](#install-grafana-or-log-in-to-grafana-cloud) * [InfluxDB data source](#influxdb-data-source) * [Before you begin](#before-you-begin) * [Create an InfluxDB data source](#create-an-influxdb-data-source) * [Query and visualize data](#query-and-visualize-data) ## Install Grafana or log in to Grafana Cloud 1. [Sign up for Grafana Cloud](https://grafana.com/products/cloud/) or follow the[Grafana installation instructions](https://grafana.com/docs/grafana/latest/setup-grafana/installation/)to install Grafana for your operating system. 2. If running Grafana locally, enable the `newInfluxDSConfigPageDesign` feature flag to use the latest InfluxDB data source plugin. [](#option-1-configuration-file-recommended) Option 1: Configuration file (recommended) Add the following to your `grafana.ini` configuration file: ``` [feature_toggles] enable = newInfluxDSConfigPageDesign ``` Configuration file locations: * **Linux**: `/etc/grafana/grafana.ini` * **macOS (Homebrew)**: `/opt/homebrew/etc/grafana/grafana.ini` * **Windows**: `\conf\grafana.ini` [](#option-2-command-line) Option 2: Command line Enable the feature flag when starting Grafana: [Linux](#linux)[macOS (Homebrew)](#macos-homebrew)[Windows](#windows) ``` grafana-server --config /etc/grafana/grafana.ini \ cfg:default.feature_toggles.enable=newInfluxDSConfigPageDesign ``` ``` /opt/homebrew/opt/grafana/bin/grafana server \ --config /opt/homebrew/etc/grafana/grafana.ini \ --homepath /opt/homebrew/opt/grafana/share/grafana \ --packaging=brew \ cfg:default.paths.logs=/opt/homebrew/var/log/grafana \ cfg:default.paths.data=/opt/homebrew/var/lib/grafana \ cfg:default.paths.plugins=/opt/homebrew/var/lib/grafana/plugins \ cfg:default.feature_toggles.enable=newInfluxDSConfigPageDesign ``` ``` grafana-server.exe --config \conf\grafana.ini ` cfg:default.feature_toggles.enable=newInfluxDSConfigPageDesign ``` For more information, see [Configure feature toggles](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/feature-toggles/) in the Grafana documentation. 3. Visit your **Grafana Cloud user interface** (UI) or, if running Grafana locally,[start Grafana](https://grafana.com/docs/grafana/latest/installation/) and visit[http://localhost:3000](http://localhost:3000) in your browser. > [!Note] > #### Using Grafana Cloud with a local InfluxDB instance > > If you need to keep your database local, consider running Grafana locally instead of using Grafana Cloud, > as this avoids the need to expose your database to the internet. > > To use InfluxDB running on your private network with Grafana Cloud, you must configure a[private data source for Grafana Cloud](https://grafana.com/docs/grafana-cloud/data-sources/private-data-sources/). ## InfluxDB data source The InfluxDB data source plugin is included in the Grafana core distribution. Use the plugin to query and visualize data from InfluxDB 3 Enterprise with both SQL and InfluxQL. > [!Note] > #### Grafana 12.2+ > > The instructions below are for **Grafana 12.2+** with the `newInfluxDSConfigPageDesign`feature flag enabled. This introduces the newest version of the InfluxDB core plugin. > The updated plugin includes **SQL support** for InfluxDB 3-based products such > as InfluxDB 3 Enterprise, and the interface dynamically adapts based on your > product and query language selection in [URL and authentication](#configure-url-and-authentication). ## Before you begin **Prerequisites:** * Grafana 12.2 or later * Administrator role in Grafana * Your [admin token](/influxdb3/enterprise/admin/tokens/database)with read access to the database ### Quick reference: InfluxDB 3 Enterprise configuration | Configuration |Value| |---------------------|-----| |**Product selection**| | **InfluxDB Enterprise 3.x**| | **URL** | Server URL –for example, `https://localhost:8181`| | **Query languages** | SQL (requires HTTP/2), InfluxQL | | **Authentication** | Admin or database token (if authentication is enabled) | | **Database/Bucket** | Database name | ## Create an InfluxDB data source 1. In your Grafana interface, click **Connections** in the left sidebar. 2. Click **Data sources**. 3. Click **Add new data source**. 4. Search for and select **InfluxDB**. The InfluxDB data source configuration page displays. 5. In the **Settings** tab, enter a **Name** for your data source. ### Configure URL and authentication In the **URL and authentication** section, configure the following: * **URL**: Your InfluxDB 3 Enterprise server URL –for example, `https://localhost:8181` * **Product**: From the dropdown, select **InfluxDB Enterprise 3.x** * **Query Language**: Select **SQL** or **InfluxQL** * *(Optional)* **Advanced HTTP Settings**, **Auth**, and **TLS/SSL Settings** as needed for your environment ### Configure database settings The fields in this section change based on your query language selection in [URL and authentication](#configure-url-and-authentication). [SQL](#sql)[InfluxQL](#influxql) #### SQL configuration When you select **SQL** as the query language, configure the following fields: * **Database**: Your [database](/influxdb3/enterprise/admin/databases/) name. * **Token**: Your [admin token](/influxdb3/enterprise/admin/tokens/database)with read access to the database SQL configuration for {{% product-name %}} #### Use SQL without TLS/SSL If using SQL to query InfluxDB 3 Enterprise without TLS/SSL, enable the**Insecure Connection** option in your InfluxDB datasource configuration: 1. Under **Database settings**, enable **Advanced Database Settings**. 2. Enable **Insecure Connection**. Query InfluxDB using SQL without TLS/SSL > [!Important] > #### Grafana queries through a proxy require HTTP/2 > > For SQL queries, Grafana uses the Flight SQL protocol (gRPC) to query InfluxDB 3 Enterprise, which requires **HTTP/2**. > If you query InfluxDB 3 Enterprise through a proxy (such as HAProxy, nginx, or a load balancer), > verify that your proxy is configured to support HTTP/2. > Without HTTP/2 support, SQL queries through Grafana will fail to connect. > > InfluxQL queries use HTTP/1.1 and are not affected by this requirement. Click **Save & test**. Grafana attempts to connect to InfluxDB 3 Enterprise and returns the result of the test. #### InfluxQL configuration When you select **InfluxQL** as the query language, configure the following fields: * **Database**: Your [database](/influxdb3/enterprise/admin/databases/) name. * **User**: A username (can be any non-empty value). * **Password**: Your [database token](/influxdb3/enterprise/admin/tokens/#database-tokens) with read access to the database. * **HTTP Method**: Select **POST** (recommended) or **GET** InfluxQL configuration for {{% product-name %}} with DBRP warning Click **Save & test**. Grafana attempts to connect to InfluxDB 3 Enterprise and returns the result of the test. ## Query and visualize data With your InfluxDB connection configured, use Grafana to query and visualize time series data. ### Query InfluxDB with Grafana After you [configure and save an InfluxDB datasource](#create-an-influxdb-data-source), use Grafana to build, run, and inspect queries against InfluxDB 3 Enterprise . [SQL](#sql-2)[InfluxQL](#influxql-2) > [!Note] > When working with the InfluxDB 3 Enterprise SQL implementation > a **bucket** is equivalent to a **database**, > a **measurement** is equivalent to a **table**, and **time**, **fields**, and**tags** are structured as **columns**. > > > To learn more, see [Query Data](/influxdb3/enterprise/query-data/sql/). > > 1. Click **Explore**. 2. In the dropdown, select the saved InfluxDB data source to query. 3. Use the SQL query form to build your query: * **Table**: Select the table (measurement) to query. * **Column**: Select one or more fields and tags to return as columns in query results. With SQL, select the `time` column to include timestamps with the data. Grafana relies on the `time` column to correctly graph time series data. * ***Optional:*** Toggle **filter** to generate **WHERE** clause statements. * **WHERE**: Configure condition expressions to include in the `WHERE` clause. * ***Optional:*** Toggle **group** to generate **GROUP BY** clause statements. * **GROUP BY**: Select columns to group by. If you include an aggregation function in the **SELECT** list, you must group by one or more of the queried columns. SQL returns the aggregation for each group. * Recommended: Toggle **order** to generate **ORDER BY** clause statements. * **ORDER BY**: Select columns to sort by. You can sort by time and multiple fields or tags. To sort in descending order, select **DESC**. 4. Recommended: Change format to **Time series**. * Use the **Format** dropdown to change the format of the query results. For example, to visualize the query results as a time series, select **Time series**. 5. Click **Run query** to execute the query. 1. Click **Explore**. 2. In the dropdown, select the **InfluxDB** data source that you want to query. 3. Use the InfluxQL query form to build your query: * **FROM**: Select the measurement that you want to query. * **WHERE**: To filter the query results, enter a conditional expression. * **SELECT**: Select fields to query and an aggregate function to apply to each. The aggregate function is applied to each time interval defined in the`GROUP BY` clause. * **GROUP BY**: By default, Grafana groups data by time to downsample results and improve query performance. You can also add other tags to group by. 4. Click **Run query** to execute the query. With your InfluxDB connection configured, use Grafana to query and visualize time series data. ### Build visualizations with Grafana For a comprehensive walk-through of creating visualizations with Grafana, see the [Grafana documentation](https://grafana.com/docs/grafana/latest/). ### Query inspection in Grafana To learn more about query management and inspection in Grafana, see the[Grafana Explore documentation](https://grafana.com/docs/grafana/latest/explore/). [query](/influxdb3/enterprise/tags/query/)[visualization](/influxdb3/enterprise/tags/visualization/) | Configuration | Value | | --- | --- | | Configuration | Value | | Product selection | | --- # Use Power BI to visualize data Source: https://docs.influxdata.com/influxdb3/enterprise/visualize-data/powerbi/ * SQL Use [Microsoft Power BI Desktop](https://powerbi.microsoft.com/) with the This guide includes Windows (64‑bit) installation steps for the Arrow Flight SQL ODBC driver and the InfluxDB 3 Power BI connector. > > > Microsoft Power BI is a collection of software services, apps, and connectors > that work together to turn your unrelated sources of data into coherent, > visually immersive, and interactive insights. > > > > – [Microsoft Power BI documentation](https://learn.microsoft.com/en-us/power-bi/fundamentals/power-bi-overview) > > > [!Important] > These instructions are for Power BI Desktop only; it uses a custom connector. 1. [Prerequisites](#prerequisites) 2. [Install the Arrow Flight SQL ODBC driver](#install-the-arrow-flight-sql-odbc-driver) 3. [Install the Power BI connector](#install-the-power-bi-connector) 4. [Enable the connector in Power BI](#enable-the-connector-in-power-bi) 5. [Connect Power BI to InfluxDB](#connect-power-bi-to-influxdb) 6. [Query and visualize data](#query-and-visualize-data) ## Prerequisites * **Windows operating system**: The custom connector requires Windows (macOS users can use virtualization software like [Parallels](https://www.parallels.com/)) * **Power BI Desktop**: [Download and install Power BI Desktop](https://powerbi.microsoft.com/desktop/)(A free trial is available) * **InfluxDB 3 Enterprise**: A running instance with data to query * **Database token**: Your [admin token](/influxdb3/enterprise/admin/tokens/database) with query permissions for the target database ## Install the Arrow Flight SQL ODBC driver The InfluxDB 3 custom connector for Power BI requires the Arrow Flight SQL ODBC driver. Install the driver before installing the Power BI connector. [Download the Arrow Flight SQL ODBC driver](https://docs.influxdata.com/downloads/apache-arrow-flight-sql-odbc-1.0.0-win64.msi) 1. Run the downloaded `.msi` installer. 2. Follow the installation wizard using default settings. 3. Complete the installation. #### Verify driver installation 1. Open the Windows application [ODBC Data Source Administrator (64-bit)](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/open-the-odbc-data-source-administrator). 2. Open the [ODBC Data Source Administrator (64-bit) Windows component](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/open-the-odbc-data-source-administrator) 3. Navigate to the **Drivers** tab 4. Verify **Arrow Flight SQL ODBC Driver** appears in the list. ## Install the Power BI connector After installing the ODBC driver, download and install the InfluxDB 3 custom connector for Power BI Desktop. [Download the InfluxDB 3 Power BI connector](https://docs.influxdata.com/downloads/InfluxDB.pqx) ### Move the connector to the custom connectors directory 1. Create the custom connectors folder if it doesn’t exist: ``` mkdir "$env:USERPROFILE\Documents\Power BI Desktop\Custom Connectors" ``` 2. Move the connector file to the custom connectors folder: ``` Move-Item "C:\Users\YOUR_USER\Downloads\InfluxDB.pqx" ` "$env:USERPROFILE\Documents\Power BI Desktop\Custom Connectors\" ``` Replace the following: * `YOUR_USER`: Your Windows username ## Enable the connector in Power BI To use custom connectors, you must adjust Power BI Desktop’s security settings: 1. Open Power BI Desktop 2. Select **File** \> **Options and settings** \> **Options** 3. Navigate to **Security** under **GLOBAL** 4. Under **Data Extensions**, select **(Not Recommended) Allow any extension to load without validation or warning** 5. Click **OK** 6. **Restart Power BI Desktop** for the changes to take effect > [!Warning] > #### Security considerations > > Enabling uncertified extensions allows any custom connector to load. > Only enable this setting if you trust the connectors you’re installing. ## Connect Power BI to InfluxDB After installing the connector and restarting Power BI Desktop: 1. Open **Power BI Desktop** 2. Click **Get Data** \> **More** 3. Search for **InfluxDB 3** and select it 4. Click **Connect** 5. In the **InfluxDB 3** connection dialog, configure the following: * **Server**: Your InfluxDB 3 Enterprise URL without the port, (for example, `http://localhost`) * **Database**: Your database name * **Port**: Your server port (for example,`8181` (default) ) * **Native Query** (optional): Enter a SQL query to limit the data loaded 1. Select **DirectQuery** as the **Data Connectivity mode** 2. Click **OK** 3. When prompted for credentials: * Select **Basic** authentication * **Username**: Leave blank or enter any value * **Password**: Enter your [admin token](/influxdb3/enterprise/admin/tokens/database) with query permissions for the target database 1. Click **Connect** 2. Preview your data and click **Load** > [!Important] > #### Limit query size for optimal performance > > InfluxDB 3 Enterprise can handle high throughput and dimensional data. > To ensure Power BI can successfully process data, limit query size by: > > * Using a `LIMIT` clause > * Specifying time ranges with `WHERE time >= ...` > * Filtering by specific columns or tags ## Query and visualize data ### Use Native Queries When connecting to InfluxDB 3, you can use the **Native Query** option to execute custom SQL queries: 1. In the connection dialog, enable **Native Query** 2. Enter your query in the provided field: ``` SELECT time, temp, room FROM home WHERE time >= now() - INTERVAL '7 days' ORDER BY time DESC LIMIT 1000 ``` 3. Select **DirectQuery** as the connectivity mode 4. Click **OK** to load the data ### Create visualizations After loading data, Power BI displays your dataset in the **Fields** pane. #### View data in a table 1. In the **Visualizations** pane, select the **Table** visualization 2. In the **Fields** pane, select the columns to display: * **time**: Timestamp column * **room**: Tag column * **temp**: Field column 3. By default, Power BI summarizes numeric fields. To display raw values: * Select the field in the **Fields** or **Visualizations** pane * Go to the **Modeling** tab * Change **Default Summarization** to **Don’t Summarize** #### Create time series visualizations 1. In the **Visualizations** pane, select a visualization type (for example, **Line chart**) 2. Drag fields to the appropriate areas: * **X-axis**: `time` field * **Y-axis**: Measurement fields (for example, `temp`) * **Legend**: Tag fields (for example, `room`) 3. Use the **Filters** pane to apply additional filters 4. Configure visualization properties in the **Format** pane 5. Save your report ### Time series best practices * Always include time range filters in your queries to limit data volume * Use the `time` column for time-based visualizations * Apply Power BI’s date hierarchy features with the `time` column * Select only the columns you need to improve query performance * Use the `LIMIT` clause to restrict the number of rows returned ## Troubleshooting ### Driver not found If Power BI or other applications can’t find the Arrow Flight SQL ODBC driver: 1. Open the [ODBC Data Source Administrator (64-bit) Windows component](https://learn.microsoft.com/en-us/sql/database-engine/configure-windows/open-the-odbc-data-source-administrator) 2. Navigate to the **Drivers** tab 3. Verify **Arrow Flight SQL ODBC Driver** appears in the list 4. If not listed, reinstall the driver ### Connector not found If Power BI Desktop doesn’t show the InfluxDB 3 connector: * Verify the `.pqx` file is in the correct location:`Documents\Power BI Desktop\Custom Connectors\` * Ensure you enabled custom connectors in Power BI security settings * Restart Power BI Desktop after copying the connector file ### Connection errors If you encounter connection errors: * Verify your InfluxDB 3 Enterprise instance is running and accessible * Check that the host URL and port are correct: * Local instances typically use `http://localhost:8181` * * Ensure `UseEncryption` is configured correctly for your connection type * Verify network connectivity and firewall rules allow connections on the specified port * Ensure your token has query permissions for the specified database ### Authentication errors If authentication fails: * Verify your database token is valid and not expired * Ensure the token is specified correctly (in the **Password** field for Power BI) * Verify the token has query permissions for the target database * Check that the token was copied correctly without extra spaces or characters * Leave the **Username** field blank or enter any value ### Query errors If queries fail or return errors: * Verify SQL syntax is correct for InfluxDB SQL * Check that referenced tables (measurements) exist in the database * Ensure column names match your schema * Review the [SQL reference](/influxdb3/enterprise/reference/sql/) for supported features * For large result sets, consider adding `LIMIT` clauses ### Query performance For better query performance: * Always use `WHERE` clauses to filter data before loading * Include time range filters (for example, `WHERE time >= now() - INTERVAL '7 days'`) to limit the data scanned * Select only the columns you need * Use the `LIMIT` clause to restrict result size * Consider using **DirectQuery** mode instead of **Import** for large datasets * Monitor query execution for optimization opportunities #### Related * [Power BI documentation](https://learn.microsoft.com/en-us/power-bi/desktop/) * [Query data with SQL](/influxdb3/enterprise/query-data/sql/) [visualization](/influxdb3/enterprise/tags/visualization/)[powerbi](/influxdb3/enterprise/tags/powerbi/)[sql](/influxdb3/enterprise/tags/sql/) --- # Write data to InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/ Use tools like the `influxdb3`CLI, Telegraf, and InfluxDB client libraries to write time series data to InfluxDB 3 Enterprise.[line protocol](#line-protocol)is the text-based format used to write data to InfluxDB. > [!Tip] > Tools are available to convert other formats (for example—[CSV](/influxdb3/enterprise/write-data/use-telegraf/csv/)) to line protocol. * [Choose the write endpoint for your workload](#choose-the-write-endpoint-for-your-workload) * [Timestamp precision across write APIs](#timestamp-precision-across-write-apis) * [Line protocol](#line-protocol) * [Line protocol elements](#line-protocol-elements) * [Write data to InfluxDB](#write-data-to-influxdb) * [Use InfluxDB client libraries to write data](#use-influxdb-client-libraries-to-write-data) * [Use the InfluxDB HTTP API to write data](#use-the-influxdb-http-api-to-write-data) * [Use Telegraf to write data](#use-telegraf-to-write-data) * [Use the influxdb3 CLI to write data](#use-the-influxdb3-cli-to-write-data) * [Best practices for writing data](#best-practices-for-writing-data) * [Troubleshoot issues writing data](#troubleshoot-issues-writing-data) > [!Tip] > #### Choose the write endpoint for your workload > > When creating new write workloads, use the[InfluxDB HTTP API `/api/v3/write_lp` endpoint](/influxdb3/enterprise/write-data/http-api/v3-write-lp/)and [client libraries](/influxdb3/enterprise/write-data/client-libraries/). > > When bringing existing *v1* write workloads, use the InfluxDB 3 Enterprise > HTTP API [`/write` endpoint](/influxdb3/core/api/write-data/#operation/PostV1Write). > > When bringing existing *v2* write workloads, use the InfluxDB 3 Enterprise > HTTP API [`/api/v2/write` endpoint](/influxdb3/enterprise/api/write-data/). > > **For Telegraf**, use the [`outputs.influxdb_v3`](/telegraf/v1/output-plugins/influxdb_v3/) output plugin. > For existing v1 or v2 write workloads, use the v1.x [`outputs.influxdb`](/telegraf/v1/output-plugins/influxdb/) or v2.x [`outputs.influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/) output plugins. > See how to [use Telegraf to write data](/influxdb3/enterprise/write-data/use-telegraf/). ## Timestamp precision across write APIs InfluxDB 3 Enterprise provides multiple write endpoints for compatibility with different InfluxDB versions. The following table compares timestamp precision support across v1, v2, and v3 write APIs: | Precision |v1 (`/write`)|v2 (`/api/v2/write`)|v3 (`/api/v3/write_lp`)| |------------------|-------------|--------------------|-----------------------| |**Auto detection**| ❌ No | ❌ No | ✅ `auto` (default) | | **Seconds** | ✅ `s` | ✅ `s` | ✅ `second` | | **Milliseconds** | ✅ `ms` | ✅ `ms` | ✅ `millisecond` | | **Microseconds** |✅ `u` or `µ` | ✅ `us` | ✅ `microsecond` | | **Nanoseconds** | ✅ `ns` | ✅ `ns` | ✅ `nanosecond` | | **Minutes** | ✅ `m` | ❌ No | ❌ No | | **Hours** | ✅ `h` | ❌ No | ❌ No | | **Default** | Nanosecond | Nanosecond | **Auto** (guessed) | * All write endpoints accept timestamps in line protocol format. * InfluxDB 3 Enterprise multiplies timestamps by the appropriate precision value to convert them to nanoseconds for internal storage. * All timestamps are stored internally as nanoseconds regardless of the precision specified when writing. ## Line protocol All data written to InfluxDB is written using[line protocol](/influxdb3/enterprise/reference/line-protocol/), a text-based format that lets you provide the necessary information to write a data point to InfluxDB. ### Line protocol elements In InfluxDB, a point contains a table name, one or more fields, a timestamp, and optional tags that provide metadata about the observation. Each line of line protocol contains the following elements: \* Required * \* **table**: A string that identifies the table to store the data in. * **tag set**: Comma-delimited list of key value pairs, each representing a tag. Tag keys and values are unquoted strings. *Spaces, commas, and equal characters must be escaped.* * \* **field set**: Comma-delimited list of key value pairs, each representing a field. Field keys are unquoted strings. *Spaces and commas must be escaped.*Field values can be [strings](/influxdb3/enterprise/reference/line-protocol/#string)(quoted),[floats](/influxdb3/enterprise/reference/line-protocol/#float),[integers](/influxdb3/enterprise/reference/line-protocol/#integer),[unsigned integers](/influxdb3/enterprise/reference/line-protocol/#uinteger), or [booleans](/influxdb3/enterprise/reference/line-protocol/#boolean). * **timestamp**: [Unix timestamp](/influxdb3/enterprise/reference/line-protocol/#unix-timestamp)associated with the data. InfluxDB supports up to nanosecond precision.*If the precision of the timestamp is not in nanoseconds, you must specify the precision when writing the data to InfluxDB.* #### Line protocol element parsing * **table**: Everything before the *first unescaped comma before the first whitespace*. * **tag set**: Key-value pairs between the *first unescaped comma* and the *first unescaped whitespace*. * **field set**: Key-value pairs between the *first and second unescaped whitespaces*. * **timestamp**: Integer value after the *second unescaped whitespace*. * Lines are separated by the newline character (`\n`). Line protocol is whitespace sensitive. myTable,tag1=val1,tag2=val2 field1="v1",field2=1i 0000000000000000000 *For schema design recommendations, see[InfluxDB schema design](/influxdb3/enterprise/write-data/best-practices/schema-design/).* ## Write data to InfluxDB ### [Use InfluxDB client libraries to write data](/influxdb3/enterprise/write-data/client-libraries/) Use InfluxDB API clients to write points as line protocol data to InfluxDB 3 Enterprise. ### [Use the InfluxDB HTTP API to write data](/influxdb3/enterprise/write-data/http-api/) Use the `/api/v3/write_lp`, `/api/v2/write`, or `/write` HTTP API endpoints to write data to InfluxDB 3 Enterprise. ### [Use Telegraf to write data](/influxdb3/enterprise/write-data/use-telegraf/) Use Telegraf to collect and write data to InfluxDB 3 Enterprise. ### [Use the influxdb3 CLI to write data](/influxdb3/enterprise/write-data/influxdb3-cli/) Use the [`influxdb3` CLI](/influxdb3/enterprise/reference/cli/influxdb3/) to write line protocol data to InfluxDB 3 Enterprise. ### [Best practices for writing data](/influxdb3/enterprise/write-data/best-practices/) Learn about the recommendations and best practices for writing data to InfluxDB 3 Enterprise. ### [Troubleshoot issues writing data](/influxdb3/enterprise/write-data/troubleshoot/) Troubleshoot issues writing data. Find response codes for failed writes. Discover how writes fail, from exceeding rate or payload limits, to syntax errors and schema conflicts. [write](/influxdb3/enterprise/tags/write/)[line protocol](/influxdb3/enterprise/tags/line-protocol/) | Precision | v1 ( /write ) | v2 ( /api/v2/write ) | v3 ( /api/v3/write_lp ) | | --- | --- | --- | --- | | Precision | v1 ( /write ) | v2 ( /api/v2/write ) | v3 ( /api/v3/write_lp ) | | Auto detection | ❌ No | ❌ No | ✅ auto (default) | | Seconds | ✅ s | ✅ s | ✅ second | | Milliseconds | ✅ ms | ✅ ms | ✅ millisecond | | Microseconds | ✅ u or µ | ✅ us | ✅ microsecond | | Nanoseconds | ✅ ns | ✅ ns | ✅ nanosecond | | Minutes | ✅ m | ❌ No | ❌ No | | Hours | ✅ h | ❌ No | ❌ No | | Default | Nanosecond | Nanosecond | Auto (guessed) | --- # Best practices for writing data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/ The following articles walk through recommendations and best practices for writing data to InfluxDB 3 Enterprise. ### [Optimize writes to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/best-practices/optimize-writes/) Tips and examples to optimize performance and system overhead when writing data to InfluxDB 3 Enterprise. ### [InfluxDB schema design recommendations](/influxdb3/enterprise/write-data/best-practices/schema-design/) Design your schema for simpler and more performant queries. --- # Optimize writes to InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/optimize-writes/ Use these tips to optimize performance and system overhead when writing data to InfluxDB 3 Enterprise. * [Batch writes](#batch-writes) * [On first write, sort tags by query priority](#on-first-write-sort-tags-by-query-priority) * [Use the coarsest time precision possible](#use-the-coarsest-time-precision-possible) * [Use gzip compression](#use-gzip-compression) * [Enable gzip compression in Telegraf](#enable-gzip-compression-in-telegraf) * [Enable gzip compression in InfluxDB client libraries](#enable-gzip-compression-in-influxdb-client-libraries) * [Use gzip compression with the InfluxDB API](#use-gzip-compression-with-the-influxdb-api) * [Synchronize hosts with NTP](#synchronize-hosts-with-ntp) * [Write multiple data points in one request](#write-multiple-data-points-in-one-request) * [Pre-process data before writing](#pre-process-data-before-writing) * [Prerequisites](#prerequisites) * [Filter data from a batch](#filter-data-from-a-batch) * [Coerce data types to avoid rejected point errors](#coerce-data-types-to-avoid-rejected-point-errors) * [Merge lines to optimize memory and bandwidth](#merge-lines-to-optimize-memory-and-bandwidth) * [Avoid sending duplicate data](#avoid-sending-duplicate-data) * [Run custom preprocessing code](#run-custom-preprocessing-code) > [!Note] > The following tools write to InfluxDB and employ *most* write optimizations by > default: > > * [Telegraf](/influxdb3/enterprise/write-data/use-telegraf/) > * InfluxDB client libraries ## Batch writes Write data in batches to minimize network overhead when writing data to InfluxDB. > [!Note] > The optimal batch size is 10,000 lines of line protocol or 10 MBs, whichever > threshold is met first. ## On first write, sort tags by query priority The first write to a table in InfluxDB 3 Enterprise determines the physical column order in storage, and that order has a direct impact on query performance. Columns that appear earlier are typically faster to filter and access during query execution. Sort your tags by query priority when performing the initial write to a table. Place the most commonly queried tags first—those you frequently use in `WHERE`clauses or joins—followed by less frequently queried ones. For example, if most of your queries filter by `region` and then by `host`, structure your first write so that `region` comes before `host`. > [!Important] > Column order is determined on the first write and cannot be changed afterward. > Tags added after the first write are added last in the column sort order. > Plan your schema with your query workload in mind to ensure the best long-term > performance. ## Use the coarsest time precision possible InfluxDB 3 Enterprise supports up to nanosecond timestamp precision. However, if your data isn’t collected in nanoseconds, there is no need to write at that precision. For better performance, use the coarsest timestamp precision you can for your use case. > [!Tip] > By default, in CLI and HTTP API write requests, InfluxDB 3 Enterprise uses the timestamp magnitude to auto-detect the precision. To specify the precision of timestamps in your write request, pass the `precision` option. For more information, see the following: * [`/api/v3/write_lp` endpoint parameters](/influxdb3/enterprise/write-data/http-api/v3-write-lp/) * [`/api/v2/write` v2 API endpoint parameters](/influxdb3/enterprise/write-data/http-api/compatibility-apis/#v2-api-write-parameters) * [`/write` v1 API endpoint parameters](/influxdb3/enterprise/write-data/http-api/compatibility-apis/#v1-api-write-parameters) ## Use gzip compression Use gzip compression to speed up writes to InfluxDB 3 Enterprise. Benchmarks have shown up to a 5x speed improvement when data is compressed. [Telegraf](#telegraf)[Client libraries](#client-libraries)[InfluxDB API](#influxdb-api) ### Enable gzip compression in Telegraf The [`influxdb_v3` output plugin](/telegraf/v1/output-plugins/influxdb_v3/)compresses write request bodies with gzip by default. If you use the `influxdb_v2` output plugin, set the `content_encoding` option to `gzip` in your `telegraf.conf`: ```toml [[outputs.influxdb_v2]] urls = ["http://localhost:8181"] # ... content_encoding = "gzip" ``` ### Enable gzip compression in InfluxDB client libraries Each [InfluxDB client library](/influxdb3/enterprise/reference/client-libraries/) provides options for compressing write requests or enforces compression by default. The method for enabling compression is different for each library. For specific instructions, see the[InfluxDB client libraries documentation](/influxdb3/enterprise/reference/client-libraries/). ### Use gzip compression with the InfluxDB API When using the InfluxDB API `/api/v2/write` endpoint to write data, compress the data with `gzip` and set the `Content-Encoding` header to `gzip`–for example: ```bash echo "mem,host=host1 used_percent=23.43234543 1641024000 mem,host=host2 used_percent=26.81522361 1641027600 mem,host=host1 used_percent=22.52984738 1641031200 mem,host=host2 used_percent=27.18294630 1641034800" | gzip > system.gzip \ curl --request POST "http://localhost:8181/api/v2/write?org=ignored&bucket=DATABASE_NAME" \ --header "Authorization: Token AUTH_TOKEN" \ --header "Content-Type: text/plain; charset=utf-8" \ --header "Content-Encoding: gzip" \ --data-binary @system.gzip ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) *Store this in a secret store or environment variable to avoid exposing the raw token string.* ## Synchronize hosts with NTP Use the Network Time Protocol (NTP) to synchronize time between hosts. If a timestamp isn’t included in line protocol, InfluxDB uses its host’s local time (in UTC) to assign timestamps to each point. If a host’s clocks isn’t synchronized with NTP, timestamps may be inaccurate. ## Write multiple data points in one request To write multiple lines in one request, each line of line protocol must be delimited by a new line (`\n`). ## Pre-process data before writing Pre-processing data in your write workload can help you avoid[write failures](/influxdb3/enterprise/write-data/troubleshoot/#troubleshoot-failures)due to schema conflicts or resource use. For example, if you have many devices that write to the same table, and some devices use different data types for the same field, then you might want to generate an alert or convert field data to fit your schema before you send the data to InfluxDB. With [Telegraf](/telegraf/v1/), you can process data from other services and files and then write it to InfluxDB. In addition to processing data with Telegraf’s included plugins, you can use the[Execd processor plugin](/telegraf/v1/plugins/#processor-execd) to integrate your own code and external applications. The following examples show how to [configure the Telegraf agent](/telegraf/v1/configuration)and [plugins](/telegraf/v1/plugins/) to optimize writes. The examples use the [File input plugin](/telegraf/v1/plugins/#input-file) to read data from a file and use the [InfluxDB v3 output plugin](/telegraf/v1/plugins/#output-influxdb_v3)to write data to a database, but you can use any input and output plugin. ### Prerequisites [Install Telegraf](/telegraf/v1/install/) if you haven’t already. ### Filter data from a batch Use Telegraf and metric filtering to filter data before writing it to InfluxDB. Configure [metric filters](/telegraf/v1/configuration/#filters) to retain or remove data elements (before processor and aggregator plugins run). 1. Enter the following command to create a Telegraf configuration that parses system usage data, removes the specified fields and tags, and then writes the data to InfluxDB: ``` cat <> ./telegraf.conf [[inputs.cpu]] # Remove the specified fields from points. fieldpass = ["usage_system", "usage_idle"] # Remove the specified tags from points. tagexclude = ["host"] [[outputs.influxdb_v3]] urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" EOF ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) *Store this in a secret store or environment variable to avoid exposing the raw token string.* 2. To test the input and processor, enter the following command: ``` telegraf --test --config telegraf.conf ``` The output is similar to the following. For each row of input data, the filters pass the metric name, tags, specified fields, and timestamp. ``` > cpu,cpu=cpu0 usage_idle=100,usage_system=0 1702067201000000000 ... > cpu,cpu=cpu-total usage_idle=99.80198019802448,usage_system=0.1980198019802045 1702067201000000000 ``` ### Coerce data types to avoid rejected point errors Use Telegraf and the [Converter processor plugin](/telegraf/v1/plugins/#processor-converter)to convert field data types to fit your schema. For example, if you write the sample data in[Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data)to a database and then try to write the following batch to the same table: ```text home,room=Kitchen temp=23.1,hum=36.6,co=22.1 1641063600 home,room=Living\ Room temp=22i,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200 ``` InfluxDB expects `co` to contain an integer value and rejects points with `co`floating-point decimal (`22.1`) values. To avoid the error, configure Telegraf to convert fields to the data types in your schema columns. The following example converts the `temp`, `hum`, and `co` fields to fit the[sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data) schema: 1. In your terminal, enter the following command to create the sample data file: ``` cat < ./home.lp home,room=Kitchen temp=23.1,hum=36.6,co=22.1 1641063600 home,room=Living\ Room temp=22i,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200 EOF ``` 2. Enter the following command to create a Telegraf configuration that parses the sample data, converts the field values to the specified data types, and then writes the data to InfluxDB: ``` cat < ./telegraf.conf [[inputs.file]] ## For each interval, parse data from files in the list. files = ["home.lp"] influx_timestamp_precision = "1s" precision = "1s" tagexclude = ["host"] [[processors.converter]] [processors.converter.fields] ## A data type and a list of fields to convert to the data type. float = ["temp", "hum"] integer = ["co"] [[outputs.influxdb_v3]] ## InfluxDB credentials and the database to write to. urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" EOF ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) *Store this in a secret store or environment variable to avoid exposing the raw token string.* 3. To test the input and processor, enter the following command: ``` telegraf --test --config telegraf.conf ``` Telegraf outputs the following to stdout, and then exits: ``` > home,room=Kitchen co=22i,hum=36.6,temp=23.1 1641063600000000000 > home,room=Living\ Room co=17i,hum=36.4,temp=22 1641067200000000000 > home,room=Kitchen co=26i,hum=36.5,temp=22.7 1641067200000000000 ``` ### Merge lines to optimize memory and bandwidth Use Telegraf and the [Merge aggregator plugin](/telegraf/v1/plugins/#aggregator-merge)to merge points that share the same measurement, tag set, and timestamp. The following example creates sample data for two series (the combination of table, tag set, and timestamp), and then merges points in each series: 1. In your terminal, enter the following command to create the sample data file and calculate the number of seconds between the earliest timestamp and *now*. The command assigns the calculated value to a `grace_duration` variable that you’ll use in the next step. ``` cat < ./home.lp home,room=Kitchen temp=23.1 1641063600 home,room=Kitchen hum=36.6 1641063600 home,room=Kitchen co=22i 1641063600 home,room=Living\ Room temp=22.7 1641063600 home,room=Living\ Room hum=36.4 1641063600 home,room=Living\ Room co=17i 1641063600 EOF grace_duration="$(($(date +%s)-1641063000))s" ``` 2. Enter the following command to configure Telegraf to parse the file, merge the points, and write the data to InfluxDB–specifically, the configuration sets the following properties: * `influx_timestamp_precision`: for parsers, specifies the timestamp precision in the input data * Optional: `aggregators.merge.grace` extends the duration for merging points. To ensure the sample data is included, the configuration uses the calculated variable from the preceding step. ``` cat < ./telegraf.conf # Parse metrics from a file [[inputs.file]] ## A list of files to parse during each interval. files = ["home.lp"] ## The precision of timestamps in your data. influx_timestamp_precision = "1s" tagexclude = ["host"] # Merge separate metrics that share a series key [[aggregators.merge]] grace = "$grace_duration" ## If true, drops the original metric. drop_original = true # Writes metrics as line protocol to InfluxDB [[outputs.influxdb_v3]] ## InfluxDB credentials and the database to write data to. urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" EOF ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) *Store this in a secret store or environment variable to avoid exposing the raw token string.* 3. To test the input and aggregator, enter the following command: ``` telegraf --test --config telegraf.conf ``` Telegraf outputs the following to stdout, and then exits: ``` > home,room=Kitchen co=22i,hum=36.6,temp=23.1 1641063600000000000 > home,room=Living\ Room co=17i,hum=36.4,temp=22.7 1641063600000000000 ``` ### Avoid sending duplicate data Use Telegraf and the [Dedup processor plugin](/telegraf/v1/plugins/#processor-dedup)to filter data whose field values are exact repetitions of previous values. Deduplicating your data can reduce your write payload size and resource usage. The following example shows how to use Telegraf to remove points that repeat field values, and then write the data to InfluxDB: 1. In your terminal, enter the following command to create the sample data file and calculate the number of seconds between the earliest timestamp and *now*. The command assigns the calculated value to a `dedup_duration` variable that you’ll use in the next step. ``` cat < ./home.lp home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.5,hum=36.4,co=17i 1641063600 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641063605 home,room=Living\ Room temp=22.5,hum=36.4,co=17i 1641063605 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063610 home,room=Living\ Room temp=23.0,hum=36.4,co=17i 1641063610 EOF dedup_duration="$(($(date +%s)-1641063000))s" ``` 2. Enter the following command to configure Telegraf to parse the file, drop duplicate points, and write the data to InfluxDB–specifically, the sample configuration sets the following: * `influx_timestamp_precision`: for parsers, specifies the timestamp precision in the input data * `processors.dedup`: configures the Dedup processor plugin * Optional: `processors.dedup.dedup_interval`. Points in the range`dedup_interval` *to now* are considered for removal. To ensure the sample data is included, the configuration uses the calculated variable from the preceding step. ``` cat < ./telegraf.conf # Parse metrics from a file [[inputs.file]] ## A list of files to parse during each interval. files = ["home.lp"] ## The precision of timestamps in your data. influx_timestamp_precision = "1s" tagexclude = ["host"] # Filter metrics that repeat previous field values [[processors.dedup]] ## Drops duplicates within the specified duration dedup_interval = "$dedup_duration" # Writes metrics as line protocol to InfluxDB [[outputs.influxdb_v3]] ## InfluxDB credentials and the database to write data to. urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" EOF ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) *Store this in a secret store or environment variable to avoid exposing the raw token string.* 3. To test the input and processor, enter the following command: ``` telegraf --test --config telegraf.conf ``` Telegraf outputs the following to stdout, and then exits: ``` > home,room=Kitchen co=22i,hum=36.6,temp=23.1 1641063600000000000 > home,room=Living\ Room co=17i,hum=36.4,temp=22.5 1641063600000000000 > home,room=Kitchen co=26i,hum=36.5,temp=22.7 1641063605000000000 > home,room=Kitchen co=22i,hum=36.6,temp=23.1 1641063610000000000 > home,room=Living\ Room co=17i,hum=36.4,temp=23 1641063610000000000 ``` ### Run custom preprocessing code Use Telegraf and the [Execd processor plugin](/telegraf/v1/plugins/#processor-execd)to execute code external to Telegraf and then write the processed data. The Execd plugin expects line protocol data in stdin, passes the data to the configured executable, and then outputs line protocol to stdout. The following example shows how to use Telegraf to execute Go code for processing metrics and then write the output to InfluxDB. The Go `multiplier.go` sample code does the following: 1. Imports `influx` parser and serializer plugins from Telegraf. 2. Parses each line of data into a Telegraf metric. 3. If the metric contains a `count` field, multiplies the field value by `2`; otherwise, prints a message to stderr and exits. 4. In your editor, enter the following sample code and save the file as `multiplier.go`: ``` package main import ( "fmt" "os" "github.com/influxdata/telegraf/plugins/parsers/influx" influxSerializer "github.com/influxdata/telegraf/plugins/serializers/influx" ) func main() { parser := influx.NewStreamParser(os.Stdin) serializer := influxSerializer.Serializer{} if err := serializer.Init(); err != nil { fmt.Fprintf(os.Stderr, "serializer init failed: %v\n", err) os.Exit(1) } for { metric, err := parser.Next() if err != nil { if err == influx.EOF { return // stream ended } if parseErr, isParseError := err.(*influx.ParseError); isParseError { fmt.Fprintf(os.Stderr, "parse ERR %v\n", parseErr) os.Exit(1) } fmt.Fprintf(os.Stderr, "ERR %v\n", err) os.Exit(1) } c, found := metric.GetField("count") if !found { fmt.Fprintf(os.Stderr, "metric has no count field\n") os.Exit(1) } switch t := c.(type) { case float64: t *= 2 metric.AddField("count", t) case int64: t *= 2 metric.AddField("count", t) default: fmt.Fprintf(os.Stderr, "count is not an unknown type, it's a %T\n", c) os.Exit(1) } b, err := serializer.Serialize(metric) if err != nil { fmt.Fprintf(os.Stderr, "ERR %v\n", err) os.Exit(1) } fmt.Fprint(os.Stdout, string(b)) } } ``` 5. Initialize the module and install dependencies: ``` go mod init processlp go mod tidy ``` 6. In your terminal, enter the following command to create the sample data file: ``` cat < ./home.lp home,room=Kitchen temp=23.1,count=1 1641063600 home,room=Living\ Room temp=22.7,count=1 1641063600 home,room=Kitchen temp=23.1 1641063601 home,room=Living\ Room temp=22.7 1641063601 EOF ``` 7. Enter the following command to configure Telegraf to parse the file, execute the Go binary, and write the data–specifically, the sample configuration sets the following: * `influx_timestamp_precision`: for parsers, specifies the timestamp precision in the input data * `processors.execd`: configures the Execd plugin * `processors.execd.command`: sets the executable and arguments for Execd to run ``` cat < ./telegraf.conf # Parse metrics from a file [[inputs.file]] ## A list of files to parse during each interval. files = ["home.lp"] ## The precision of timestamps in your data. influx_timestamp_precision = "1s" tagexclude = ["host"] # Filter metrics that repeat previous field values [[processors.execd]] ## A list that contains the executable command and arguments to run as a daemon. command = ["go", "run", "multiplier.go"] # Writes metrics as line protocol to InfluxDB [[outputs.influxdb_v3]] ## InfluxDB credentials and the database to write data to. urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" EOF ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) *Store this in a secret store or environment variable to avoid exposing the raw token string.* 8. To test the input and processor, enter the following command: ``` telegraf --test --config telegraf.conf ``` Telegraf outputs the following to stdout, and then exits: ``` > home,room=Kitchen count=2,temp=23.1 1641063600000000000 > home,room=Living\ Room count=2,temp=22.7 1641063600000000000 ``` #### Related * [How to Ingest Data in InfluxDB (Video)](/resources/videos/ingest-data/) * [Use Telegraf to write data](/influxdb3/enterprise/write-data/use-telegraf/) --- # InfluxDB schema design recommendations Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/best-practices/schema-design/ Use the following guidelines to design your [schema](/influxdb3/enterprise/reference/glossary/#schema)for simpler and more performant queries. * [InfluxDB data structure](#influxdb-data-structure) * [Primary keys](#primary-keys) * [Tags versus fields](#tags-versus-fields) * [Schema restrictions](#schema-restrictions) * [Do not use duplicate names for tags and fields](#do-not-use-duplicate-names-for-tags-and-fields) * [Maximum number of columns per table](#maximum-number-of-columns-per-table) * [Design for performance](#design-for-performance) * [Sort tags by query priority](#sort-tags-by-query-priority) * [Avoid wide schemas](#avoid-wide-schemas) * [Avoid sparse schemas](#avoid-sparse-schemas) * [Table schemas should be homogenous](#table-schemas-should-be-homogenous) * [Use the best data type for your data](#use-the-best-data-type-for-your-data) * [Design for query simplicity](#design-for-query-simplicity) * [Keep table names, tags, and fields simple](#keep-table-names-tags-and-fields-simple) * [Avoid keywords and special characters](#avoid-keywords-and-special-characters) ## InfluxDB data structure The InfluxDB 3 Enterprise data model organizes time series data into databases and tables. A database can contain multiple tables. Tables contain multiple tags and fields. * **Database**: A named location where time series data is stored. In InfluxDB 3 Enterprise, *database* is synonymous with *bucket* in InfluxDB Cloud Serverless and InfluxDB TSM implementations. A database can contain multiple *tables*. * **Table**: A logical grouping for time series data. In InfluxDB 3 Enterprise, *table* is synonymous with *measurement* in InfluxDB Cloud Serverless and InfluxDB TSM implementations. All *points* in a given table should have the same *tags*. A table contains multiple *tags* and *fields*. * **Tags**: Key-value pairs that store metadata string values for each point–for example, a value that identifies or differentiates the data source or context–for example, host, location, station, etc. Tag values may be null. * **Fields**: Key-value pairs that store data for each point–for example, temperature, pressure, stock price, etc. Field values may be null, but at least one field value is not null on any given row. * **Timestamp**: Timestamp associated with the data. When stored on disk and queried, all data is ordered by time. In InfluxDB, a timestamp is a nanosecond-scale[Unix timestamp](/influxdb3/enterprise/reference/glossary/#unix-timestamp)in UTC. A timestamp is never null. > [!Note] > #### What happened to buckets and measurements? > > If coming from earlier versions of InfluxDB, InfluxDB Cloud (TSM), or > InfluxDB Cloud Serverless, you’re likely familiar with the concepts *bucket*and *measurement*: > > * ***Bucket*** in InfluxDB v2 or InfluxDB Cloud Serverless is synonymous with***database*** in InfluxDB 3 Enterprise. > * ***Measurement*** in InfluxDB v1, v2, or InfluxDB Cloud Serverless is synonymous > with ***table*** in InfluxDB 3 Enterprise. ### Primary keys In time series data, the primary key for a row of data is typically a combination of timestamp and other attributes that uniquely identify each data point. In InfluxDB 3 Enterprise, the primary key for a row is the combination of the point’s timestamp and *tag set*—the collection of[tag keys](/influxdb3/enterprise/reference/glossary/#tag-key) and[tag values](/influxdb3/enterprise/reference/glossary/#tag-value) on the point. A row’s primary key tag set does not include tags with null values. ### Tags versus fields When designing your schema for InfluxDB, a common question is, “what should be a tag and what should be a field?” The following guidelines should help answer that question as you design your schema. * Use tags to store metadata, or identifying information, about the source or context of the data. * Use fields to store measured values. * Tag values can only be strings. * Field values can be any of the following data types: * Integer * Unsigned integer * Float * String * Boolean > [!Note] > The InfluxDB 3 storage engine supports infinite tag value and series cardinality. > Unlike previous versions of InfluxDB, **tag value** cardinality doesn’t affect > the overall performance of your database. ## Schema restrictions ### Do not use duplicate names for tags and fields Use unique names for tags and fields within the same table. InfluxDB 3 Enterprise stores tags and fields as unique columns in a table. If you attempt to write a table that contains tags or fields with the same name, the write fails due to a column conflict. ### Maximum number of columns per table A table has a [maximum number of columns](/influxdb3/enterprise/admin/databases/#column-limit). Each row must include a time column. As a result, a table can have the following: * a time column * field and tag columns up to the configured maximum If you attempt to write to a table and exceed the column limit, then the write request fails and InfluxDB returns an error. InfluxData identified the[column limit](/influxdb3/enterprise/admin/databases/#column-limit)as the safe limit for maintaining system performance and stability. Exceeding this threshold can result in[wide schemas](#avoid-wide-schemas), which can negatively impact performance and resource use, depending on your queries, the shape of your schema, and data types in the schema. ## Design for performance How you structure your schema within a table can affect resource use and the performance of queries against that table. The following guidelines help to optimize query performance: * [Sort tags by query priority](#sort-tags-by-query-priority) * [Avoid wide schemas](#avoid-wide-schemas) * [Avoid sparse schemas](#avoid-sparse-schemas) * [Table schemas should be homogenous](#table-schemas-should-be-homogenous) * [Use the best data type for your data](#use-the-best-data-type-for-your-data) ### Sort tags by query priority The first write to a table in InfluxDB 3 Enterprise determines the physical column order in storage, and that order has a direct impact on query performance. Columns that appear earlier are typically faster to filter and access during query execution. Sort your tags by query priority when performing the initial write to a table. Place the most commonly queried tags first—those you frequently use in `WHERE`clauses or joins—followed by less frequently queried ones. For example, if most of your queries filter by `region` and then by `host`, structure your first write so that `region` comes before `host`. > [!Important] > Column order is determined on the first write and cannot be changed afterward. > Tags added after the first write are added last in the column sort order. > Plan your schema with your query workload in mind to ensure the best long-term > performance. ### Avoid wide schemas A wide schema refers to a schema with a large number of columns (tags and fields). Wide schemas can lead to the following issues: * Increased resource usage for persisting data during ingestion. * Reduced sorting performance due to complex primary keys with [too many tags](#avoid-too-many-tags). * Reduced query performance when selecting too many columns To prevent wide schema issues, limit the number of tags and fields stored in a table. If you need to store more than the [maximum number of columns](/influxdb3/enterprise/admin/databases/#column-limit), consider segmenting your fields into separate tables. If you collect Prometheus metrics with Telegraf, use metric version `1`. Metric version `2` writes every Prometheus metric name as a field in a single table, which creates a wide, sparse schema. See [Choose a metric version](/telegraf/v1/examples/scrape-prometheus/#choose-a-metric-version). #### Avoid too many tags In InfluxDB 3 Enterprise, the primary key for a row is the combination of the point’s timestamp and *tag set* - the collection of[tag keys](/influxdb3/enterprise/reference/glossary/#tag-key)and [tag values](/influxdb3/enterprise/reference/glossary/#tag-value) on the point. A point that contains more tags has a more complex primary key, which could impact sorting performance if you sort using all parts of the key. ### Avoid sparse schemas A sparse schema is one where, for many rows, columns contain null values. These generally stem from the following: * [non-homogenous table schemas](#table-schemas-should-be-homogenous) * [writing individual fields with different timestamps](#writing-individual-fields-with-different-timestamps) Sparse schemas require the InfluxDB query engine to evaluate many null columns, adding unnecessary overhead to storing and querying data. *For an example of a sparse schema,[view the non-homogenous schema example below](#view-example-of-a-sparse-non-homogenous-schema).* #### Writing individual fields with different timestamps Reporting fields at different times with different timestamps creates distinct rows that contain null values–for example: You report `fieldA` with `tagset`, and then report `field B` with the same`tagset`, but with a different timestamp. The result is two rows: one row has a *null* value for **field A** and the other has a *null* value for **field B**. In contrast, if you report fields at different times while using the same tagset and timestamp, the existing row is updated. This requires slightly more resources at ingestion time, but then gets resolved at persistence time or compaction time and avoids a sparse schema. ### Table schemas should be homogenous Data stored in a table should be “homogenous,” meaning each row should have the same tag and field keys. All rows stored in a table share the same columns, but if a point doesn’t include a value for a column, the column value is *null*. A table full of *null* values has a [“sparse” schema](#avoid-sparse-schemas). [](#view-example-of-a-sparse-non-homogenous-schema) View example of a sparse, non-homogenous schema Non-homogenous schemas are often caused by writing points to a table with inconsistent tag or field sets. In the following example, data is collected from two different sources and each source returns data with different tag and field sets. ##### Source 1 tags and fields: * tags: * source * code * crypto * fields: * price ##### Source 2 tags and fields: * tags: * src * currency * crypto * fields: * cost * volume These sets of data written to the same table result in a table full of null values (also known as a *sparse schema*): | time |source|src|code|currency|crypto | price | cost | volume | |--------------------|------|---|----|--------|-------|-----------|----------|-----------| |2025-01-01T12:00:00Z| src1 | |USD | |bitcoin|16588.45865| | | |2025-01-01T12:00:00Z| | 2 | | EUR |bitcoin| |16159.5806|16749450200| |2025-01-01T13:00:00Z| src1 | |USD | |bitcoin|16559.49871| | | |2025-01-01T13:00:00Z| | 2 | | EUR |bitcoin| |16131.3694|16829683245| |2025-01-01T14:00:00Z| src1 | |USD | |bitcoin|16577.46667| | | |2025-01-01T14:00:00Z| | 2 | | EUR |bitcoin| |16148.8727|17151722208| |2025-01-01T15:00:00Z| src1 | |USD | |bitcoin|16591.36998| | | |2025-01-01T15:00:00Z| | 2 | | EUR |bitcoin| |16162.4167|17311854919| ### Use the best data type for your data When writing data to a field, use the most appropriate[data type](/influxdb3/enterprise/reference/glossary/#data-type) for your data–write integers as integers, decimals as floats, and booleans as booleans. A query against a field that stores integers outperforms a query against string data; querying over many long string values can negatively affect performance. ## Design for query simplicity Naming conventions for tables, tag keys, and field keys can simplify or complicate the process of writing queries for your data. The following guidelines help to ensure writing queries for your data is as simple as possible. * [Keep table names, tags, and fields simple](#keep-table-names-tags-and-fields-simple) * [Avoid keywords and special characters](#avoid-keywords-and-special-characters) ### Keep table names, tags, and fields simple Use one tag or one field for each data attribute. If your source data contains multiple data attributes in a single parameter, split each attribute into its own tag or field. Table names, tag keys, and field keys should be simple and accurately describe what each contains. Keep names free of data. The most common cause of a complex naming convention is when you try to “embed” data attributes into a table name, tag key, or field key. When each key and value represents one attribute (not multiple concatenated attributes) of your data, you’ll reduce the need for regular expressions in your queries. Without regular expressions, your queries will be easier to write and more performant. #### Not recommended For example, consider the following [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/)that embeds multiple attributes (location, model, and ID) into a `sensor` tag value: ```text home,sensor=loc-kitchen.model-A612.id-1726ZA temp=72.1 home,sensor=loc-bath.model-A612.id-2635YB temp=71.8 ``` [](#view-written-data) View written data **table**: home | time | sensor |temp| |--------------------|--------------------------------|----| |2025-01-01T00:00:00Z|loc-kitchen.model-A612.id-1726ZA|72.1| |2025-01-01T00:00:00Z| loc-bath.model-A612.id-2635YB |71.8| To query data from the sensor with ID `1726ZA`, you have to use either SQL pattern matching or regular expressions to evaluate the `sensor` tag: [SQL](#sql)[InfluxQL](#influxql) ```sql SELECT * FROM home WHERE sensor LIKE '%id-1726ZA%' ``` ```sql SELECT * FROM home WHERE sensor =~ /id-1726ZA/ ``` SQL pattern matching and regular expressions both complicate the query and are less performant than simple equality expressions. #### Recommended The better approach would be to write each sensor attribute as a separate tag: ```text home,location=kitchen,sensor_model=A612,sensor_id=1726ZA temp=72.1 home,location=bath,sensor_model=A612,sensor_id=2635YB temp=71.8 ``` [](#view-written-data) View written data **table**: home | time |location|sensor\_model|sensor\_id|temp| |--------------------|--------|-------------|----------|----| |2023-01-01T00:00:00Z|kitchen | A612 | 1726ZA |72.1| |2023-01-01T00:00:00Z| bath | A612 | 2635YB |71.8| To query data from the sensor with ID `1726ZA` using this schema, you can use a simple equality expression: ```sql SELECT * FROM home WHERE sensor_id = '1726ZA' ``` This query is easier to write and is more performant than using pattern matching or regular expressions. ### Avoid keywords and special characters To simplify query writing, avoid using reserved keywords or special characters in table names, tag keys, and field keys. * [SQL keywords](/influxdb3/enterprise/reference/sql/#keywords) * [InfluxQL keywords](/influxdb3/enterprise/reference/influxql/#keywords) When using SQL or InfluxQL to query tables, tags, and fields with special characters or keywords, you have to wrap these identifiers in **double quotes**. ```sql SELECT "example-field", "tag@1-23" FROM "example-table" WHERE "tag@1-23" = 'ABC' ``` #### Related * [Manage databases](/influxdb3/enterprise/admin/databases/) * [influxdb3 CLI](/influxdb3/enterprise/reference/cli/influxdb3/) | time | source | src | code | currency | crypto | price | cost | volume | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | time | source | src | code | currency | crypto | price | cost | volume | | 2025-01-01T12:00:00Z | src1 | | USD | | bitcoin | 16588.45865 | | | | 2025-01-01T12:00:00Z | | 2 | | EUR | bitcoin | | 16159.5806 | 16749450200 | | 2025-01-01T13:00:00Z | src1 | | USD | | bitcoin | 16559.49871 | | | | 2025-01-01T13:00:00Z | | 2 | | EUR | bitcoin | | 16131.3694 | 16829683245 | | 2025-01-01T14:00:00Z | src1 | | USD | | bitcoin | 16577.46667 | | | | 2025-01-01T14:00:00Z | | 2 | | EUR | bitcoin | | 16148.8727 | 17151722208 | | 2025-01-01T15:00:00Z | src1 | | USD | | bitcoin | 16591.36998 | | | | 2025-01-01T15:00:00Z | | 2 | | EUR | bitcoin | | 16162.4167 | 17311854919 | | time | sensor | temp | | --- | --- | --- | | time | sensor | temp | | 2025-01-01T00:00:00Z | loc-kitchen.model-A612.id-1726ZA | 72.1 | | 2025-01-01T00:00:00Z | loc-bath.model-A612.id-2635YB | 71.8 | | time | location | sensor_model | sensor_id | temp | | --- | --- | --- | --- | --- | | time | location | sensor_model | sensor_id | temp | | 2023-01-01T00:00:00Z | kitchen | A612 | 1726ZA | 72.1 | | 2023-01-01T00:00:00Z | bath | A612 | 2635YB | 71.8 | --- # Use InfluxDB client libraries to write data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/client-libraries/ Use InfluxDB 3 client libraries that integrate with your code to construct data as time series points, and then write them as line protocol to an InfluxDB 3 Enterprise database. * [Set up your project](#set-up-your-project) * [Initialize a project directory](#initialize-a-project-directory) * [Install the client library](#install-the-client-library) * [Construct line protocol](#construct-line-protocol) * [Example home schema](#example-home-schema) ## Set up your project Set up your InfluxDB 3 Enterprise project and credentials to write data using the InfluxDB 3 client library for your programming language of choice. 1. [Install InfluxDB 3 Enterprise](/influxdb3/enterprise/install/) 2. [Set up InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/setup/) 3. Create a project directory and store your InfluxDB 3 Enterprise credentials as environment variables or in a project configuration file, such as a `.env` (“dotenv”) file. After setting up InfluxDB 3 Enterprise and your project, you should have the following: * InfluxDB 3 Enterprise credentials: * [Database](/influxdb3/enterprise/admin/databases/) * [Authorization token](/influxdb3/enterprise/admin/tokens/) * InfluxDB 3 Enterprise URL * A directory for your project. * Credentials stored as environment variables or in a project configuration file–for example, a `.env` (“dotenv”) file. ### Initialize a project directory Create a project directory and initialize it for your programming language. [Go](#go)[Node.js](#nodejs)[Python](#python) 1. Install [Go 1.13 or later](https://golang.org/doc/install). 2. Create a directory for your Go module and change to the directory–for example: ``` mkdir iot-starter-go && cd $_ ``` 3. Initialize a Go module–for example: ``` go mod init iot-starter ``` 1. Install [Node.js](https://nodejs.org/en/download/). 2. Create a directory for your JavaScript project and change to the directory–for example: ``` mkdir -p iot-starter-js && cd $_ ``` 3. Initialize a project–for example, using `npm`: ``` npm init ``` 1. Install [Python](https://www.python.org/downloads/) 2. Inside of your project directory, create a directory for your Python module and change to the module directory–for example: ``` mkdir -p iot-starter-py && cd $_ ``` 3. **Optional, but recommended**: Use[`venv`](https://docs.python.org/3/library/venv.html) or[`conda`](https://docs.continuum.io/anaconda/install/) to activate a virtual environment for installing and executing code–for example, enter the following command using `venv` to create and activate a virtual environment for the project: ``` python3 -m venv envs/iot-starter && source ./envs/iot-starter/bin/activate ``` ### Install the client library Install the InfluxDB 3 client library for your programming language of choice. [C#](#c)[Go](#go-2)[Java](#java)[Node.js](#nodejs-2)[Python](#python-2) Add the [InfluxDB 3 C# client library](https://github.com/InfluxCommunity/influxdb3-csharp) to your project using the[`dotnet` CLI](https://docs.microsoft.com/dotnet/core/tools/dotnet) or by adding the package to your project file–for example: ```bash dotnet add package InfluxDB3.Client ``` Add the[InfluxDB 3 Go client library](https://github.com/InfluxCommunity/influxdb3-go)to your project using the[`go get` command](https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them)–for example: ```bash go mod init path/to/project/dir && cd $_ go get github.com/InfluxCommunity/influxdb3-go/v2/influxdb3 ``` Add the [InfluxDB 3 Java client library](https://github.com/InfluxCommunity/influxdb3-java) to your project dependencies using the [Maven](https://maven.apache.org/)[Gradle](https://gradle.org/) build tools. For example, to add the library to a Maven project, add the following dependency to your `pom.xml` file: ```xml com.influxdb influxdb3-java 1.1.0 ``` To add the library to a Gradle project, add the following dependency to your `build.gradle` file: ```groovy dependencies { implementation 'com.influxdb:influxdb3-java:1.1.0' } ``` For a Node.js project, use `@influxdata/influxdb3-client`, which provides main (CommonJS), module (ESM), and browser (UMD) exports. Add the [InfluxDB 3 JavaScript client library](https://github.com/InfluxCommunity/influxdb3-js) using your preferred package manager–for example, using [`npm`](https://www.npmjs.com/): ```bash npm install --save @influxdata/influxdb3-client ``` Install the [InfluxDB 3 Python client library](https://github.com/InfluxCommunity/influxdb3-python) using[`pip`](https://pypi.org/project/pip/). To use Pandas features, such as `to_pandas()`, provided by the Python client library, you must also install the[`pandas` package](https://pandas.pydata.org/). ```bash pip install influxdb3-python pandas ``` ## Construct line protocol With a [basic understanding of line protocol](/influxdb3/enterprise/write-data/#line-protocol), you can construct line protocol data and write it to InfluxDB 3 Enterprise. Use client library write methods to provide data as raw line protocol or as `Point` objects that the client library converts to line protocol. If your program creates the data you write to InfluxDB, the `Point`interface to take advantage of type safety in your program. Client libraries provide one or more `Point` constructor methods. Some libraries support language-native data structures, such as Go’s `struct`, for creating points. Examples in this guide show how to construct `Point` objects that follow the [example `home` schema](#example-home-schema), and then write the points as line protocol data to an InfluxDB 3 Enterprise database. ### Example home schema Consider a use case where you collect data from sensors in your home. Each sensor collects temperature, humidity, and carbon monoxide readings. To collect this data, use the following schema: * **table**: `home` * **tags** * `room`: Living Room or Kitchen * **fields** * `temp`: temperature in °C (float) * `hum`: percent humidity (float) * `co`: carbon monoxide in parts per million (integer) * **timestamp**: Unix timestamp in *second* precision [Go](#go-3)[Node.js](#nodejs-3)[Python](#python-3) 1. Create a file for your module–for example: `main.go`. 2. In `main.go`, enter the following sample code: ``` package main import ( "context" "os" "fmt" "time" "github.com/InfluxCommunity/influxdb3-go/v2/influxdb3" "github.com/influxdata/line-protocol/v2/lineprotocol" ) func Write() error { url := os.Getenv("INFLUX_HOST") token := os.Getenv("INFLUX_TOKEN") database := os.Getenv("INFLUX_DATABASE") // To instantiate a client, call New() with InfluxDB credentials. client, err := influxdb3.New(influxdb3.ClientConfig{ Host: url, Token: token, Database: database, }) /** Use a deferred function to ensure the client is closed when the * function returns. **/ defer func (client *influxdb3.Client) { err = client.Close() if err != nil { panic(err) } }(client) /** Use the NewPoint method to construct a point. * NewPoint(measurement, tags map, fields map, time) **/ point := influxdb3.NewPoint("home", map[string]string{ "room": "Living Room", }, map[string]any{ "temp": 24.5, "hum": 40.5, "co": 15i}, time.Now(), ) /** Use the NewPointWithMeasurement method to construct a point with * method chaining. **/ point2 := influxdb3.NewPointWithMeasurement("home"). SetTag("room", "Living Room"). SetField("temp", 23.5). SetField("hum", 38.0). SetField("co", 16i). SetTimestamp(time.Now()) fmt.Println("Writing points") points := []*influxdb3.Point{point, point2} /** Write points to InfluxDB. * You can specify WriteOptions, such as Gzip threshold, * default tags, and timestamp precision. Default precision is lineprotocol.Nanosecond **/ err = client.WritePoints(context.Background(), points, influxdb3.WithPrecision(lineprotocol.Second)) return nil } func main() { Write() } ``` 3. To run the module and write the data to your InfluxDB 3 Enterprise database, enter the following command in your terminal: ``` go run main.go ``` 1. Create a file for your module–for example: `write-points.js`. 2. In `write-points.js`, enter the following sample code: ``` // write-points.js import { InfluxDBClient, Point } from '@influxdata/influxdb3-client'; /** * Set InfluxDB credentials. */ const host = process.env.INFLUX_HOST ?? ''; const database = process.env.INFLUX_DATABASE; const token = process.env.INFLUX_TOKEN; /** * Write line protocol to InfluxDB using the JavaScript client library. */ export async function writePoints() { /** * Instantiate an InfluxDBClient. * Provide the host URL and the database token. */ const client = new InfluxDBClient({ host, token }); /** Use the fluent interface with chained methods to construct Points. */ const point = Point.measurement('home') .setTag('room', 'Living Room') .setFloatField('temp', 22.2) .setFloatField('hum', 35.5) .setIntegerField('co', 7) .setTimestamp(new Date().getTime() / 1000); const point2 = Point.measurement('home') .setTag('room', 'Kitchen') .setFloatField('temp', 21.0) .setFloatField('hum', 35.9) .setIntegerField('co', 0) .setTimestamp(new Date().getTime() / 1000); /** Write points to InfluxDB. * The write method accepts an array of points, the target database, and * an optional configuration object. * You can specify WriteOptions, such as Gzip threshold, default tags, * and timestamp precision. Default precision is lineprotocol.Nanosecond **/ try { await client.write([point, point2], database, '', { precision: 's' }); console.log('Data has been written successfully!'); } catch (error) { console.error(`Error writing data to InfluxDB: ${error.body}`); } client.close(); } writePoints(); ``` 3. To run the module and write the data to your {{\< product-name \>}} database, enter the following command in your terminal: ``` node writePoints.js ``` 1. Create a file for your module–for example: `write-points.py`. 2. In `write-points.py`, enter the following sample code to write data in batching mode: ``` import os from influxdb_client_3 import ( InfluxDBClient3, InfluxDBError, Point, WritePrecision, WriteOptions, write_client_options) host = os.getenv('INFLUX_HOST') token = os.getenv('INFLUX_TOKEN') database = os.getenv('INFLUX_DATABASE') # Create an array of points with tags and fields. points = [Point("home") .tag("room", "Kitchen") .field("temp", 25.3) .field('hum', 20.2) .field('co', 9)] # With batching mode, define callbacks to execute after a successful or # failed write request. # Callback methods receive the configuration and data sent in the request. def success(self, data: str): print(f"Successfully wrote batch: data: {data}") def error(self, data: str, exception: InfluxDBError): print(f"Failed writing batch: config: {self}, data: {data} due: {exception}") def retry(self, data: str, exception: InfluxDBError): print(f"Failed retry writing batch: config: {self}, data: {data} retry: {exception}") # Configure options for batch writing. write_options = WriteOptions(batch_size=500, flush_interval=10_000, jitter_interval=2_000, retry_interval=5_000, max_retries=5, max_retry_delay=30_000, exponential_base=2) # Create an options dict that sets callbacks and WriteOptions. wco = write_client_options(success_callback=success, error_callback=error, retry_callback=retry, write_options=write_options) # Instantiate a synchronous instance of the client with your # InfluxDB credentials and write options, such as Gzip threshold, default tags, # and timestamp precision. Default precision is nanosecond ('ns'). with InfluxDBClient3(host=host, token=token, database=database, write_client_options=wco) as client: client.write(points, write_precision='s') ``` 3. To run the module and write the data to your InfluxDB 3 Enterprise database, enter the following command in your terminal: ``` python write-points.py ``` The sample code does the following: 1. Instantiates a client configured with the InfluxDB URL and API token. 2. Constructs `home`table `Point` objects. 3. Sends data as line protocol format to InfluxDB and waits for the response. 4. If the write succeeds, logs the success message to stdout; otherwise, logs the failure message and error details. 5. Closes the client to release resources. #### Related * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/) * [InfluxDB 3 API client libraries](/influxdb3/enterprise/reference/client-libraries/v3/) * [/api/v3/write\_lp endpoint](/influxdb3/enterprise/api/write-data/#operation/PostWriteLP) --- # Use the InfluxDB HTTP API to write data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/http-api/ Use the InfluxDB HTTP API to write data to InfluxDB 3 Enterprise. Different APIs are available depending on your integration method. > [!Tip] > #### Choose the write endpoint for your workload > > When creating new write workloads, use the[InfluxDB HTTP API `/api/v3/write_lp` endpoint](/influxdb3/enterprise/write-data/http-api/v3-write-lp/)and [client libraries](/influxdb3/enterprise/write-data/client-libraries/). > > When bringing existing *v1* write workloads, use the InfluxDB 3 Enterprise > HTTP API [`/write` endpoint](/influxdb3/core/api/write-data/#operation/PostV1Write). > > When bringing existing *v2* write workloads, use the InfluxDB 3 Enterprise > HTTP API [`/api/v2/write` endpoint](/influxdb3/enterprise/api/write-data/). > > **For Telegraf**, use the [`outputs.influxdb_v3`](/telegraf/v1/output-plugins/influxdb_v3/) output plugin. > For existing v1 or v2 write workloads, use the v1.x [`outputs.influxdb`](/telegraf/v1/output-plugins/influxdb/) or v2.x [`outputs.influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/) output plugins. > See how to [use Telegraf to write data](/influxdb3/enterprise/write-data/use-telegraf/). ### [Use the v3 write\_lp API to write data](/influxdb3/enterprise/write-data/http-api/v3-write-lp/) Use the `/api/v3/write_lp` HTTP API endpoint to write data to InfluxDB 3 Enterprise. ### [Use compatibility APIs and client libraries to write data](/influxdb3/enterprise/write-data/http-api/compatibility-apis/) Use HTTP API endpoints compatible with InfluxDB v2 and v1 clients to write points as line protocol data to InfluxDB 3 Enterprise. #### Related * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/) * [/api/v3/write\_lp endpoint](/influxdb3/enterprise/api/write-data/#operation/PostWriteLP) --- # Use compatibility APIs and client libraries to write data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/http-api/compatibility-apis/ Use compatibility APIs when you need to migrate existing InfluxDB v1 or v2 write workloads to InfluxDB 3 Enterprise. The `/api/v2/write` (v2-compatible) and `/write` (v1-compatible) HTTP API endpoints work with InfluxDB [client libraries](/influxdb3/enterprise/reference/client-libraries/), [Telegraf](/telegraf/v1/), and third-party integrations to write points as line protocol data to InfluxDB 3 Enterprise. > [!Tip] > #### Choose the write endpoint for your workload > > When creating new write workloads, use the[InfluxDB HTTP API `/api/v3/write_lp` endpoint](/influxdb3/enterprise/write-data/http-api/v3-write-lp/)and [client libraries](/influxdb3/enterprise/write-data/client-libraries/). > > When bringing existing v1 write workloads, use the[v1-compatible `/write` endpoint](#influxdb-v1-compatibility). > > When bringing existing v2 write workloads, use the[v2-compatible `/api/v2/write` endpoint](#influxdb-v2-compatibility). > > **For Telegraf**, use the InfluxDB v1.x [`outputs.influxdb`](/telegraf/v1/output-plugins/influxdb/) or v2.x [`outputs.influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/) output plugins. > For new write workloads, use the [`outputs.influxdb_v3`](/telegraf/v1/output-plugins/influxdb_v3/) output plugin, which writes to the native [`/api/v3/write_lp` endpoint](/influxdb3/enterprise/write-data/http-api/v3-write-lp/). > See how to [use Telegraf to write data](/influxdb3/enterprise/write-data/use-telegraf/). > [!Note] > #### Compatibility APIs differ from native APIs > > Keep in mind that the compatibility APIs differ from the v1 and v2 APIs in previous versions in the following ways: > > * Tags in a table (measurement) are *immutable* > * A tag and a field can’t have the same name within a table. The `/write` and `/api/v2/write` endpoints are all-or-nothing. If any line in a batch is invalid, the endpoint rejects the whole batch and stores nothing. The `accept_partial` parameter has no effect on these endpoints. To store the valid lines of a batch that contains invalid lines, use the[`/api/v3/write_lp` endpoint](/influxdb3/enterprise/write-data/http-api/v3-write-lp/), which accepts partial writes by default. ## InfluxDB v2 compatibility The `/api/v2/write` InfluxDB v2 compatibility endpoint provides backwards compatibility with clients that can write data to InfluxDB OSS v2.x and Cloud 2 (TSM). ``` POST /api/v2/write?bucket=mydb&precision=ns ``` ### Authenticate v2 API requests InfluxDB 3 Enterprise requires each API request to be authenticated with a [token](/influxdb3/enterprise/admin/tokens/)with write access to the database . Use the `Authorization: Bearer` or `Authorization: Token` scheme to authenticate v2 API write requests: #### Syntax ```http Authorization: Bearer DATABASE_TOKEN ``` ```http Authorization: Token DATABASE_TOKEN ``` #### Examples Use `Bearer` to authenticate a v2 write request: ```sh curl -i "http://localhost:8181/api/v2/write?bucket=DATABASE_NAME&precision=s" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` Use `Token` to authenticate a v2 write request: ```sh curl -i "http://localhost:8181/api/v2/write?bucket=DATABASE_NAME&precision=s" \ --header "Authorization: Token DATABASE_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` ### v2 API write parameters For InfluxDB 3 Enterprise v2 API `/api/v2/write` requests, set parameters as listed in the following table: | Parameter | Allowed in |Ignored| Value | |------------------|------------|-------|-----------------------------------------------------| | `bucket` \* |Query string|Honored| Database name | | `precision` |Query string|Honored| [Timestamp precision](#timestamp-precision-v2) | |`Content-Encoding`| Header |Honored|`gzip` (compressed data) or `identity` (uncompressed)| | `Authorization` | Header |Honored| `Bearer DATABASE_TOKEN` or `Token DATABASE_TOKEN` | \* = Required #### Timestamp precision > [!Note] > By default, InfluxDB 3 Enterprise uses the timestamp magnitude to auto-detect the precision. > To avoid any ambiguity, you can specify the precision of timestamps in your data. Use one of the following `precision` values in v2 API `/api/v2/write` requests: * `ns`: nanoseconds * `us`: microseconds * `ms`: milliseconds * `s`: seconds * `m`: minutes * `h`: hours ## InfluxDB v1 compatibility The `/write` InfluxDB v1 compatibility endpoint provides backwards compatibility with clients that can write data to InfluxDB v1.x. ``` POST /write?db=mydb&precision=ns ``` ### Authenticate v1 API requests InfluxDB 3 Enterprise requires each API request to be authenticated with a [token](/influxdb3/enterprise/admin/tokens/)with write access to the database . With InfluxDB v1-compatible endpoints in InfluxDB 3, you can use database tokens in InfluxDB 1.x username and password schemes, in the InfluxDB v2 `Authorization: Token` scheme, or in the OAuth `Authorization: Bearer` scheme. * [Authenticate with a username and password scheme](#authenticate-with-a-username-and-password-scheme) * [Authenticate with a token scheme](#authenticate-with-a-token-scheme) #### Authenticate with a username and password scheme With InfluxDB v1-compatible endpoints, you can use the InfluxDB 1.x convention of username and password to authenticate database writes by passing a [token](/influxdb3/enterprise/admin/tokens/)with write access to the database as the `password` credential. When authenticating requests to the v1 API `/write` endpoint, InfluxDB 3 Enterprise checks that the `password` (`p`) value is an authorized [token](/influxdb3/enterprise/admin/tokens/)with write access to the database . InfluxDB 3 Enterprise ignores the `username` (`u`) parameter in the request. Use one of the following authentication schemes with clients that support Basic authentication or query parameters: * [Basic authentication](#basic-authentication-v1) * [Query string authentication](#query-string-authentication-v1) ##### Basic authentication Use the `Authorization` header with the `Basic` scheme to authenticate v1 API `/write` requests. When authenticating requests, InfluxDB 3 Enterprise checks that the `password` part of the decoded credential is an authorized [token](/influxdb3/enterprise/admin/tokens/)with write access to the database . InfluxDB 3 Enterprise ignores the `username` part of the decoded credential. ###### Syntax ```http Authorization: Basic ``` Encode the `[USERNAME]:DATABASE_TOKEN` credential using base64 encoding, and then append the encoded string to the `Authorization: Basic` header. ###### Example The following example shows how to use cURL with the `Basic` authentication scheme: ```sh curl -i "http://localhost:8181/write?db=DATABASE_NAME&precision=s" \ --user "any:DATABASE_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` ##### Query string authentication In the URL, pass the `p` query parameter to authenticate `/write` requests. When authenticating requests, InfluxDB 3 Enterprise checks that the `p` (*password*) value is an authorized [token](/influxdb3/enterprise/admin/tokens/)with write access to the database and ignores the `u` (*username*) parameter. ###### Syntax ```sh http://localhost:8181/write/?u=any&p=DATABASE_TOKEN ``` ###### Example The following example shows how to use cURL with query string authentication: ```sh curl -i "http://localhost:8181/write?db=DATABASE_NAME&precision=s&p=DATABASE_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` #### Authenticate with a token scheme Use the `Authorization: Bearer` or the `Authorization: Token` scheme to pass a [token](/influxdb3/enterprise/admin/tokens/)with write access to the database for authenticating v1 API `/write` requests. `Bearer` and `Token` are equivalent in InfluxDB 3 Enterprise. The `Token` scheme is used in the InfluxDB 2.x API.`Bearer` is defined by the [OAuth 2.0 Framework](https://www.rfc-editor.org/rfc/rfc6750#page-14). Support for one or the other may vary across InfluxDB API clients. ##### Syntax ```http Authorization: Bearer DATABASE_TOKEN ``` ```http Authorization: Token DATABASE_TOKEN ``` ##### Examples Use `Bearer` to authenticate a v1 write request: ```sh curl -i "http://localhost:8181/write?db=DATABASE_NAME&precision=s" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` Use `Token` to authenticate a v1 write request: ```sh curl -i "http://localhost:8181/write?db=DATABASE_NAME&precision=s" \ --header "Authorization: Token DATABASE_TOKEN" \ --header "Content-type: text/plain; charset=utf-8" \ --data-binary 'home,room=kitchen temp=72 1641024000' ``` ### v1 API write parameters For InfluxDB 3 Enterprise v1 API `/write` requests, set parameters as listed in the following table: | Parameter | Allowed in | Ignored | Value | |---------------------------------|------------|------------------------|------------------------------------------------------------------------------------------------------------------| | `consistency` |Query string| Ignored | N/A | | `db` \* |Query string| Honored | Database name | | `precision` |Query string| Honored | [Timestamp precision](#timestamp-precision-v1) | | `rp` |Query string|Honored, but discouraged| Retention policy | | `u` |Query string| Ignored | For [query string authentication](#query-string-authentication-v1), any arbitrary string | | `p` |Query string| Honored |For [query string authentication](#query-string-authentication-v1), a [token](/influxdb3/enterprise/admin/tokens/)| |with write access to the database| | | | `Content-Encoding` | Header | Honored | `gzip` (compressed data) or `identity` (uncompressed)`Authorization` | Header | Honored | `Bearer DATABASE_TOKEN`, `Token DATABASE_TOKEN`, or `Basic ` \* = Required #### Timestamp precision > [!Note] > By default, InfluxDB 3 Enterprise uses the timestamp magnitude to auto-detect the precision. > To avoid any ambiguity, you can specify the precision of timestamps in your data. Use one of the following `precision` values in v1 API `/write` requests: * `ns`: nanoseconds * `us`: microseconds * `ms`: milliseconds * `s`: seconds * `m`: minutes * `h`: hours ## Client library examples Use language-specific client libraries with your custom code to write data to InfluxDB 3 Enterprise. ### v1 client libraries v1 client libraries send data in [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) syntax to the v1 API `/write` endpoint. [Node.js](#nodejs)[Python](#python) Create a v1 API client using the [node-influx](/influxdb/v1/tools/api_client_libraries/#javascript) JavaScript client library: ```js const Influx = require('influx') // Instantiate a client for writing to InfluxDB 3 Enterprise v1 API const client = new Influx.InfluxDB({ host: 'localhost:8181', port: 443, protocol: 'https', database: 'DATABASE_NAME', username: 'ignored', password: 'DATABASE_TOKEN' }) ``` Create a v1 API client using the [influxdb-python](/influxdb/v1/tools/api_client_libraries/#python) Python client library: ```python from influxdb import InfluxDBClient # Instantiate a client for writing to InfluxDB 3 Enterprise v1 API client = InfluxDBClient( host='localhost:8181', ssl=True, database='DATABASE_NAME', username='', password='DATABASE_TOKEN', headers={'Content-Type': 'text/plain; charset=utf-8'} ) ``` ### v2 client libraries v2 client libraries send data in [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) syntax to the v2 API `/api/v2/write` endpoint. For more information about using v2 client libraries, see [v2 client libraries](/influxdb3/enterprise/reference/client-libraries/v2/). ## Telegraf configuration If you have existing v1 workloads that use Telegraf, you can use the [InfluxDB v1.x `influxdb` Telegraf output plugin](/telegraf/v1/output-plugins/influxdb/) to write data. The following table shows `outputs.influxdb` plugin parameters and values for writing to the InfluxDB 3 Enterprise v1 API: | Parameter | Ignored | Value | |---------------------------------|------------------------|--------------------------------------------------------------| | `database` | Honored | Database name | | `retention_policy` |Honored, but discouraged|[Duration](/influxdb3/enterprise/reference/glossary/#duration)| | `username` | Ignored | String or empty | | `password` | Honored | [token](/influxdb3/enterprise/admin/tokens/) | |with write access to the database| | | `content_encoding` | Honored | `gzip` (compressed data) or `identity` (uncompressed)`skip_database_creation` | Ignored | N/A (see how to [create a database](/influxdb3/enterprise/admin/databases/create/)) To configure the v1.x output plugin for writing to InfluxDB 3 Enterprise, add the following `outputs.influxdb` configuration in your `telegraf.conf` file: ```toml [[outputs.influxdb]] urls = ["http://localhost:8181"] database = "DATABASE_NAME" skip_database_creation = true retention_policy = "" username = "ignored" password = "DATABASE_TOKEN" content_encoding = "gzip" ``` Replace the following configuration values: * `DATABASE_NAME`: the name of the [database](/influxdb3/enterprise/admin/databases/) to write to * `DATABASE_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/)with write access to the database #### Related * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/) * [InfluxDB v2 API client libraries](/influxdb3/enterprise/reference/client-libraries/v2/) * [/api/v2/write (v2-compatible) endpoint](/influxdb3/enterprise/api/write-data/#operation/PostV2Write) * [/write (v1-compatible) endpoint](/influxdb3/enterprise/api/write-data/#operation/PostV1Write) | Parameter | Allowed in | Ignored | Value | | --- | --- | --- | --- | | Parameter | Allowed in | Ignored | Value | | bucket * | Query string | Honored | Database name | | precision | Query string | Honored | Timestamp precision | | Content-Encoding | Header | Honored | gzip (compressed data) or identity (uncompressed) | | Authorization | Header | Honored | Bearer DATABASE_TOKEN or Token DATABASE_TOKEN | | Parameter | Allowed in | Ignored | Value | | --- | --- | --- | --- | | Parameter | Allowed in | Ignored | Value | | consistency | Query string | Ignored | N/A | | db * | Query string | Honored | Database name | | precision | Query string | Honored | Timestamp precision | | rp | Query string | Honored, but discouraged | Retention policy | | u | Query string | Ignored | For query string authentication , any arbitrary string | | p | Query string | Honored | For query string authentication , a token | | with write access to the database | | | | | Parameter | Ignored | Value | | --- | --- | --- | | Parameter | Ignored | Value | | database | Honored | Database name | | retention_policy | Honored, but discouraged | Duration | | username | Ignored | String or empty | | password | Honored | token | | with write access to the database | | | --- # Use the v3 write_lp API to write data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/http-api/v3-write-lp/ Use the `/api/v3/write_lp` endpoint to write data to InfluxDB 3 Enterprise. This endpoint accepts the same [line protocol](/influxdb3/enterprise/reference/line-protocol/)syntax as previous versions of InfluxDB, and supports the following: ## Query parameters * `?accept_partial=`: Accept or reject partial writes (default is `true`). * `?no_sync=`: Control when writes are acknowledged: * `no_sync=true`: Acknowledges writes before WAL persistence completes. * `no_sync=false`: Acknowledges writes after WAL persistence completes (default). * `?precision=`: Specify the precision of the timestamp. By default, InfluxDB 3 Enterprise uses the timestamp magnitude to auto-detect the precision (`auto`). To avoid any ambiguity, you can specify the precision of timestamps in your data. The InfluxDB 3 Enterprise `/api/v3/write_lp` API endpoint supports the following timestamp precisions: * `auto` (automatic detection, default) * `nanosecond` (nanoseconds) * `microsecond` (microseconds) * `millisecond` (milliseconds) * `second` (seconds) ### Auto precision detection When you use `precision=auto` (or omit the parameter), InfluxDB 3 Enterprise automatically detects the timestamp precision based on the magnitude of the timestamp value: * Timestamps \< 5e9 → Second precision (multiplied by 1,000,000,000 to convert to nanoseconds) * Timestamps \< 5e12 → Millisecond precision (multiplied by 1,000,000) * Timestamps \< 5e15 → Microsecond precision (multiplied by 1,000) * Larger timestamps → Nanosecond precision (no conversion needed) ### Precision examples The following examples show how to write data with different timestamp precisions: [Auto (default)](#auto-default)[Nanoseconds](#nanoseconds)[Milliseconds](#milliseconds)[Seconds](#seconds) ```bash # Auto precision (default) - timestamp magnitude determines precision curl "http://localhost:8181/api/v3/write_lp?db=sensors" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-raw "cpu,host=server1 usage=50.0 1708976567" ``` The timestamp `1708976567` is automatically detected as seconds. ```bash # Explicit nanosecond precision curl "http://localhost:8181/api/v3/write_lp?db=sensors&precision=nanosecond" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-raw "cpu,host=server1 usage=50.0 1708976567000000000" ``` ```bash # Millisecond precision curl "http://localhost:8181/api/v3/write_lp?db=sensors&precision=millisecond" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-raw "cpu,host=server1 usage=50.0 1708976567000" ``` ```bash # Second precision curl "http://localhost:8181/api/v3/write_lp?db=sensors&precision=second" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-raw "cpu,host=server1 usage=50.0 1708976567" ``` ## Configure gzip compression The `/api/v3/write_lp` endpoint supports gzip-encoded request bodies for efficient data transfer. When sending gzip-compressed data to InfluxDB, include the `Content-Encoding: gzip` header in your InfluxDB API request. ### Multi-member gzip support InfluxDB 3 Enterprise supports multi-member gzip payloads (concatenated gzip files per [RFC 1952](https://www.rfc-editor.org/rfc/rfc1952)). This allows you to: * Concatenate multiple gzip files and send them in a single request * Maintain compatibility with InfluxDB v1 and v2 write endpoints * Simplify batch operations using standard compression tools #### Example: Write concatenated gzip files ```bash # Create multiple gzip files echo "cpu,host=server1 usage=50.0 1708976567" | gzip > batch1.gz echo "cpu,host=server2 usage=60.0 1708976568" | gzip > batch2.gz # Concatenate and send in a single request cat batch1.gz batch2.gz | curl "http://localhost:8181/api/v3/write_lp?db=sensors" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --header "Content-Encoding: gzip" \ --data-binary @- ``` ## Request body * Line protocol ``` POST /api/v3/write_lp?db=mydb&precision=nanosecond&accept_partial=true&no_sync=false ``` *The following example uses [cURL](https://curl.se/) to send a write request using the [Home sensor sample data](/influxdb3/enterprise/reference/sample-data/#home-sensor-data), but you can use any HTTP client.* ```bash curl -v "http://localhost:8181/api/v3/write_lp?db=sensors&precision=second" \ --data-raw "home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 home,room=Living\ Room temp=22.3,hum=36.1,co=0i 1641045600 home,room=Kitchen temp=22.8,hum=36.3,co=1i 1641045600 home,room=Living\ Room temp=22.3,hum=36.1,co=1i 1641049200 home,room=Kitchen temp=22.7,hum=36.2,co=3i 1641049200 home,room=Living\ Room temp=22.4,hum=36.0,co=4i 1641052800 home,room=Kitchen temp=22.4,hum=36.0,co=7i 1641052800 home,room=Living\ Room temp=22.6,hum=35.9,co=5i 1641056400 home,room=Kitchen temp=22.7,hum=36.0,co=9i 1641056400 home,room=Living\ Room temp=22.8,hum=36.2,co=9i 1641060000 home,room=Kitchen temp=23.3,hum=36.9,co=18i 1641060000 home,room=Living\ Room temp=22.5,hum=36.3,co=14i 1641063600 home,room=Kitchen temp=23.1,hum=36.6,co=22i 1641063600 home,room=Living\ Room temp=22.2,hum=36.4,co=17i 1641067200 home,room=Kitchen temp=22.7,hum=36.5,co=26i 1641067200" ``` * [Partial writes](#partial-writes) * [Accept partial writes](#accept-partial-writes) * [Do not accept partial writes](#do-not-accept-partial-writes) * [Write responses](#write-responses) * [Use no\_sync for immediate write responses](#use-no_sync-for-immediate-write-responses) > [!Note] > #### InfluxDB client libraries > > InfluxData provides supported InfluxDB 3 client libraries that you can > integrate with your code to construct data as time series points, and then > write them as line protocol to an InfluxDB 3 Enterprise database. > For more information, see how to [use InfluxDB client libraries to write data](/influxdb3/enterprise/write-data/client-libraries/). ## Partial writes The `/api/v3/write_lp` endpoint lets you accept or reject partial writes using the `accept_partial` parameter. This parameter changes the behavior of the API when the write request contains invalid line protocol or schema conflicts. For example, the following line protocol contains two points, each using a different datatype for the `temp` field, which causes a schema conflict: ```lp home,room=Sunroom temp=96 1735545600 home,room=Sunroom temp="hi" 1735549200 ``` ### Accept partial writes With `accept_partial=true` (default), InfluxDB: * Accepts and writes line `1` * Rejects line `2` * Returns a `400 Bad Request` status code and the following response body: ``` < HTTP/1.1 400 Bad Request ... { "error": "partial write of line protocol occurred", "data": [ { "original_line": "home,room=Sunroom temp=hi 1735549200", "line_number": 2, "error_message": "invalid column type for column 'temp', expected iox::column_type::field::float, got iox::column_type::field::string" } ] } ``` ### Do not accept partial writes With `accept_partial=false`, InfluxDB: * Rejects *all* points in the batch * Returns a `400 Bad Request` status code and the following response body: ``` < HTTP/1.1 400 Bad Request ... { "error": "parsing failed for write_lp endpoint", "data": { "original_line": "home,room=Sunroom temp=hi 1735549200", "line_number": 2, "error_message": "invalid column type for column 'temp', expected iox::column_type::field::float, got iox::column_type::field::string" } } ``` *For more information about the ingest path and data flow, see[Data durability](/influxdb3/enterprise/reference/internals/durability/).* ## Write responses By default, InfluxDB 3 Enterprise acknowledges writes after flushing the WAL file to the Object store (occurring every second). For high write throughput, you can send multiple concurrent write requests. ### Use no\_sync for immediate write responses To reduce the latency of writes, use the `no_sync` write option, which acknowledges writes *before* WAL persistence completes. When `no_sync=true`, InfluxDB validates the data, writes the data to the WAL, and then immediately responds to the client, without waiting for persistence to the Object store. > [!Tip] > Using `no_sync=true` is best when prioritizing high-throughput writes over > absolute durability. * Default behavior (`no_sync=false`): Waits for data to be written to the Object store before acknowledging the write. Reduces the risk of data loss, but increases the latency of the response. * With `no_sync=true`: Reduces write latency, but increases the risk of data loss in case of a crash before WAL persistence. The following example immediately returns a response without waiting for WAL persistence: ```bash curl "http://localhost:8181/api/v3/write_lp?db=sensors&no_sync=true" \ --data-raw "home,room=Sunroom temp=96" ``` ## Response headers All HTTP responses from InfluxDB 3 Enterprise include the following standard headers: ### cluster-uuid The `cluster-uuid` response header contains the catalog UUID of your InfluxDB 3 Enterprise instance. This header is included in all HTTP API responses, including: * Write requests (`/api/v3/write_lp`, `/api/v2/write`, `/write`) * Query requests * Administrative operations * Authentication failures * CORS preflight requests #### Use cases The `cluster-uuid` header enables you to: * **Identify cluster instances**: Programmatically determine which InfluxDB instance handled a request * **Monitor deployments**: Track requests across multiple InfluxDB instances in load-balanced or multi-cluster environments * **Debug and troubleshooting**: Correlate client requests with specific server instances in distributed systems #### Example response ```bash curl -v "http://localhost:8181/api/v3/write_lp?db=sensors" \ --header "Authorization: Bearer DATABASE_TOKEN" \ --data-raw "cpu,host=server1 usage=50.0" ``` The response headers contain the `cluster-uuid`: ``` < HTTP/1.1 204 No Content < cluster-uuid: 01234567-89ab-cdef-0123-456789abcdef < date: Tue, 19 Nov 2025 20:00:00 GMT ``` #### Related * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/) * [/api/v3/write\_lp endpoint](/influxdb3/enterprise/api/write-data/#operation/PostWriteLP) --- # Use the influxdb3 CLI to write data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/influxdb3-cli/ Use the [`influxdb3` CLI](/influxdb3/enterprise/reference/cli/influxdb3/)to write line protocol data to InfluxDB 3 Enterprise. * [Construct line protocol](#construct-line-protocol) * [Write the line protocol to InfluxDB](#write-the-line-protocol-to-influxdb) > [!Note] > #### Use the API for batching and higher-volume writes > > The `influxdb3` CLI lets you quickly get started writing data to InfluxDB 3 Enterprise. > For batching and higher-volume write workloads, use the[InfluxDB HTTP API](/influxdb3/enterprise/write-data/http-api),[API client libraries](/influxdb3/enterprise/write-data/client-libraries/)or [Telegraf](/influxdb3/enterprise/write-data/use-telegraf/). ## Construct line protocol With a [basic understanding of line protocol](/influxdb3/enterprise/write-data/#line-protocol), you can construct data in line protocol format and write it to InfluxDB 3 Enterprise. Consider a use case where you collect data from sensors in your home. Each sensor collects temperature, humidity, and carbon monoxide readings. To collect this data, use the following schema: * **table**: `home` * **tags** * `room`: Living Room or Kitchen * **fields** * `temp`: temperature in °C (float) * `hum`: percent humidity (float) * `co`: carbon monoxide in parts per million (integer) * **timestamp**: Unix timestamp in *second* precision The following line protocol represents the schema described above: ```text home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000 ``` For this tutorial, you can either pass this line protocol directly to the`influxdb3 write` command as a string, via `stdin`, or you can save it to and read it from a file. ## Write the line protocol to InfluxDB Use the [`influxdb3 write` command](/influxdb3/enterprise/reference/cli/influxdb3/write/)to write the home sensor sample data to InfluxDB 3 Enterprise. Provide the following: * The [database](/influxdb3/enterprise/admin/databases/) name using the`--database` option * Your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/)using the `-t`, `--token` option * [Line protocol](#construct-line-protocol). Provide the line protocol in one of the following ways: * a string * a path to a file that contains the line protocol using the `--file` option * from stdin > [!Note] > By default, InfluxDB 3 Enterprise uses the timestamp magnitude to auto-detect the precision. > To specify the precision of timestamps in your data, use the [`--precision {ns|us|ms|s}` option](/influxdb3/enterprise/reference/cli/influxdb3/write/#options). [string](#string)[file](#file)[stdin](#stdin) ```sh influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000' ``` 1. In your terminal, enter the following command to create the sample data file: ``` echo 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000' > ./home.lp ``` 2. Enter the following CLI command to write the data from the sample file: ``` influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN \ --file ./home.lp ``` 1. In your terminal, enter the following command to create the sample data file: ``` echo 'home,room=Living\ Room temp=21.1,hum=35.9,co=0i 1641024000 home,room=Kitchen temp=21.0,hum=35.9,co=0i 1641024000 home,room=Living\ Room temp=21.4,hum=35.9,co=0i 1641027600 home,room=Kitchen temp=23.0,hum=36.2,co=0i 1641027600 home,room=Living\ Room temp=21.8,hum=36.0,co=0i 1641031200 home,room=Kitchen temp=22.7,hum=36.1,co=0i 1641031200 home,room=Living\ Room temp=22.2,hum=36.0,co=0i 1641034800 home,room=Kitchen temp=22.4,hum=36.0,co=0i 1641034800 home,room=Living\ Room temp=22.2,hum=35.9,co=0i 1641038400 home,room=Kitchen temp=22.5,hum=36.0,co=0i 1641038400 home,room=Living\ Room temp=22.4,hum=36.0,co=0i 1641042000 home,room=Kitchen temp=22.8,hum=36.5,co=1i 1641042000' > ./home.lp ``` 2. Enter the following CLI command to write the data from the sample file: ``` cat ./home.lp | influxdb3 write \ --database DATABASE_NAME \ --token AUTH_TOKEN ``` Replace the following: * `DATABASE_NAME`: the name of the database to write to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) #### Related * [influxdb3 write](/influxdb3/enterprise/reference/cli/influxdb3/write/) * [Write data to InfluxDB 3 Enterprise](/influxdb3/enterprise/get-started/write/) --- # Troubleshoot issues writing data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/troubleshoot/ Learn how to avoid unexpected results and recover from errors when writing to InfluxDB 3 Enterprise. * [Handle write responses](#handle-write-responses) * [Review HTTP status codes](#review-http-status-codes) * [Troubleshoot failures](#troubleshoot-failures) * [Troubleshoot rejected points](#troubleshoot-rejected-points) * [Troubleshoot write performance issues](#troubleshoot-write-performance-issues) * [Use debug logs for troubleshooting](#use-debug-logs-for-troubleshooting) ## Handle write responses InfluxDB 3 Enterprise does the following when you send a write request: 1. Validates the request. 2. If successful, attempts to ingest data from the request body; otherwise, responds with an [error status](#review-http-status-codes). 3. Ingests or rejects data in the batch and returns one of the following HTTP status codes: * `204 No Content`: All data in the batch is ingested. * `400 Bad Request`: Some or all of the data has been rejected. Data that has not been rejected is ingested and queryable. The response body contains error details about[rejected points](#troubleshoot-rejected-points), up to 100 points. Writes are synchronous–the response status indicates the final status of the write and all ingested data is queryable. To ensure that InfluxDB handles writes in the order you request them, wait for the response before you send the next request. ### Review HTTP status codes InfluxDB 3 Enterprise uses conventional HTTP status codes to indicate the success or failure of a request. The `message` property of the response body may contain additional details about the error. Write requests return the following status codes: | HTTP response code | Message | Description | |-----------------------------|------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `204 "Success"` | | If InfluxDB ingested the data | | `400 "Bad request"` |error details about rejected points, up to 100 points: `line` contains the first rejected line, `message` describes rejections| If some or all request data isn’t allowed (for example, if it is malformed or falls outside of the bucket’s retention period)–the response body indicates whether a partial write has occurred or if all data has been rejected | | `401 "Unauthorized"` | |If the `Authorization` header is missing or malformed or if the [token](/influxdb3/enterprise/admin/tokens/) doesn’t have permission to write to the database. See [write API examples](/influxdb3/enterprise/write-data/http-api/) using credentials.| | `404 "Not found"` | requested **resource type** (for example, “organization” or “database”), and **resource name** | If a requested resource (for example, organization or database) wasn’t found | |`500 "Internal server error"`| | Default status for an error | | `503` “Service unavailable” | | If the server is temporarily unavailable to accept writes. The `Retry-After` header describes when to try the write again. | If your data did not write to the database, see how to [troubleshoot rejected points](#troubleshoot-rejected-points). ## Troubleshoot failures If you notice data is missing in your database, do the following: * Check the `message` property in the response body for details about the error. * If the `message` describes a field error, [troubleshoot rejected points](#troubleshoot-rejected-points). * Verify all lines contain valid syntax ([line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/)). * Verify the timestamps in your data match the [precision parameter](/influxdb3/enterprise/reference/glossary/#precision) in your request. * Minimize payload size and network errors by [optimizing writes](/influxdb3/enterprise/write-data/best-practices/optimize-writes/). ## Troubleshoot rejected points InfluxDB rejects points that don’t match the schema of existing data. Check for [field data type](/influxdb3/enterprise/reference/syntax/line-protocol/#data-types-and-format)differences between the rejected data point and points within the same database–for example, did you attempt to write `string` data to an `int` field? ## Troubleshoot write performance issues If you experience slow write performance or timeouts during high-volume ingestion, consider the following: ### Memory configuration InfluxDB 3 Enterprise uses memory for both query processing and internal data operations, including converting data to Parquet format during persistence. For write-heavy workloads, insufficient memory allocation can cause performance issues. **Symptoms of memory-related write issues:** * Slow write performance during data persistence (typically every 10 minutes) * Increased response times during high-volume ingestion * Memory-related errors in server logs **Solutions:** * Increase the [`exec-mem-pool-size`](/influxdb3/enterprise/reference/config-options/#exec-mem-pool-size)configuration to allocate more memory for data operations. For write-heavy workloads, consider setting this to 30-40% of available memory. * Monitor memory usage during peak write periods to identify bottlenecks. * Adjust the [`gen1-duration`](/influxdb3/enterprise/reference/config-options/#gen1-duration)to control how frequently data is persisted to Parquet format. ### Example configuration for write-heavy workloads ```bash influxdb3 serve \ --exec-mem-pool-size PERCENTAGE \ --gen1-duration 15m \ # ... other options ``` Replace `PERCENTAGE` with the percentage of available memory to allocate (for example, `35%` for write-heavy workloads). ### Use debug logs for troubleshooting For deeper investigation of write issues, enable debug logging for specific components. Debug logs provide detailed information about write buffer operations and WAL activity. To enable debug logs for write operations, restart InfluxDB 3 Enterprise with targeted log filters: ```sh influxdb3 serve --log-filter info,influxdb3_write_buffer=debug ``` ```sh influxdb3 serve --log-filter info,influxdb3_wal=debug ``` For more information about log levels and targeted filtering, see[log-filter configuration](/influxdb3/enterprise/reference/config-options/#log-filter). #### Related * [Best practices for writing data](/influxdb3/enterprise/write-data/best-practices/) * [InfluxDB 3 Enterprise Internals](/influxdb3/enterprise/reference/internals/durability/) [write](/influxdb3/enterprise/tags/write/)[line protocol](/influxdb3/enterprise/tags/line-protocol/)[errors](/influxdb3/enterprise/tags/errors/) | HTTP response code | Message | Description | | --- | --- | --- | | HTTP response code | Message | Description | | 204 "Success" | | If InfluxDB ingested the data | | 400 "Bad request" | error details about rejected points, up to 100 points: line contains the first rejected line, message describes rejections | If some or all request data isn’t allowed (for example, if it is malformed or falls outside of the bucket’s retention period)–the response body indicates whether a partial write has occurred or if all data has been rejected | | 401 "Unauthorized" | | If the Authorization header is missing or malformed or if the token doesn’t have permission to write to the database. See write API examples using credentials. | | 404 "Not found" | requested resource type (for example, “organization” or “database”), and resource name | If a requested resource (for example, organization or database) wasn’t found | | 500 "Internal server error" | | Default status for an error | | 503 “Service unavailable” | | If the server is temporarily unavailable to accept writes. The Retry-After header describes when to try the write again. | --- # Use Telegraf to write data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/use-telegraf/ [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/) is a data collection agent for collecting and reporting metrics. Its vast library of input plugins and “plug-and-play” architecture lets you quickly and easily collect metrics from many different sources. For a list of available plugins, see [Telegraf plugins](/telegraf/v1/plugins/). #### Requirements * **Telegraf 1.38 or greater** to use the `influxdb_v3` output plugin. Earlier versions of Telegraf (1.9.2 or greater) can use the[`influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/) or[`influxdb`](/telegraf/v1/output-plugins/influxdb/) (v1) output plugins to write to InfluxDB 3 Enterprise[compatibility APIs](/influxdb3/enterprise/write-data/http-api/compatibility-apis/).*For information about installing Telegraf, see the[Telegraf Installation instructions](/telegraf/v1/install/).* ## Basic Telegraf usage Telegraf is a plugin-based agent with plugins that are enabled and configured in your Telegraf configuration file (`telegraf.conf`). Each Telegraf configuration must **have at least one input plugin and one output plugin**. Telegraf input plugins retrieve metrics from different sources. Telegraf output plugins write those metrics to a destination. Use the [`outputs.influxdb_v3`](/telegraf/v1/plugins/#output-influxdb_v3) plugin to connect to the InfluxDB 3 Enterprise native write API and write metrics collected by Telegraf to InfluxDB 3 Enterprise. ```toml # ... [[outputs.influxdb_v3]] urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" # ... ``` Replace the following: * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/).*Store this in a secret store or environment variable to avoid exposing the raw token string.* * `DATABASE_NAME`: the name of the database to write data to *See how to [Configure Telegraf to write to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/use-telegraf/configure/).* ## Use Telegraf with InfluxDB ### [Configure Telegraf to write to InfluxDB 3 Enterprise](/influxdb3/enterprise/write-data/use-telegraf/configure/) Update existing or create new Telegraf configurations to use the `influxdb_v3` output plugin to write to InfluxDB 3 Enterprise. Start Telegraf using the custom configuration. ### [Use Telegraf to dual write to InfluxDB](/influxdb3/enterprise/write-data/use-telegraf/dual-write/) Configure Telegraf to write data to multiple InfluxDB instances or clusters simultaneously. ### [Use Telegraf to write CSV data](/influxdb3/enterprise/write-data/use-telegraf/csv/) Use the Telegraf `file` input plugin to read and parse CSV data into [line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/) and write it to InfluxDB 3 Enterprise. ![InfluxDB University](/svgs/influxdbu-full-white.svg) #### Data Collection with Telegraf Learn how to use Telegraf to make data time series data collection easy in this **free** InfluxDB University course. [Take the course](https://university.influxdata.com/courses/data-collection-with-telegraf-tutorial/) --- # Configure Telegraf to write to InfluxDB 3 Enterprise Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/use-telegraf/configure/ Use the Telegraf [`influxdb_v3` output plugin](/telegraf/v1/output-plugins/influxdb_v3/)to collect and write metrics to InfluxDB 3 Enterprise. This plugin uses the InfluxDB 3 Enterprise native HTTP API[`/api/v3/write_lp` endpoint](/influxdb3/enterprise/write-data/http-api/v3-write-lp/)and requires **Telegraf 1.38 or greater**. Learn how to enable and configure the `influxdb_v3` output plugin to write data to InfluxDB 3 Enterprise. If you use an earlier version of Telegraf or bring an existing Telegraf configuration from InfluxDB v1 or v2, you can use the[`influxdb_v2`](#use-the-influxdb-v2-output-plugin) or[`influxdb`](#use-the-influxdb-v1-output-plugin) (v1) output plugins to write to InfluxDB 3 Enterprise[compatibility APIs](/influxdb3/enterprise/write-data/http-api/compatibility-apis/). > [!Note] > *View the [requirements](/influxdb3/enterprise/write-data/use-telegraf#requirements)for using Telegraf with InfluxDB 3 Enterprise.* * [Configure Telegraf input and output plugins](#configure-telegraf-input-and-output-plugins) * [Add Telegraf plugins](#add-telegraf-plugins) * [Collect Prometheus metrics](#collect-prometheus-metrics) * [Enable and configure the InfluxDB v3 output plugin](#enable-and-configure-the-influxdb-v3-output-plugin) * [urls](#urls) * [token](#token) * [database](#database) * [Additional plugin options](#additional-plugin-options) * [Use the InfluxDB v2 output plugin](#use-the-influxdb-v2-output-plugin) * [Use the InfluxDB v1 output plugin](#use-the-influxdb-v1-output-plugin) * [Other Telegraf configuration options](#other-telegraf-configuration-options) * [Start Telegraf](#start-telegraf) ## Configure Telegraf input and output plugins Configure Telegraf input and output plugins in the Telegraf configuration file (typically named `telegraf.conf`). Input plugins collect metrics. Output plugins define destinations where metrics are sent. This guide assumes you have already [installed InfluxDB 3 Enterprise](/influxdb3/enterprise/install/)and have been through the [getting started guide](/influxdb3/enterprise/get-started/). ### Add Telegraf plugins To add any of the available [Telegraf plugins](/telegraf/v1/plugins/), follow the steps below. 1. Find the plugin you want to enable from the complete list of available[Telegraf plugins](/telegraf/v1/plugins/). 2. Click **View** to the right of the plugin name to open the plugin page on GitHub. For example, view the [MQTT plugin GitHub page](https://github.com/influxdata/telegraf/blob/master/plugins/inputs/mqtt_consumer/README.md). 3. Copy and paste the example configuration into your Telegraf configuration file (typically named `telegraf.conf`). ### Collect Prometheus metrics If you collect Prometheus metrics with the[`prometheus` input plugin](/telegraf/v1/input-plugins/prometheus/), set`metric_version = 1`. If you accept pushed Prometheus data with a listener plugin, such as[`http_listener_v2`](/telegraf/v1/input-plugins/http_listener_v2/), set`prometheus_metric_version = 1` for the `prometheus` and`prometheusremotewrite` data formats, or `openmetrics_metric_version = 1`for the `openmetrics` data format. These data formats default to version `2`. ```toml [[inputs.prometheus]] urls = ["http://localhost:9100/metrics"] metric_version = 1 [[inputs.http_listener_v2]] service_address = ":1234" paths = ["/receive"] data_format = "prometheusremotewrite" prometheus_metric_version = 1 ``` Metric version `2` writes every Prometheus metric name as a separate field in a single table. In InfluxDB 3 Enterprise, that produces a[wide](/influxdb3/enterprise/write-data/best-practices/schema-design/#avoid-wide-schemas),[sparse](/influxdb3/enterprise/write-data/best-practices/schema-design/#avoid-sparse-schemas)table that slows persistence and queries. Metric version `1` writes each Prometheus metric to its own table. For details, see[Choose a metric version](/telegraf/v1/examples/scrape-prometheus/#choose-a-metric-version)in the Telegraf documentation. ### Enable and configure the InfluxDB v3 output plugin To send data to InfluxDB 3 Enterprise, enable the[`influxdb_v3` output plugin](/telegraf/v1/output-plugins/influxdb_v3/)in the `telegraf.conf`. ```toml [[outputs.influxdb_v3]] urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/).*Store this in a secret store or environment variable to avoid exposing the raw token string.* The InfluxDB v3 output plugin configuration contains the following options: #### urls An array of URL strings. To write to InfluxDB 3 Enterprise, include your InfluxDB 3 Enterprise URL: ```toml ["http://localhost:8181"] ``` If you specify multiple URLs, Telegraf randomly selects one of them for each write interval and fails over to another if the write doesn’t succeed. #### token Your InfluxDB 3 Enterprise authorization token. > [!Tip] > ##### Store your authorization token as an environment variable > > To prevent a plain text token in your Telegraf configuration file, we > recommend that you store the token as an environment variable and then > reference the environment variable in your configuration file using string > interpolation. For example: > > ``` > [[outputs.influxdb_v3]] > urls = ["http://localhost:8181"] > token = "${INFLUX_TOKEN}" > # ... > ``` #### database The name of the InfluxDB 3 Enterprise database to write data to. #### Additional plugin options The plugin provides additional options for controlling write behavior, including: * **`database_tag`**: the metric tag to use to determine the destination database, overriding `database`. * **`sync`**: set to `false` to acknowledge writes before WAL persistence completes, which reduces write latency but increases the risk of data loss. Default is `true`.*See [Use no\_sync for immediate write responses](/influxdb3/enterprise/write-data/http-api/v3-write-lp/#use-no_sync-for-immediate-write-responses).* * **`content_encoding`**: the plugin compresses write request bodies with gzip by default. For all plugin options, see the[`influxdb_v3` output plugin](/telegraf/v1/output-plugins/influxdb_v3/) reference. ### Use the InfluxDB v2 output plugin If you bring an existing InfluxDB v2 write workload or use a Telegraf version earlier than 1.38, use the[`influxdb_v2` output plugin](/telegraf/v1/output-plugins/influxdb_v2/)to write to InfluxDB 3 Enterprise through the InfluxDB[v2 compatibility API](/influxdb3/enterprise/write-data/http-api/compatibility-apis/#influxdb-v2-compatibility). ```toml [[outputs.influxdb_v2]] urls = ["http://localhost:8181"] token = "AUTH_TOKEN" organization = "" bucket = "DATABASE_NAME" ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) For InfluxDB 3 Enterprise, set the following v2-specific options: * **`organization`**: set to an empty string (`""`). * **`bucket`**: the name of the database to write data to. > [!Note] > An InfluxDB v2 ***bucket*** is synonymous with an InfluxDB 3 Enterprise ***database***. ### Use the InfluxDB v1 output plugin If you bring an existing InfluxDB v1 write workload, use the[`influxdb` output plugin](/telegraf/v1/output-plugins/influxdb/)to write to InfluxDB 3 Enterprise through the InfluxDB[v1 compatibility API](/influxdb3/enterprise/write-data/http-api/compatibility-apis/#influxdb-v1-compatibility). ```toml [[outputs.influxdb]] urls = ["http://localhost:8181"] database = "DATABASE_NAME" skip_database_creation = true username = "ignored" password = "AUTH_TOKEN" ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/) For InfluxDB 3 Enterprise, set the following v1-specific options: * **`skip_database_creation`**: set to `true`. InfluxDB 3 Enterprise doesn’t support creating databases through the v1 API. * **`password`**: your authorization token. The v1 compatibility API authenticates with the token passed as the password credential and ignores `username`. ### Other Telegraf configuration options For more plugin configuration options, see the[`influxdb_v3`](/telegraf/v1/output-plugins/influxdb_v3/),[`influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/), and[`influxdb` (v1)](/telegraf/v1/output-plugins/influxdb/) output plugin references. ## Start Telegraf Start the Telegraf service using the `--config` flag to specify the location of your `telegraf.conf`. ```sh telegraf --config /path/to/custom/telegraf.conf ``` #### Related * [Telegraf plugins](/telegraf/v1/plugins/) * [Telegraf InfluxDB v3 output plugin](/telegraf/v1/output-plugins/influxdb_v3/) [telegraf](/influxdb3/enterprise/tags/telegraf/) --- # Use Telegraf to write CSV data Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/use-telegraf/csv/ Use the Telegraf `file` input plugin to read and parse CSV data into[line protocol](/influxdb3/enterprise/reference/syntax/line-protocol/)and write it to InfluxDB 3 Enterprise.[Telegraf](/telegraf/v1/) is a plugin-based agent that collects metrics from different sources and writes them to specified destinations. * [Configure Telegraf to read CSV files](#configure-telegraf-to-read-csv-files) * [Configure Telegraf to write to InfluxDB](#configure-telegraf-to-write-to-influxdb) * [Other Telegraf configuration options](#other-telegraf-configuration-options) ## Configure Telegraf to read CSV files 1. Add and enable the [`inputs.file` plugin](/telegraf/v1/plugins/#input-file)in your Telegraf configuration file. 2. Use the `files` option to specify the list of CSV files to read. CSV files must be accessible by the Telegraf agent. 3. Set the `data_format` option to `csv`. 4. Define all other `csv_` configuration options specific to the CSV data you want to write to InfluxDB 3 Enterprise.*For detailed information about each of the CSV format configuration options, see [CSV input data format](/telegraf/v1/data_formats/input/csv/).* ```toml [[inputs.file]] files = ["/path/to/example.csv"] data_format = "csv" csv_header_row_count = 0 csv_column_names = [] csv_column_types = [] csv_skip_rows = 0 csv_metadata_rows = 0 csv_metadata_separators = [":", "="] csv_metadata_trim_set = "" csv_skip_columns = 0 csv_delimiter = "," csv_comment = "" csv_trim_space = false csv_tag_columns = [] csv_measurement_column = "" csv_timestamp_column = "" csv_timestamp_format = "" csv_timezone = "" csv_skip_values = [] csv_skip_errors = false csv_reset_mode = "none" ``` ## Configure Telegraf to write to InfluxDB To send data to InfluxDB 3 Enterprise, enable and configure the[`influxdb_v3` output plugin](/influxdb3/enterprise/write-data/use-telegraf/configure/#enable-and-configure-the-influxdb-v3-output-plugin)in your `telegraf.conf`. ```toml [[inputs.file]] files = ["/path/to/example.csv"] data_format = "csv" csv_header_row_count = 0 csv_column_names = [] csv_column_types = [] csv_skip_rows = 0 csv_metadata_rows = 0 csv_metadata_separators = [":", "="] csv_metadata_trim_set = "" csv_skip_columns = 0 csv_delimiter = "," csv_comment = "" csv_trim_space = false csv_tag_columns = [] csv_measurement_column = "" csv_timestamp_column = "" csv_timestamp_format = "" csv_timezone = "" csv_skip_values = [] csv_skip_errors = false csv_reset_mode = "none" [[outputs.influxdb_v3]] urls = ["http://localhost:8181"] token = "AUTH_TOKEN" database = "DATABASE_NAME" ``` Replace the following: * `DATABASE_NAME`: the name of the database to write data to * `AUTH_TOKEN`: your InfluxDB 3 Enterprise [token](/influxdb3/enterprise/admin/tokens/).*Store this in a secret store or environment variable to avoid exposing the raw token string.* ##### Store your authorization token as an environment variable ##### Avoid storing a plain text token in your Telegraf configuration file. Store the token as an environment variable and then reference the environment variable in your configuration file using string interpolation. For example: ``` [[outputs.influxdb_v3]] urls = ["http://localhost:8181"] token = "${INFLUX_TOKEN}" # ... ``` **Restart the Telegraf agent** to apply the configuration change and write the CSV data to InfluxDB 3 Enterprise. #### Other Telegraf configuration options The preceding examples describe Telegraf configurations necessary for writing to InfluxDB 3 Enterprise. The `influxdb_v3` output plugin provides several other configuration options. For more information, see the[`influxdb_v3` plugin options](/telegraf/v1/output-plugins/influxdb_v3/). #### Related * [CSV input data format](/telegraf/v1/data_formats/input/csv/) * [Use Telegraf to write data](/influxdb3/enterprise/write-data/use-telegraf/) --- # Use Telegraf to dual write to InfluxDB Source: https://docs.influxdata.com/influxdb3/enterprise/write-data/use-telegraf/dual-write/ Use Telegraf to write your data simultaneously to multiple InfluxDB instances or clusters. This method, known as “dual writing,” is useful for backing up data to a separate instance or for migrating from other versions of InfluxDB to InfluxDB 3 Enterprise. The following example configures Telegraf for dual writing to InfluxDB 3 Enterprise and an InfluxDB v2 OSS instance. Specifically, it uses the following: * The [InfluxDB v3 output plugin](/telegraf/v1/output-plugins/influxdb_v3/)pointing to InfluxDB 3 Enterprise and the[InfluxDB v2 output plugin](/telegraf/v1/output-plugins/influxdb_v2/)pointing to an InfluxDB v2 OSS instance. * Two different tokens: one for InfluxDB v2 OSS and one for InfluxDB 3 Enterprise. Configure both tokens as environment variables and use string interpolation in your Telegraf configuration file to reference each environment variable. ## Sample configuration ```toml # Include any other input, processor, or aggregator plugins that you want to # include in your configuration. # Send data to InfluxDB 3 Enterprise [[outputs.influxdb_v3]] ## The InfluxDB 3 Enterprise URL urls = ["http://localhost:8181"] ## InfluxDB 3 Enterprise authorization token token = "${INFLUX_TOKEN}" ## Destination database to write into database = "DATABASE_NAME" # Send data to InfluxDB v2 OSS [[outputs.influxdb_v2]] ## The InfluxDB v2 OSS URL urls = ["http://localhost:8086"] ## OSS token for authentication token = "${INFLUX_TOKEN_OSS}" ## Organization is the name of the organization you want to write to. organization = "ORG_NAME_OSS" ## Destination bucket to write to bucket = "BUCKET_NAME_OSS" ``` Telegraf lets you dual write data to any version of InfluxDB using the[`influxdb` (v1)](/telegraf/v1/output-plugins/influxdb/),[`influxdb_v2`](/telegraf/v1/output-plugins/influxdb_v2/), and[`influxdb_v3`](/telegraf/v1/output-plugins/influxdb_v3/) output plugins. A single Telegraf agent sends identical data sets to all target outputs. You cannot filter data based on the output. > [!Note] > InfluxDB v1 does *not* support the unsigned integer data type. > You can only write unsigned integer field values to InfluxDB v2- and 3-based > products.