Documentation

InfluxDB Cloud Dedicated data durability

InfluxDB Cloud Dedicated writes data to multiple Write-Ahead-Log (WAL) files on local storage and retains WALs until the data is persisted to Parquet files in object storage. Parquet data files in object storage are redundantly stored on multiple devices across a minimum of three availability zones in a cloud region.

Data storage

In InfluxDB Cloud Dedicated, all measurements are stored in Apache Parquet files that represent a point-in-time snapshot of the data. The Parquet files are immutable and are never replaced nor modified. Parquet files are stored in object storage.

The InfluxDB catalog is a relational, PostreSQL-compatible database that contains references to all Parquet files in object storage and is used as an index to find the appropriate Parquet files for a particular set of data.

Data deletion

When data is deleted or when the retention period is reached for data within a database, the associated Parquet files are marked as deleted in the catalog, but the actual Parquet files are not removed from object storage. All queries filter out data that has been marked as deleted. Parquet files remain in object storage for approximately 100 days after the youngest data in the Parquet file ages out of retention.

Data ingest

When data is written to InfluxDB Cloud Dedicated, the data is first written to a Write-Ahead-Log (WAL) on locally-attached storage on the ingester node before the write request is acknowledged. After acknowledging the write request, the ingester holds the data in memory temporarily and then writes the contents of the WAL to Parquet files in object storage and updates the InfluxDB catalog to reference the newly created Parquet files. If an ingester is gracefully shut down (for example, during a new software deployment), it flushes the contents of the WAL to the Parquet files before shutting down.

Backups

InfluxDB Cloud Dedicated implements the following data backup strategies:

  • Backup of WAL file: The WAL file is written on locally-attached storage. If an ingester process fails, the new ingester simply reads the WAL file on startup and continues normal operation. WAL files are maintained until their contents have been written to the Parquet files in object storage. For added protection, ingesters can be configured for write replication, where each measurement is written to two different WAL files before acknowledging the write.

  • Backup of Parquet files: Parquet files are stored in object storage where they are redundantly stored on multiple devices across a minimum of three availability zones in a cloud region. Parquet files associated with each database are kept in object storage for the duration of database retention period plus an additional time period (approximately 100 days).

  • Backup of catalog: InfluxData keeps a transaction log of all recent updates to the InfluxDB catalog and generates a daily backup of the catalog. Backups are preserved for at least 100 days in object storage across a minimum of three availability zones.

Recovery

InfluxData can perform the following recovery operations:

  • Recovery after ingester failure: If an ingester fails, a new ingester is started up and reads from the WAL file for the recently ingested data.

  • Recovery of Parquet files: InfluxDB Cloud Dedicated uses the provided object storage data durability to recover Parquet files.

  • Recovery of the catalog: InfluxData can restore the InfluxDB catalog to the most recent daily backup of the catalog and then reapply any transactions that occurred since the interruption.


Was this page helpful?

Thank you for your feedback!


The future of Flux

Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

Read more

InfluxDB v3 enhancements and InfluxDB Clustered is now generally available

New capabilities, including faster query performance and management tooling advance the InfluxDB v3 product line. InfluxDB Clustered is now generally available.

InfluxDB v3 performance and features

The InfluxDB v3 product line has seen significant enhancements in query performance and has made new management tooling available. These enhancements include an operational dashboard to monitor the health of your InfluxDB cluster, single sign-on (SSO) support in InfluxDB Cloud Dedicated, and new management APIs for tokens and databases.

Learn about the new v3 enhancements


InfluxDB Clustered general availability

InfluxDB Clustered is now generally available and gives you the power of InfluxDB v3 in your self-managed stack.

Talk to us about InfluxDB Clustered