Manage data partitioning

When writing data to InfluxDB Clustered, the InfluxDB 3 storage engine stores data in Apache Parquet format in the Object store. Each Parquet file represents a partition–a logical grouping of data. By default, InfluxDB partitions each table by day. If this default strategy yields unsatisfactory performance for single-series queries, you can define a custom partitioning strategy by specifying tag values and different time intervals to optimize query performance for your specific schema and workload.

• Advantages
• Disadvantages
• Limitations
• Plan for custom partitioning
• How partitioning works
  • Partition templates
  • Partition keys
• Partitions in the query life cycle
• Partition guides
  • Define custom partitions
  • Use partition templates
  • Partitioning best practices
  • View partition information

Advantages

The primary advantage of custom partitioning is that it lets you customize your storage structure to improve query performance specific to your schema and workload.

• Optimized storage for improved performance on specific types of queries. For example, if queries often select data with a specific tag value, you can partition by that tag to improve the performance of those queries.
• Optimized storage for specific types of data. For example, if the data you store is sparse and the time ranges you query are often much larger than a day, you could partition your data by month instead of by day.

Disadvantages

Using custom partitioning may increase the load on other parts of the InfluxDB 3 storage engine, but you can scale each part individually to address the added load.

• Increased load on the Ingester as it groups data into smaller partitions and files.
• Increased load on the Catalog as more references to partition Parquet file locations are stored and queried.
• Increased load on the Compactor as it needs to compact more partition Parquet files.
• Increased costs associated with Object storage as more partition Parquet files are created and stored.
• Increased latency. The amount of time for InfluxDB to process a query and return results increases linearly, although slightly, with the total partition count for a table.
• Risk of decreased performance for queries that don’t use tags in the WHERE clause. These queries might read many partitions and smaller files, which can degrade performance.

Limitations

Custom partitioning has the following limitations:

• Define database and table partitions only during creation; you can’t update the partition strategy afterward.
• Include a time part in a partition template.
• You can partition by up to eight dimensions (seven tags and a time interval).

Plan for custom partitioning

After you have considered the advantages, disadvantages, and limitations of custom partitioning, use the guides in this section to:

1. Learn how partitioning works
2. Follow best practices for defining partitions and managing partition growth
3. Define custom partitions for your data
4. Take steps to limit the number of partition files

How partitioning works

Partition templates

A partition template defines the pattern used for partition keys and determines the time interval that InfluxDB partitions data by. Partition templates use tag values and Rust strftime date and time formatting syntax.

For more detailed information, see Partition templates.

Partition keys

A partition key uniquely identifies a partition. A partition template defines the partition key format. Partition keys are composed of up to 8 dimensions (1 time part and up to 7 tag or tag bucket parts). A partition key uses the partition key separator (|) to delimit parts.

The default format for partition keys is %Y-%m-%d (for example, 2024-01-01), which creates 1 partition for each day.

production,line=A,station=cnc temp=81.2,qty=35i 1704063600000000000
production,line=A,station=wld temp=92.8,qty=35i 1704063600000000000
production,line=B,station=cnc temp=101.1,qty=43i 1704063600000000000
production,line=B,station=wld temp=102.4,qty=43i 1704063600000000000
production,line=A,station=cnc temp=81.9,qty=36i 1704067200000000000
production,line=A,station=wld temp=110.0,qty=22i 1704067200000000000
production,line=B,station=cnc temp=101.8,qty=44i 1704067200000000000
production,line=B,station=wld temp=105.7,qty=44i 1704067200000000000
production,line=A,station=cnc temp=82.2,qty=35i 1704070800000000000
production,line=A,station=wld temp=92.1,qty=30i 1704070800000000000
production,line=B,station=cnc temp=102.4,qty=43i 1704070800000000000
production,line=B,station=wld temp=106.5,qty=43i 1704070800000000000

Partitions in the query life cycle

When querying data:

1. The Catalog provides the v3 query engine (Querier) with the locations of partitions that contain the queried time series data.
2. The query engine reads all rows in the returned partitions to identify what rows match the logic in the query and should be included in the query result.

The faster the query engine can identify what partitions to read and then read the data in those partitions, the more performant queries are.

For more information about the query lifecycle, see InfluxDB 3 query life cycle.

Query example

Consider the following query that selects everything in the production table where the line tag is A and the station tag is cnc:

SELECT *
FROM production
WHERE
  time >= now() - INTERVAL '1 week'
  AND line = 'A'
  AND station = 'cnc'

Using the default partitioning strategy (by day), the query engine reads eight separate partitions (one partition for today and one for each of the last seven days):

• 2025-01-14
• 2025-01-13
• 2025-01-12
• 2025-01-11
• 2025-01-10
• 2025-01-09
• 2025-01-08
• 2025-01-07

The query engine must scan all rows in the partitions to identify rows where line is A and station is cnc. This process takes valuable time and results in less performant queries.

However, including tags in your partitioning strategy allows the query engine to identify partitions containing only the required tag values. This avoids scanning rows for tag values.

For example, if you partition data by line, station, and day, although the number of files increases, the query engine can quickly identify and read only those with data relevant to the query:

Partition guides

SELECT * FROM system.partitions WHERE table_name = 'example-table'