Scale your InfluxDB cluster

InfluxDB Clustered lets you scale individual components of your cluster both vertically and horizontally to match your specific workload. Use the AppInstance resource defined in your influxdb.yml to manage resources available to each component.

• Scaling strategies
  • Vertical scaling
  • Horizontal scaling
• Scale components in your cluster
  • Horizontally scale a component
  • Vertically scale a component
  • Apply your changes
• Scale your cluster as a whole
• Recommended scaling strategies per component
  • Ingester
  • Querier
  • Router
  • Compactor
  • Garbage collector
  • Catalog
  • Object store

Scaling strategies

The following scaling strategies can be applied to components in your InfluxDB cluster.

Vertical scaling

Vertical scaling (also known as “scaling up”) involves increasing the resources (such as RAM or CPU) available to a process or system. Vertical scaling is typically used to handle resource-intensive tasks that require more processing power.

Horizontal scaling

Horizontal scaling (also known as “scaling out”) involves increasing the number of nodes or processes available to perform a given task. Horizontal scaling is typically used to increase the amount of workload or throughput a system can manage, but also provides additional redundancy and failover.

Scale components in your cluster

The following components of your InfluxDB cluster are scaled by modifying properties in your AppInstance resource:

• Ingester
• Querier
• Compactor
• Router
• Garbage collector

apiVersion: kubecfg.dev/v1alpha1
kind: AppInstance
# ...
spec:
  package:
    spec:
      # The following settings tune the various pods for their cpu/memory/replicas
      # based on workload needs. Only uncomment the specific resources you want
      # to change. Anything left commented will use the package default.
      resources:
        ingester:
          requests:
            cpu: 
INGESTER_CPU_MIN
            memory: 
INGESTER_MEMORY_MIN
            replicas: 
INGESTER_REPLICAS # Default is 3
          limits:
            cpu: 
INGESTER_CPU_MAX
            memory: 
INGESTER_MEMORY_MAX
        compactor:
          requests:
            cpu: 
COMPACTOR_CPU_MIN
            memory: 
COMPACTOR_MEMORY_MIN
            replicas: 
COMPACTOR_REPLICAS # Default is 1
          limits:
            cpu: 
COMPACTOR_CPU_MAX
            memory: 
COMPACTOR_MEMORY_MAX
        querier:
          requests:
            cpu: 
QUERIER_CPU_MIN
            memory: 
QUERIER_MEMORY_MIN
            replicas: 
QUERIER_REPLICAS # Default is 1          
          limits:
            cpu: 
QUERIER_CPU_MAX
            memory: 
QUERIER_MEMORY_MAX
        router:
          requests:
            cpu: 
ROUTER_CPU_MIN
            memory: 
ROUTER_MEMORY_MIN
            replicas: 
ROUTER_REPLICAS # Default is 1
          limits:
            cpu: 
ROUTER_CPU_MAX
            memory: 
ROUTER_MEMORY_MAX
        garbage-collector:
          requests:
            cpu: 
GC_CPU_MIN
            memory: 
GC_MEMORY_MIN
          limits:
            cpu: 
GC_CPU_MAX
            memory: 
GC_MEMORY_MAX

# ...
  resources:
    ingester:
      requests:
        cpu: 
INGESTER_CPU_MIN
        memory: 
INGESTER_MEMORY_MIN
        replicas: 
INGESTER_REPLICAS # Default is 3
      limits:
        cpu: 
INGESTER_CPU_MAX
        memory: 
INGESTER_MEMORY_MAX
    compactor:
      requests:
        cpu: 
COMPACTOR_CPU_MIN
        memory: 
COMPACTOR_MEMORY_MIN
        replicas: 
COMPACTOR_REPLICAS # Default is 1
      limits:
        cpu: 
COMPACTOR_CPU_MAX
        memory: 
COMPACTOR_MEMORY_MAX
    querier:
      requests:
        cpu: 
QUERIER_CPU_MIN
        memory: 
QUERIER_MEMORY_MIN
        replicas: 
QUERIER_REPLICAS # Default is 1          
      limits:
        cpu: 
QUERIER_CPU_MAX
        memory: 
QUERIER_MEMORY_MAX
    router:
      requests:
        cpu: 
ROUTER_CPU_MIN
        memory: 
ROUTER_MEMORY_MIN
        replicas: 
ROUTER_REPLICAS # Default is 1
      limits:
        cpu: 
ROUTER_CPU_MAX
        memory: 
ROUTER_MEMORY_MAX
    garbage-collector:
      requests:
        cpu: 
GC_CPU_MIN
        memory: 
GC_MEMORY_MIN
      limits:
        cpu: 
GC_CPU_MAX
        memory: 
GC_MEMORY_MAX

Related Kubernetes documentation

• CPU resource units
• Memory resource units

Horizontally scale a component

To horizontally scale a component in your InfluxDB cluster, increase or decrease the number of replicas for the component and apply the change.

For example–to horizontally scale your Ingester:

apiVersion: kubecfg.dev/v1alpha1
kind: AppInstance
# ...
spec:
  package:
    spec:
      resources:
        ingester:
          requests:
            # ...
            replicas: 6

# ...
resources:
  ingester:
    requests:
      # ...
      replicas: 6

Vertically scale a component

To vertically scale a component in your InfluxDB cluster, increase or decrease the CPU and memory resource units to assign to component pods and apply the change.

apiVersion: kubecfg.dev/v1alpha1
kind: AppInstance
# ...
spec:
  package:
    spec:
      resources:
        ingester:
          requests:
            cpu: "500m"
            memory: "512MiB"
            # ...
          limits:
            cpu: "1000m"
            memory: "1024MiB"

# ...
resources:
  ingester:
    requests:
      cpu: "500m"
      memory: "512MiB"
      # ...
    limits:
      cpu: "1000m"
      memory: "1024MiB"

Apply your changes

After modifying the AppInstance resource, use kubectl apply to apply the configuration changes to your cluster and scale the updated components.

kubectl apply \
  --filename myinfluxdb.yml \
  --namespace influxdb

helm upgrade \
  influxdata/influxdb3-clustered \
  -f ./values.yml \
  --namespace influxdb

Scale your cluster as a whole

Scaling your entire InfluxDB Cluster is done by scaling your Kubernetes cluster and is managed outside of InfluxDB. The process of scaling your entire Kubernetes cluster depends on your underlying Kubernetes provider. You can also use Kubernetes autoscaling to automatically scale your cluster as needed.

Recommended scaling strategies per component

• Router
• Ingester
• Querier
• Compactor
• Garbage collector
• Catalog
• Object store

Router

The Router can be scaled both vertically and horizontally. Horizontal scaling increases write throughput and is typically the most effective scaling strategy for the Router. Vertical scaling (specifically increased CPU) improves the Router’s ability to parse incoming line protocol with lower latency.

Ingester

The Ingester can be scaled both vertically and horizontally. Vertical scaling increases write throughput and is typically the most effective scaling strategy for the Ingester.

Ingester storage volume

Ingesters use an attached storage volume to store the Write-Ahead Log (WAL). With more storage available, Ingesters can keep bigger WAL buffers, which improves query performance and reduces pressure on the Compactor. Storage speed also helps with query performance.

Configure the storage volume attached to Ingester pods in the spec.package.spec.ingesterStorage property of your AppInstance resource or, if using Helm, the ingesterStorage property of your values.yaml.

apiVersion: kubecfg.dev/v1alpha1
kind: AppInstance
# ...
spec:
  package:
    spec:
      # ...
      ingesterStorage:
        # (Optional) Set the storage class. This will differ based on the K8s
        #environment and desired storage characteristics.
        # If not set, the default storage class is used.
        storageClassName: 
STORAGE_CLASS
        # Set the storage size (minimum 2Gi recommended)
        storage: 
STORAGE_SIZE

# ...
ingesterStorage:
  # (Optional) Set the storage class. This will differ based on the K8s
  #environment and desired storage characteristics.
  # If not set, the default storage class is used.
  storageClassName: 
STORAGE_CLASS
  # Set the storage size (minimum 2Gi recommended)
  storage: 
STORAGE_SIZE

Querier

The Querier can be scaled both vertically and horizontally. Horizontal scaling increases query throughput to handle more concurrent queries. Vertical scaling improves the Querier’s ability to process computationally intensive queries.

Compactor

The Compactor can be scaled both vertically and horizontally. Because compaction is a compute-heavy process, vertical scaling (especially increasing the available CPU) is the most effective scaling strategy for the Compactor. Horizontal scaling increases compaction throughput, but not as efficiently as vertical scaling.

Garbage collector

The Garbage collector can be scaled vertically. It is a light-weight process that typically doesn’t require many system resources, but if you begin to see high resource consumption on the garbage collector, you can scale it vertically to address the added workload.

Catalog

Scaling strategies available for the Catalog depend on the PostgreSQL-compatible database used to run the catalog. All support vertical scaling. Most support horizontal scaling for redundancy and failover.

Object store

Scaling strategies available for the Object store depend on the underlying object storage services used to run the object store. Most support horizontal scaling for redundancy, failover, and increased capacity.