Documentation

Replicate data from InfluxDB OSS

Use InfluxDB replication streams (InfluxDB Edge Data Replication) to replicate the incoming data of select buckets to one or more buckets on a remote InfluxDB OSS, InfluxDB Cloud, or InfluxDB Enterprise instance.

Replicate data from InfluxDB OSS to InfluxDB Cloud, InfluxDB OSS, or InfluxDB Enterprise.

Configure a replication stream

Use the influx CLI or the InfluxDB Cloud API to configure a replication stream.

To replicate data to InfluxDB OSS or InfluxDB Enterprise, adjust the remote connection values accordingly.

  1. In your InfluxDB OSS instance, use the influx remote create command to create a remote connection to replicate data to.

    Provide the following:

    • Remote connection name
  2. InfluxDB Cloud region URL
  3. <ul>
    
    • Copy
    • Fill window
  4. InfluxDB Cloud API token (API token must have write access to the target bucket)
  5. <ul>
    
    • Copy
    • Fill window
  6. InfluxDB Cloud organization ID
  7. ```sh
    influx remote create \
      --name example-remote-name \
      --remote-url https://cloud2.influxdata.com \
      --remote-api-token mYsuP3r5Ecr37t0k3n \
      --remote-org-id 00xoXXoxXX00
    ```
    
    If you already have remote InfluxDB connections configured, you can use an existing connection. To view existing connections, run `influx remote list`.
    
    • Copy
    • Fill window
    1. In your InfluxDB OSS instance, use the influx replication create command to create a replication stream.

      Provide the following:

      • Replication stream name
    2. Remote connection ID
    3. <ul>
      
      • Copy
      • Fill window
    4. InfluxDB OSS bucket ID to replicate writes from
    5. <ul>
      
      • Copy
      • Fill window
    6. InfluxDB Cloud bucket ID to replicate writes to
    7. ```sh
      influx replication create \
        --name REPLICATION_STREAM_NAME \
        --remote-id REPLICATION_REMOTE_ID \
        --local-bucket-id INFLUX_BUCKET_ID \
        --remote-bucket REMOTE_INFLUX_BUCKET_NAME
      ```
      
      • Copy
      • Fill window

      Once a replication stream is created, InfluxDB will replicate all writes to the specified bucket to the InfluxDB Cloudbucket. Use the influx replication list command to view information such as the current queue size, max queue size, and latest status code.

      Important things to note

      • Only write operations are replicated. Other data operations (like deletes or restores) are not replicated.
      • In InfluxDB OSS, large write request bodies are written entirely. When replicated, write requests are sent to the remote bucket in batches. The maximum batch size is 500 kB (typically between 250 to 500 lines of line protocol). This may result in scenarios where some batches succeed and others fail.

      Replicate downsampled or processed data

      In some cases, you may not want to write raw, high-precision data to a remote InfluxDB Cloud instance. To replicate only downsampled or processed data:

      1. Create a bucket in your InfluxDB OSS instance to store downsampled or processed data in.

      2. Create an InfluxDB task that downsamples or processes data and stores it in the new bucket. For example:

        import "influxdata/influxdb/tasks"
        import "types"
        
        // omit this line if adding task via the UI
        option task = {name: "Downsample raw data", every: 10m}
        
        data = () => from(bucket: "example-bucket")
            |> range(start: tasks.lastSuccess(orTime: -task.every))
        
        numeric = data()
            |> filter(fn: (r) => types.isType(v: r._value, type: "float") or types.isType(v: r._value, type: "int") or types.isType(v: r._value, type: "uint"))
            |> aggregateWindow(every: task.every, fn: mean)
        
        nonNumeric = data()
            |> filter(fn: (r) => types.isType(v: r._value, type: "string") or types.isType(v: r._value, type: "bool"))
            |> aggregateWindow(every: task.every, fn: last)
        
        union(tables: [numeric, nonNumeric])
            |> to(bucket: "example-downsampled-bucket")
        
        • Copy
        • Fill window
      3. Create a replication stream to replicate data from the downsampled bucket to the remote InfluxDB Cloud instance.

      View InfluxDB OSS replication service metrics

      In addition to replication stream information that you can access using the CLI or API, you can view replication service-level metrics for your InfluxDB OSS instance, such as /api/v2/write error details, the number and duration of calls to the service, and the total number of points queued.

      To view replication service-level metrics send a GET request to your local InfluxDB OSS /metrics endpoint.

      For more information, see InfluxDB OSS metrics.


      Was this page helpful?

      Thank you for your feedback!


      The future of Flux

      Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

      Read more

      Now Generally Available

      InfluxDB 3 Core and Enterprise

      Start fast. Scale faster.

      Get the Updates

      InfluxDB 3 Core is an open source, high-speed, recent-data engine that collects and processes data in real-time and persists it to local disk or object storage. InfluxDB 3 Enterprise builds on Core’s foundation, adding high availability, read replicas, enhanced security, and data compaction for faster queries and optimized storage. A free tier of InfluxDB 3 Enterprise is available for non-commercial at-home or hobbyist use.

      For more information, check out:

      InfluxDB Cloud powered by TSM