Replicate data from InfluxDB OSS
Use InfluxDB replication streams (InfluxDB Edge Data Replication) to replicate the incoming data of select buckets to one or more buckets on a remote InfluxDB OSS, InfluxDB Cloud, or InfluxDB Enterprise instance.
Replicate data from InfluxDB OSS to InfluxDB Cloud, InfluxDB OSS, or InfluxDB Enterprise.
Configure a replication stream
Use the influx
CLI or the
InfluxDB 2.7 API to configure
a replication stream.
To replicate data to InfluxDB OSS or InfluxDB Enterprise, adjust the remote connection values accordingly.
Send a
POST
request to your local InfluxDB OSS/api/v2/remotes
endpoint to create a remote connection to replicate data to.Include the following in your request:
- Request method:
POST
- Headers:
- Authorization:
Token
scheme with your local InfluxDB OSS API token - Content-type:
application/json
- Authorization:
- Request body: JSON object with the following fields:
* Required- * allowInsecureTLS: All insecure TLS connections
- description: Remote description
- * name: Remote connection name
- * orgID: local InfluxDB OSS organization ID
- * remoteAPIToken: Remote InfluxDB API token (API token must have write access to the target bucket)
- * remoteOrgID: Remote InfluxDB organization ID
- * remoteURL: Remote InfluxDB instance URL
curl --request POST http://localhost:8086/api/v2/remotes \ --header 'Authorization: Token INFLUX_OSS_TOKEN' \ --data '{ "allowInsecureTLS": false, "description": "Example remote description", "name": "Example remote name", "orgID": "INFLUX_OSS_ORG_ID", "remoteAPIToken": "REMOTE_INFLUX_TOKEN", "remoteOrgID": "REMOTE_INFLUX_ORG_ID", "remoteURL": "https://cloud2.influxdata.com" }'
If you already have remote InfluxDB connections configured, you can use an existing connection. To view existing connections, use the
/api/v2/remotes
endpoint with theGET
request method.Include the following in your request:
- Request method:
GET
- Headers:
- Authorization:
Token
scheme with your local InfluxDB OSS API token
- Authorization:
- Query parameters:
- orgID: Local InfluxDB OSS organization ID
curl --request GET \ http://localhost:8086/api/v2/remotes?orgID=INFLUX_OSS_ORG_ID \ --header 'Authorization: Token INFLUX_OSS_TOKEN' \
- Request method:
Send a
POST
request to your local InfluxDB OSS/api/v2/replications
endpoint to create a replication stream.Include the following in your request:
- Request method:
POST
- Headers:
- Authorization:
Token
scheme with your local InfluxDB OSS API token - Content-type:
application/json
- Authorization:
- Request body: JSON object with the following fields:
* Required
- dropNonRetryableData: Drop data when a non-retryable error is encountered.
- * localBucketID: Local InfluxDB OSS bucket ID to replicate writes from.
- * maxAgeSeconds: Maximum age of data in seconds before it is dropped (default is
604800
, must be greater than or equal to0
). - * maxQueueSizeBytes: Maximum replication queue size in bytes (default is
67108860
, must be greater than or equal to33554430
). - * name: Replication stream name.
- * orgID: Local InfluxDB OSS organization ID.
- * remoteBucketID: Remote bucket ID to replicate writes to.
- * remoteBucketName: Remote bucket name to replicate writes to. If replicating to InfluxDB Enterprise, use the
db-name/rp-name
bucket name syntax. - * remoteID: Remote connection ID
- Request method:
remoteBucketID
and remoteBucketName
are mutually exclusive.
If replicating to InfluxDB Enterprise, use remoteBucketName
with the db-name/rp-name
bucket name syntax.
curl --request POST http://localhost:8086/api/v2/replications \
--header 'Authorization: Token INFLUX_OSS_TOKEN' \
--data '{
"dropNonRetryableData": false,
"localBucketID": "INFLUX_OSS_BUCKET_ID",
"maxAgeSeconds": 604800,
"maxQueueSizeBytes": 67108860,
"name": "Example replication stream name",
"orgID": "INFLUX_OSS_ORG_ID",
"remoteBucketName": "REMOTE_INFLUX_BUCKET_NAME",
"remoteID": "REMOTE_ID",
}'
Once a replication stream is created, InfluxDB OSS
will replicate all writes from the specified local bucket to the remote
InfluxDB bucket.
To get
information such as the current queue size, max queue size, and latest status
code for each replication stream, send a GET
request to your local InfluxDB OSS /api/v2/replications
endpoint.
Include the following in your request:
- Request method:
GET
- Headers:
- Authorization:
Token
scheme with your local InfluxDB OSS API token
- Authorization:
- Query parameters:
- orgID: Local InfluxDB OSS organization ID
curl --request GET \
http://localhost:8086/api/v2/replications?orgID=INFLUX_OSS_ORG_ID \
--header 'Authorization: Token INFLUX_OSS_TOKEN' \
Important things to note
- Only write operations are replicated. Other data operations (like deletes or restores) are not replicated.
- In InfluxDB OSS, large write request bodies are written entirely. When replicated, write requests are sent to the remote bucket in batches. The maximum batch size is 500 kB (typically between 250 to 500 lines of line protocol). This may result in scenarios where some batches succeed and others fail.
Replicate downsampled or processed data
In some cases, you may not want to write raw, high-precision data to a remote InfluxDB instance. To replicate only downsampled or processed data:
Create a bucket in your InfluxDB OSS instance to store downsampled or processed data in.
Create an InfluxDB task that downsamples or processes data and stores it in the new bucket. For example:
import "influxdata/influxdb/tasks" import "types" // omit this line if adding task via the UI option task = {name: "Downsample raw data", every: 10m} data = () => from(bucket: "example-bucket") |> range(start: tasks.lastSuccess(orTime: -task.every)) numeric = data() |> filter(fn: (r) => types.isType(v: r._value, type: "float") or types.isType(v: r._value, type: "int") or types.isType(v: r._value, type: "uint")) |> aggregateWindow(every: task.every, fn: mean) nonNumeric = data() |> filter(fn: (r) => types.isType(v: r._value, type: "string") or types.isType(v: r._value, type: "bool")) |> aggregateWindow(every: task.every, fn: last) union(tables: [numeric, nonNumeric]) |> to(bucket: "example-downsampled-bucket")
Create a replication stream to replicate data from the downsampled bucket to the remote InfluxDB instance.
View InfluxDB OSS replication service metrics
In addition to replication stream information that you can access using the CLI or API, you can view replication service-level metrics for your InfluxDB OSS instance, such as /api/v2/write
error details, the number and duration of calls to the service, and the total number of points queued.
To view replication service-level metrics send a GET
request to your local InfluxDB OSS /metrics
endpoint.
For more information, see InfluxDB OSS metrics.
Was this page helpful?
Thank you for your feedback!
Support and feedback
Thank you for being part of our community! We welcome and encourage your feedback and bug reports for InfluxDB and this documentation. To find support, use the following resources:
Customers with an annual or support contract can contact InfluxData Support.