Documentation

InfluxDB OSS

v2

Replicate data from InfluxDB OSS

Use InfluxDB replication streams (InfluxDB Edge Data Replication) to replicate the incoming data of select buckets to one or more buckets on a remote InfluxDB OSS, InfluxDB Cloud, or InfluxDB Enterprise instance.

Replicate data from InfluxDB OSS to InfluxDB Cloud, InfluxDB OSS, or InfluxDB Enterprise.

- [View replication service metrics](#view-influxdb-oss-replication-service-metrics)

Configure a replication stream

Use the influx CLI or the InfluxDB 2.7 API to configure a replication stream.

To replicate data to InfluxDB OSS or InfluxDB Enterprise, adjust the remote connection values accordingly.

CLI API

  1. In your local InfluxDB OSS instance, use the influx remote create command to create a remote connection to replicate data to.

    Provide the following:

    • Remote connection name
    • Remote InfluxDB instance URL
    • Remote InfluxDB API token (API token must have write access to the target bucket)
    • Remote InfluxDB organization ID
    influx remote create \
  --name example-remote-name \
  --remote-url https://cloud2.influxdata.com \
  --remote-api-token mYsuP3r5Ecr37t0k3n \
  --remote-org-id 00xoXXoxXX00

    If you already have remote InfluxDB connections configured, you can use an existing connection. To view existing connections, run influx remote list.

  2. In your local InfluxDB OSS instance, use the influx replication create command to create a replication stream.

    Provide the following:

    • Replication stream name
    • Remote connection ID
    • Local bucket ID to replicate writes from
    • Remote bucket name or ID to replicate writes to. If replicating to InfluxDB Enterprise, use the db-name/rp-name bucket name syntax.
    influx replication create \
  --name REPLICATION_STREAM_NAME \
  --remote-id REPLICATION_REMOTE_ID \
  --local-bucket-id INFLUX_BUCKET_ID \
  --remote-bucket REMOTE_INFLUX_BUCKET_NAME

Once a replication stream is created, InfluxDB OSS will replicate all writes to the specified bucket to the remote InfluxDB bucket. Use the influx replication list command to view information such as the current queue size, max queue size, and latest status code.

  1. Send a POST request to your local InfluxDB OSS /api/v2/remotes endpoint to create a remote connection to replicate data to.

    POST localhost:8086/api/v2/remotes

    Include the following in your request:

    • Request method: POST
    • Headers:
      • Authorization: Token scheme with your local InfluxDB OSS API token
      • Content-type: application/json
    • Request body: JSON object with the following fields:
      * Required
      • * allowInsecureTLS: All insecure TLS connections
      • description: Remote description
      • * name: Remote connection name
      • * orgID: local InfluxDB OSS organization ID
      • * remoteAPIToken: Remote InfluxDB API token (API token must have write access to the target bucket)
      • * remoteOrgID: Remote InfluxDB organization ID
      • * remoteURL: Remote InfluxDB instance URL
    curl --request POST http://localhost:8086/api/v2/remotes \
  --header 'Authorization: Token INFLUX_OSS_TOKEN' \
  --data '{
    "allowInsecureTLS": false,
    "description": "Example remote description",
    "name": "Example remote name",
    "orgID": "INFLUX_OSS_ORG_ID",
    "remoteAPIToken": "REMOTE_INFLUX_TOKEN",
    "remoteOrgID": "REMOTE_INFLUX_ORG_ID",
    "remoteURL": "https://cloud2.influxdata.com"
  }'

    If you already have remote InfluxDB connections configured, you can use an existing connection. To view existing connections, use the /api/v2/remotes endpoint with the GET request method.

    GET localhost:8086/api/v2/remotes

    Include the following in your request:

    • Request method: GET
    • Headers:
      • Authorization: Token scheme with your local InfluxDB OSS API token
    • Query parameters:
      • orgID: Local InfluxDB OSS organization ID
    curl --request GET \
  http://localhost:8086/api/v2/remotes?orgID=INFLUX_OSS_ORG_ID \
  --header 'Authorization: Token INFLUX_OSS_TOKEN' \

  2. Send a POST request to your local InfluxDB OSS /api/v2/replications endpoint to create a replication stream.

    POST localhost:8086/api/v2/remotes

    Include the following in your request:

    • Request method: POST
    • Headers:
      • Authorization: Token scheme with your local InfluxDB OSS API token
      • Content-type: application/json
    • Request body: JSON object with the following fields: * Required
      • dropNonRetryableData: Drop data when a non-retryable error is encountered.
      • * localBucketID: Local InfluxDB OSS bucket ID to replicate writes from.
      • * maxAgeSeconds: Maximum age of data in seconds before it is dropped (default is 604800, must be greater than or equal to 0).
      • * maxQueueSizeBytes: Maximum replication queue size in bytes (default is 67108860, must be greater than or equal to 33554430).
      • * name: Replication stream name.
      • * orgID: Local InfluxDB OSS organization ID.
      • * remoteBucketID: Remote bucket ID to replicate writes to.
      • * remoteBucketName: Remote bucket name to replicate writes to. If replicating to InfluxDB Enterprise, use the db-name/rp-name bucket name syntax.
      • * remoteID: Remote connection ID

remoteBucketID and remoteBucketName are mutually exclusive. If replicating to InfluxDB Enterprise, use remoteBucketName with the db-name/rp-name bucket name syntax.

curl --request POST http://localhost:8086/api/v2/replications \
  --header 'Authorization: Token INFLUX_OSS_TOKEN' \
  --data '{
    "dropNonRetryableData": false,
    "localBucketID": "INFLUX_OSS_BUCKET_ID",
    "maxAgeSeconds": 604800,
    "maxQueueSizeBytes": 67108860,
    "name": "Example replication stream name",
    "orgID": "INFLUX_OSS_ORG_ID",
    "remoteBucketName": "REMOTE_INFLUX_BUCKET_NAME",
    "remoteID": "REMOTE_ID",
  }'

Once a replication stream is created, InfluxDB OSS will replicate all writes from the specified local bucket to the remote InfluxDB bucket. To get information such as the current queue size, max queue size, and latest status code for each replication stream, send a GET request to your local InfluxDB OSS /api/v2/replications endpoint.

GET localhost:8086/api/v2/replications

Include the following in your request:

  • Request method: GET
  • Headers:
    • Authorization: Token scheme with your local InfluxDB OSS API token
  • Query parameters:
    • orgID: Local InfluxDB OSS organization ID
curl --request GET \
  http://localhost:8086/api/v2/replications?orgID=INFLUX_OSS_ORG_ID \
  --header 'Authorization: Token INFLUX_OSS_TOKEN' \

Important things to note

  • Only write operations are replicated. Other data operations (like deletes or restores) are not replicated.
  • In InfluxDB OSS, large write request bodies are written entirely. When replicated, write requests are sent to the remote bucket in batches. The maximum batch size is 500 kB (typically between 250 to 500 lines of line protocol). This may result in scenarios where some batches succeed and others fail.

Replicate downsampled or processed data

In some cases, you may not want to write raw, high-precision data to a remote InfluxDB instance. To replicate only downsampled or processed data:

  1. Create a bucket in your InfluxDB OSS instance to store downsampled or processed data in.

  2. Create an InfluxDB task that downsamples or processes data and stores it in the new bucket. For example:

    import "influxdata/influxdb/tasks"
import "types"

// omit this line if adding task via the UI
option task = {name: "Downsample raw data", every: 10m}

data = () => from(bucket: "example-bucket")
    |> range(start: tasks.lastSuccess(orTime: -task.every))

numeric = data()
    |> filter(fn: (r) => types.isType(v: r._value, type: "float") or types.isType(v: r._value, type: "int") or types.isType(v: r._value, type: "uint"))
    |> aggregateWindow(every: task.every, fn: mean)

nonNumeric = data()
    |> filter(fn: (r) => types.isType(v: r._value, type: "string") or types.isType(v: r._value, type: "bool"))
    |> aggregateWindow(every: task.every, fn: last)

union(tables: [numeric, nonNumeric])
    |> to(bucket: "example-downsampled-bucket")

  3. Create a replication stream to replicate data from the downsampled bucket to the remote InfluxDB instance.

View InfluxDB OSS replication service metrics

In addition to replication stream information that you can access using the CLI or API, you can view replication service-level metrics for your InfluxDB OSS instance, such as /api/v2/write error details, the number and duration of calls to the service, and the total number of points queued.

To view replication service-level metrics send a GET request to your local InfluxDB OSS /metrics endpoint.

For more information, see InfluxDB OSS metrics.

write replication

Was this page helpful?

Thank you for your feedback!

© 2024 InfluxData, Inc.

What is your InfluxDB OSS URL?

Customize your InfluxDB OSS URL and we’ll update code examples for you.

Default
Custom

For more information, see InfluxDB OSS URLs.

Thank you for your feedback!

Let us know what we can do better:

Thank you!

No thanks

The future of Flux

Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

Read more

InfluxDB v3 enhancements and InfluxDB Clustered is now generally available

New capabilities, including faster query performance and management tooling advance the InfluxDB v3 product line. InfluxDB Clustered is now generally available.

InfluxDB v3 performance and features

The InfluxDB v3 product line has seen significant enhancements in query performance and has made new management tooling available. These enhancements include an operational dashboard to monitor the health of your InfluxDB cluster, single sign-on (SSO) support in InfluxDB Cloud Dedicated, and new management APIs for tokens and databases.

Learn about the new v3 enhancements

InfluxDB Clustered general availability

InfluxDB Clustered is now generally available and gives you the power of InfluxDB v3 in your self-managed stack.

Talk to us about InfluxDB Clustered