Migrate data from InfluxDB 1.x to InfluxDB Clustered

To migrate data from an InfluxDB 1.x OSS or Enterprise instance to InfluxDB Clustered, export the data as line protocol and write the exported data to an InfluxDB database.

• Before you migrate
• Tools to use
• Migrate data

Before you migrate

Before you migrate from InfluxDB 1.x to InfluxDB Clustered, there are schema design practices supported by the TSM storage engine that are not supported in the InfluxDB v3 storage engine. Specifically, InfluxDB v3 enforces the following schema restrictions:

• You can’t use duplicate names for tags and fields.
• By default, measurements can contain up to 250 columns where each column represents time, a field, or a tag.

For more information, see Schema restrictions.

If your schema does not adhere to these restrictions, you must update your schema before migrating to InfluxDB Clustered.

home,room=Kitchen,temp=F co=0i,hum=56.6,temp=71.0 1672531200000000000

home,room=Kitchen,tempScale=F co=0i,hum=56.6,temp=71.0 1672531200000000000

SELECT 
  field1,
  field2,
  field3,
  field4
INTO
  "new-database".."new-measurement-1"
FROM
  "example-measurement"
GROUP BY *

SELECT 
  field5,
  field6,
  field7,
  field8
INTO
  "new-database".."new-measurement-2"
FROM
  "example-measurement"
GROUP BY *

example-measurement field1=0,field2=0,field3=0,field4=0,field5=0,field6=0,field7=0,field8=0 1672531200000000000

new-measurement-1 field1=0,field2=0,field3=0,field4=0 1672531200000000000
new-measurement-2 field5=0,field6=0,field7=0,field8=0 1672531200000000000

Tools to use

The migration process uses the following tools:

• influx_inspect utility: The influx_inspect utility is packaged with InfluxDB 1.x OSS and Enterprise.
• influxctl admin CLI.
• v1 API /write endpoint or v2 API /api/v2/write endpoint and API client libraries.

Migrate data

1. Export data from your InfluxDB 1.x instance as line protocol.

   Use the InfluxDB 1.x influx_inspect export utility to export data as line protocol and store it in a file. Include the following:

   • (Required) -lponly flag to export line protocol without InfluxQL DDL or DML.
   • (Required) -out flag with a path to an output file. Default is ~/.influxdb/export. Any subsequent export commands without the output file defined will overwrite the existing export file.
   • -compress flag to use gzip to compress the output.
   • -datadir flag with the path to your InfluxDB 1.x data directory. Only required if the data directory is at a non-default location. For information about default locations, see InfluxDB OSS 1.x file system layout or InfluxDB Enterprise 1.x file system layout.
   • -waldir flag with the path to your InfluxDB 1.x wal directory. Only required if the wal directory is at a non-default location. For information about default locations, see InfluxDB OSS 1.x file system layout or InfluxDB Enterprise 1.x file system layout.
   • -database flag with a specific database name to export. By default, all databases are exported.
   • -retention flag with a specific retention policy to export. By default, all retention policies are exported.
   • -start flag with an RFC3339 timestamp that defines the earliest time to export. Default is 1677-09-20T16:27:54-07:44.
   • -end flag with an RFC3339 timestamp that defines the latest time to export. Default is 2262-04-11T16:47:16-07:00.

   We recommend exporting each database and retention policy combination separately to easily write the exported data into corresponding InfluxDB Clustered databases.

   Export all data in a database and retention policy to a file

   influx_inspect export \
     -lponly \
     -database example-db \
     -retention example-rp \
     -out path/to/export-file.lp
   

   View more export command examples:

   Export all data to a file

   influx_inspect export \
     -lponly \
     -out path/to/export-file.lp.gzip
   

   Export all data to a compressed file

   influx_inspect export \
     -lponly \
     -compress \
     -out path/to/export-file.lp.gzip
   

   Export data within time bounds to a file

   influx_inspect export \
     -lponly \
     -start 2020-01-01T00:00:00Z \
     -end 2023-01-01T00:00:00Z \
     -out path/to/export-file.lp
   

   Export a database and all its retention policies to a file

   influx_inspect export \
     -lponly \
     -database example-db \
     -out path/to/export-file.lp
   

   Export a specific database and retention policy to a file

   influx_inspect export \
     -lponly \
     -database example-db \
     -retention example-rp \
     -out path/to/export-file.lp
   

   Export all data from non-default data and wal directories

   influx_inspect export \
     -lponly \
     -datadir path/to/influxdb/data/ \
     -waldir path/to/influxdb/wal/ \
     -out path/to/export-file.lp
   

2. Create InfluxDB Clustered databases for each InfluxDB 1.x database and retention policy combination.

   If coming from InfluxDB v1, the concepts of databases and retention policies have been combined into a single concept–database. Retention policies are no longer part of the InfluxDB data model. However, InfluxDB Clustered does support InfluxQL, which requires databases and retention policies. See InfluxQL DBRP naming convention.

   If coming from InfluxDB v2, InfluxDB Cloud (TSM), or InfluxDB Cloud Serverless, database and bucket are synonymous.

   View example 1.x databases and retention policies as InfluxDB Clustered databases

   If you have the following InfluxDB 1.x data structure:

   • example-db (database)
     • autogen (retention policy)
     • historical-1mo (retention policy)
     • historical-6mo (retention policy)
     • historical-1y (retention policy)

   You would create the following InfluxDB Clustered databases:

   • example-db/autogen
   • example-db/historical-1mo
   • example-db/historical-6mo
   • example-db/historical-1y

   Use the influxctl database create command to create a database in your InfluxDB cluster.

   Provide the following arguments:

   • (Optional) Database retention period (default is infinite)
   • Database name (see Database naming restrictions)

   influxctl database create --retention-period 30d <DATABASE_NAME>
   

   To learn more about databases in InfluxDB Clustered, see Manage databases.

3. Create a database token for writing to your InfluxDB Clustered database.

   Use the influxctl token create command to create a database token with write permission to your database.

   Provide the following:

   • Permission grants
     • --read-database: Grants read access to a database
     • --write-database Grants write access to a database
   • Token description

   influxctl token create \
     --read-database example-db \
     --write-database example-db \
     "Read/write token for example-db database"
   

4. Write the exported line protocol to your InfluxDB cluster.

   Use the v1 API or v2 API endpoints to write data to your InfluxDB cluster.

   Choose from the following options:

   • The v1 API /write endpoint with v1 client libraries or HTTP clients.
   • The v2 API /api/v2/write endpoint with v2 client libraries or HTTP clients.

   Write each export file to the target database.

   v2.x influx CLI not supported

   Don’t use the influx CLI with InfluxDB Clustered. While it may coincidentally work, it isn’t officially supported.

   For help finding the best workflow for your situation, contact Support.