Migrate data from InfluxDB 1.x to InfluxDB Cloud Serverless

To migrate data from an InfluxDB 1.x OSS or Enterprise instance to InfluxDB Cloud Serverless powered by the v3 storage engine, export the data as line protocol and write the exported data to a bucket in your InfluxDB Cloud Serverless organization. Because full data migrations will likely exceed your organizations’ limits and adjustable quotas, migrate your data in batches.

• Before you migrate
• Tools to use
• Migrate data

Before you migrate

Before you migrate from InfluxDB 1.x to InfluxDB Cloud Serverless, there are schema design practices supported by the TSM storage engine that are not supported in the InfluxDB v3 storage engine. Specifically, InfluxDB v3 enforces the following schema restrictions:

• You can’t use duplicate names for tags and fields
• Measurements can contain up to 200 columns where each column represents time, a field, or a tag.

For more information, see Schema restrictions.

If your schema does not adhere to these restrictions, you must update your schema before migrating to InfluxDB Cloud Serverless.

home,room=Kitchen,temp=F co=0i,hum=56.6,temp=71.0 1672531200000000000

home,room=Kitchen,tempScale=F co=0i,hum=56.6,temp=71.0 1672531200000000000

SELECT 
  field1,
  field2,
  field3,
  field4
INTO
  "new-database".."new-measurement-1"
FROM
  "example-measurement"
GROUP BY *

SELECT 
  field5,
  field6,
  field7,
  field8
INTO
  "new-database".."new-measurement-2"
FROM
  "example-measurement"
GROUP BY *

example-measurement field1=0,field2=0,field3=0,field4=0,field5=0,field6=0,field7=0,field8=0 1672531200000000000

new-measurement-1 field1=0,field2=0,field3=0,field4=0 1672531200000000000
new-measurement-2 field5=0,field6=0,field7=0,field8=0 1672531200000000000

Tools to use

The migration process uses the following tools:

• influx_inspect utility:
  The influx_inspect utility is packaged with InfluxDB 1.x OSS and Enterprise.

• InfluxDB 2.x influx CLI:
  The 2.x influx CLI is packaged separately from InfluxDB OSS 2.x and InfluxDB Cloud Serverless. Download and install the 2.x CLI.

• InfluxDB Cloud user interface (UI):
  Visit cloud2.influxdata.com to access the InfluxDB Cloud UI.

Migrate data

1. Export data from your InfluxDB 1.x instance as line protocol.

   Use the InfluxDB 1.x influx_inspect export utility to export data as line protocol and store it in a file. Include the following:

   • (Required) -lponly flag to export line protocol without InfluxQL DDL or DML.
   • (Required) -out flag with a path to an output file. Default is ~/.influxdb/export. Any subsequent export commands without the output file defined will overwrite the existing export file.
   • -compress flag to use gzip to compress the output.
   • -datadir flag with the path to your InfluxDB 1.x data directory. Only required if the data directory is at a non-default location. For information about default locations, see InfluxDB OSS 1.x file system layout or InfluxDB Enterprise 1.x file system layout.
   • -waldir flag with the path to your InfluxDB 1.x wal directory. Only required if the wal directory is at a non-default location. For information about default locations, see InfluxDB OSS 1.x file system layout or InfluxDB Enterprise 1.x file system layout.
   • -database flag with a specific database name to export. By default, all databases are exported.
   • -retention flag with a specific retention policy to export. By default, all retention policies are exported.
   • -start flag with an RFC3339 timestamp that defines the earliest time to export. Default is 1677-09-20T16:27:54-07:44.
   • -end flag with an RFC3339 timestamp that defines the latest time to export. Default is 2262-04-11T16:47:16-07:00.

   We recommend exporting each database and retention policy combination separately to easily write the exported data into corresponding InfluxDB Cloud Serverless buckets.

   Export all data in a database and retention policy to a file

   influx_inspect export \
     -lponly \
     -database example-db \
     -retention example-rp \
     -out path/to/export-file.lp
   

   View more export command examples:

   Export all data to a file

   influx_inspect export \
     -lponly \
     -out path/to/export-file.lp.gzip
   

   Export all data to a compressed file

   influx_inspect export \
     -lponly \
     -compress \
     -out path/to/export-file.lp.gzip
   

   Export data within time bounds to a file

   influx_inspect export \
     -lponly \
     -start 2020-01-01T00:00:00Z \
     -end 2023-01-01T00:00:00Z \
     -out path/to/export-file.lp
   

   Export a database and all its retention policies to a file

   influx_inspect export \
     -lponly \
     -database example-db \
     -out path/to/export-file.lp
   

   Export a specific database and retention policy to a file

   influx_inspect export \
     -lponly \
     -database example-db \
     -retention example-rp \
     -out path/to/export-file.lp
   

   Export all data from non-default data and wal directories

   influx_inspect export \
     -lponly \
     -datadir path/to/influxdb/data/ \
     -waldir path/to/influxdb/wal/ \
     -out path/to/export-file.lp
   

2. Create InfluxDB Cloud Serverless buckets for each InfluxDB 1.x database and retention policy combination. InfluxDB Cloud Serverless combines InfluxDB 1.x databases and retention policies into buckets–named locations for time series data with specified retention periods.

   View example 1.x databases and retention policies as InfluxDB Cloud buckets

   If you have the following InfluxDB 1.x data structure:

   • example-db (database)
     • autogen (retention policy)
     • historical-1mo (retention policy)
     • historical-6mo (retention policy)
     • historical-1y (retention policy)

   You would create the following InfluxDB Cloud Serverless buckets:

   • example-db/autogen
   • example-db/historical-1mo
   • example-db/historical-6mo
   • example-db/historical-1y

   Use the InfluxDB 2.x influx CLI or the InfluxDB Cloud Serverless user interface (UI) to create a bucket.

   influx CLI InfluxDB UI

   Use the influx bucket create command to create a new bucket.

   Provide the following:

   • InfluxDB Cloud Serverless connection and authentication credentials
   • -n, --name flag with the bucket name.
   • -r, --retention flag with the bucket’s retention period duration. Supported retention periods depend on your InfluxDB Cloud Serverless plan.

   influx bucket create \
     --name example-db/autogen \
     --retention 7d
   

   1. Go to cloud2.influxdata.com in a browser to log in and access the InfluxDB UI.

   2. Navigate to Load Data > Buckets using the left navigation bar.

   Load Data

   3. Click + Create bucket.

   4. Provide a bucket name (for example: example-db/autogen) and select a retention period. Supported retention periods depend on your InfluxDB Cloud Serverless plan.

   5. Click Create.

3. Write the exported line protocol to your InfluxDB Cloud Serverless organization.

   Use the InfluxDB 2.x CLI to write data to InfluxDB Cloud Serverless. While you can use the /api/v2/write API endpoint to write data directly, the influx write command lets you define the rate at which data is written to avoid exceeding your organization’s rate limits.

   Use the influx write command and include the following:

   • InfluxDB Cloud Serverless connection and authentication credentials
   • -b, --bucket flag to identify the target bucket.
   • -f, --file flag with the path to the line protocol file to import.
   • -rate-limit flag with a rate limit that matches your InfluxDB Cloud organization’s write rate limit.
   • --compression flag to identify the compression type of the import file. Options are none or gzip. Default is none.

   Authentication credentials

   The examples below assume your InfluxDB host, organization, and token are provided by either the active influx CLI configuration or by environment variables (INFLUX_HOST, INFLUX_ORG, and INFLUX_TOKEN). If you do not have a CLI configuration set up or the environment variables set, include these required credentials for each command with the following flags:

   • --host: InfluxDB host
   • -o, --org or --org-id: InfluxDB organization name or ID
   • -t, --token: InfluxDB API token

   Uncompressed Compressed

   influx write \
     --bucket example-db/autogen \
     --file path/to/export-file.lp \
     --rate-limit "300 MB / 5 min"
   

   influx write \
     --bucket example-db/autogen \
     --file path/to/export-file.lp.gzip \
     --rate-limit "300 MB / 5 min" \
     --compression gzip
   

   Repeat for each export file and target bucket.