Back up and restore data

Use the InfluxDB OSS 1.11 backup, restore, export, and import utilities to prevent unexpected data loss and preserve the ability to restore data if it ever is lost.

You can use these tools in your back up and restore procedures to:

• Provide disaster recovery due to unexpected events
• Migrate data to new environments or servers
• Restore instances to a consistent state
• Export and import data for debugging

Depending on the volume of data to be protected and your application requirements, InfluxDB OSS 1.11 offers two methods, described below, for managing backups and restoring data:

• Backup and restore utilities: Use for most applications
• Exporting and importing data: Use for very large datasets and to debug data

Backup and restore utilities

Use InfluxDB OSS 1.11 backup and restore utilities to:

• Back up and restore multiple databases at a time.
• Back up specific time ranges.
• Create backup files compatible with InfluxDB Enterprise.

Use the backup and restore utilities to back up and restore data for influxd instances of InfluxDB OSS version 1.4 and earlier, version 1.5 and later, and InfluxDB Enterprise.

• Backup formats
  • Specify your backup format
  • Determine your backup’s format
• Backup and restore requirements
• Important notes
  • Time-based backups
• Back up data
  • Backup examples (portable format)
  • Backup examples (legacy format)
• Restore data
  • Restore examples (portable format)
  • Online restore examples (legacy format)
  • Offline restore examples (legacy format)
  • Restore data to an existing database
• Configure backup and restore services

Backup formats

The InfluxDB backup utility outputs data backups in one of two formats: legacy or portable. Each format provides different functionality and support for other versions of InfluxDB. The major difference between the formats is that the legacy format can only be used with InfluxDB OSS 1.x. The portable format is “portable” between InfluxDB 1.5–1.11 and InfluxDB Enterprise.

Specify your backup format

By default, the backup and restore utilities create and restore backups in the legacy format. To create or restore from a backup or in the portable format, include the -portable flag with your backup command.

# Create a backup in the portable format
influxd backup -portable /path/to/backup-destination

# Restore from a portable backup
influxd restore -portable /path/to/backup-destination

Determine your backup’s format

Use the directory structure of the backup directory to determine the format of the backup.

Portable format directory structure

Legacy format directory structure

Backup and restore requirements

• The InfluxDB service (influxd) must be running except when performing an offline legacy restore.
• Both the source and target InfluxDB instances must be the same InfluxDB version or differ by only a minor version. For example, you can back up data from InfluxDB 1.7 and restore it to an InfluxDB 1.8 instance. The only exception is restoring data from InfluxDB 1.8 into 1.11.

Important notes

Time-based backups

When executing a backup with -start or -end flags, InfluxDB backs up shards, not individual points. Each shard is annotated with starting and ending timestamps for the time interval included in the shard. If a shard contains any points in the specified time range, the entire shard is included in the backup.

Since most shards are highly compacted, extracting shards to inspect each point would be highly inefficient and add a significant burden to the underlying system.

When restoring data from a time-based backup, you are likely to see data outside of the backed up time range.

Back up data

Use the influxd backup utility to create a backup of time series data and metadata stored in InfluxDB.

Provide the following for each backup format:

influxd backup -portable /path/to/backup-directory

influxd backup -portable \
  -host 203.0.113.0:8088 \
  /path/to/backup-directory

influxd backup -portable \
  -start 2022-01-01T00:00:00Z \
  -stop 2022-02-01T00:00:00Z \
  /path/to/backup-directory

influxd backup -portable \
  -start 2022-01-01T00:00:00Z \
  /path/to/backup-directory

influxd backup -portable \
  -db example-db \
  /path/to/backup-directory

influxd backup -portable \
  -db example-db \
  -rp example-retention-policy \
  /path/to/backup-directory

influxd backup -portable \
  -rp example-retention-policy \
  -shard 123 \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  -host 203.0.113.0:8088 \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  -start 2022-01-01T00:00:00Z \
  -stop 2022-02-01T00:00:00Z \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  -start 2022-01-01T00:00:00Z \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  -rp example-retention-policy \
  /path/to/backup-directory

influxd backup -portable \
  -db example-db \
  -rp example-retention-policy \
  -shard 123 \
  /path/to/backup-directory

Restore data

Use the influxd restore utility to restore time series data and metadata to InfluxDB from an InfluxDB backup.

error updating meta: DB metadata not changed. database may already exist

Requirements and commands necessary to restore a backup depend on the backup format:

influxd restore -portable /path/to/backup-directory

influxd restore -portable \
  -host 203.0.113.0:8088 \
  /path/to/backup-directory

influxd restore -portable \
  -db example-db \
  /path/to/backup-directory

influxd restore -portable \
  -db example-db \
  -newdb example-new-db \
  /path/to/backup-directory

influxd backup -portable \
  -db example-db \
  -rp example-retention-policy \
  /path/to/backup-directory

influxd restore -portable \
  -db example-db \
  -rp example-rp \
  -newrp example-new-rp \
  /path/to/backup-directory

influxd backup -portable \
  -db example-db \
  -rp example-rp \
  -shard 123 \
  /path/to/backup-directory

influxd restore -online \
  -db example-db \
  /path/to/backup-directory

influxd restore -online \
  -db example-db \
  -host 203.0.113.0:8088 \
  /path/to/backup-directory

influxd restore -online \
  -db example-db \
  -newdb example-new-db \
  /path/to/backup-directory

influxd backup -online \
  -db example-db \
  -rp example-retention-policy \
  /path/to/backup-directory

influxd restore -online \
  -db example-db \
  -rp example-rp \
  -newrp example-new-rp \
  /path/to/backup-directory

influxd backup -online \
  -db example-db \
  -rp example-rp \
  -shard 123 \
  /path/to/backup-directory

influxd restore \
  -db example-db \
  -datadir /path/to/data-directory \
  -metadir /path/to/meta-directory \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  -rp example-retention-policy \
  -datadir /path/to/data-directory \
  -metadir /path/to/meta-directory \
  /path/to/backup-directory

influxd backup \
  -db example-db \
  -rp example-rp \
  -shard 123 \
  -datadir /path/to/data-directory \
  -metadir /path/to/meta-directory \
  /path/to/backup-directory

Restore data to an existing database

While backups can’t be restored directly to an existing database (except for offline legacy restores), you can do the following to restore data to an existing database:

1. Restore a backup to a temporary database.

   Portable format Legacy format

   influxd restore -portable \
     -db example-db \
     -newdb example-tmp-db \
     /path/to/backup-directory/
   

   influxd restore \
     -db example-db \
     -newdb example-tmp-db \
     /path/to/backup-directory/
   

2. Use InfluxQL or Flux to query data from the temporary database and write it back to the existing database.

   InfluxQL Flux

   -- Repeat for each retention policy in the temporary database
   SELECT *
   INTO "example-db".autogen.:MEASUREMENT
   FROM "example-tmp-db".autogen./.*/ GROUP BY *
   

   // Repeat for each retention policy in the temporary database
   from(bucket: "example-tmp-db/autogen")
       |> range(start: 0)
       |> to(bucket: "example-db/autogen")
   

   InfluxDB handles duplicate points added by this query as it normally does. For more information, see How does InfluxDB handle duplicate points?

3. Use InfluxQL to delete the temporary database.

   DROP DATABASE "example-tmp-db"
   

Configure backup and restore services

InfluxDB OSS 1.11 backup and restore utilities execute over a TCP connection. The default IP and port used for backup and restore remote procedure calls (RPCs) are 127.0.0.1:8088.

To customize the TCP IP and port the backup and restore services use, uncomment and update the bind-address configuration setting at the root level of your InfluxDB configuration file (influxdb.conf).

# Bind address to use for the RPC service for backup and restore.
bind-address = "127.0.0.1:8088"

Exporting and importing data

For most applications, the backup and restore utilities provide the tools you need for your backup and restore strategy. However, in some cases, the standard backup and restore utilities might not adequately handle the volumes of data in your application.

As an alternative to the standard backup and restore utilities, use the InfluxDB influx_inspect export and influx -import commands to create backup and restore procedures for your disaster recovery and backup strategy. These commands can be executed manually or included in shell scripts that run the export and import operations at scheduled intervals. You can use the commands to export and import data between an InfluxDB OSS 1.11 instance and an InfluxDB Enterprise cluster.

• Exporting data
• Importing data
• Example: export and import for disaster recovery

Exporting data

Use the influx_inspect export command to export data in line protocol format from your InfluxDB OSS 1.11 instance. Options include the following:

• -database: Export all or specific databases
• -start and -end: Filter with starting and ending timestamps
• -compress: Use GNU zip (gzip) compression for smaller files and faster exports

The following example shows how to export data filtered to one day and compressed for optimal speed and file size:

influx_inspect export \
  -database DATABASE_NAME \
  -compress \
  -start 2019-05-19T00:00:00.000Z \
  -end 2019-05-19T23:59:59.999Z

The exported file contains the following:

# DDL
CREATE DATABASE <DATABASE_NAME> WITH NAME <RETENTION_POLICY>
# DML
# CONTEXT-DATABASE:<DATABASE_NAME>
# CONTEXT-RETENTION-POLICY:<RETENTION_POLICY>

<LINE_PROTOCOL_DATA>

• DDL: an InfluxQL CREATE statement to create the target database when importing the data
• DML: Context metadata that specifies the target database and retention policy for importing the data
• the line protocol data

For details on optional settings and usage, see influx_inspect export command.

Importing data

To import line protocol data from a file, use the influx -import CLI command.

In your import file, include the following sections:

• Optional: DDL (Data Definition Language): Contains the InfluxQL commands for creating the relevant database and managing the retention policy. If your database and retention policy already exist, your file can skip this section.
• DML (Data Manipulation Language): Context metadata that specifies the database and (if desired) retention policy for the import and contains the data in line protocol.

In the following example, the compressed data file (in GNU zip format) is imported into the database specified in the file’s DML metadata.

influx -import -path -compressed

For details on using the influx -import command, see Import data from a file.

Example: export and import for disaster recovery

For an example of using the exporting and importing data approach for disaster recovery, see the presentation from Influxdays 2019 on “Architecting for Disaster Recovery.”. In this presentation, Capital One discusses the following:

• Exporting data every 15 minutes from an active InfluxDB Enterprise cluster to an AWS S3 bucket.
• Replicating the export file in the S3 bucket using the AWS S3 copy command.
• Importing data every 15 minutes from the AWS S3 bucket to an InfluxDB Enterprise cluster available for disaster recovery.
• Advantages of the export-import approach over the standard backup and restore utilities for large volumes of data.
• Managing users and scheduled exports and imports with a custom administration tool.