Data ingest lifecycle best practices

Limited availability

InfluxDB Clustered is currently only available to a limited group of InfluxData customers. If interested in being part of the limited access group, please contact the InfluxData Sales team.

Data ingested into InfluxDB must conform to the retention period of the database in which it is stored. Points with timestamps outside of the retention period are no longer queryable, but may still have references maintained in Object storage or the Catalog, resulting in an increase in operational overhead and cost. To reduce these factors, it is important to manage the lifecycle of ingested data.

Use the following best practices to manage the lifecycle of data in your InfluxDB cluster:

Use appropriate retention periods

When creating or updating a database, use a retention period that is appropriate for your requirements. Storing data longer than is required adds unnecessary operational cost to your InfluxDB cluster.

Tune garbage collection

Once data falls outside of a database’s retention period, the garbage collection service can remove all artifacts associated with the data from the Catalog and Object store. Tune the garbage collector cutoff period to ensure that data is removed in a timely manner.

Use the following environment variables to tune the garbage collector:

  • INFLUXDB_IOX_GC_OBJECTSTORE_CUTOFF: the age at which Parquet files not referenced in the Catalog become eligible for deletion from Object storage. The default is 30d.
  • INFLUXDB_IOX_GC_PARQUETFILE_CUTOFF: how long to retain rows in the Catalog that reference Parquet files marked for deletion. The default is 30d.

To ensure there is a grace period before files and references are removed, the minimum garbage collector (GC) object store and Parquet file cutoff time is three hours (3h).

We recommend setting these options to a value aligned to your organization’s backup and recovery strategy. For example, a value of 6h (6 hours) would be appropriate for running a lean Catalog that only maintains references to recent data and does not require backups.

Use case examples

Use the following scenarios as a guide for different use cases:

Leading edge data with no backups

Custom backup window with object storage versioning

Custom backup window without object storage versioning

Was this page helpful?

Thank you for your feedback!

The future of Flux

Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

Flux is going into maintenance mode and will not be supported in InfluxDB 3.0. This was a decision based on the broad demand for SQL and the continued growth and adoption of InfluxQL. We are continuing to support Flux for users in 1.x and 2.x so you can continue using it with no changes to your code. If you are interested in transitioning to InfluxDB 3.0 and want to future-proof your code, we suggest using InfluxQL.

For information about the future of Flux, see the following: