Documentation

Use the PyArrow library to analyze data

Use PyArrow to read and analyze query results from InfluxDB Cloud Dedicated. The PyArrow library provides efficient computation, aggregation, serialization, and conversion of Arrow format data.

Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enable big data systems to store, process and move data fast.

The Arrow Python bindings (also named “PyArrow”) have first-class integration with NumPy, pandas, and built-in Python objects. They are based on the C++ implementation of Arrow.

Install prerequisites

The examples in this guide assume using a Python virtual environment and the InfluxDB v3 influxdb3-python Python client library. For more information, see how to get started using Python to query InfluxDB.

Installing influxdb3-python also installs the pyarrow library that provides Python bindings for Apache Arrow.

Use PyArrow to read query results

The following example shows how to use influxdb3-python and pyarrow to query InfluxDB and view Arrow data as a PyArrow Table.

  1. In your editor, copy and paste the following sample code to a new file–for example, pyarrow-example.py:

    # pyarrow-example.py
    
    from influxdb_client_3 import InfluxDBClient3
    import pandas
    
    def querySQL():
      
      # Instantiate an InfluxDB client configured for a database
      client = InfluxDBClient3(
        "https://cluster-id.a.influxdb.io",
        database="
    DATABASE_NAME
    "
    ,
    token="
    DATABASE_TOKEN
    "
    )
    # Execute the query to retrieve all record batches in the stream formatted as a PyArrow Table. table = client.query( '''SELECT * FROM home WHERE time >= now() - INTERVAL '90 days' ORDER BY time''' ) client.close() print(querySQL())
  2. Replace the following configuration values:

    • DATABASE_TOKEN: An InfluxDB database token with read permissions on the databases you want to query.
    • DATABASE_NAME: The name of the InfluxDB database to query.
  3. In your terminal, use the Python interpreter to run the file:

    python pyarrow-example.py
    

The InfluxDBClient3.query() method sends the query request, and then returns a pyarrow.Table that contains all the Arrow record batches from the response stream.

Next, use PyArrow to analyze data.

Use PyArrow to analyze data

Group and aggregate data

With a pyarrow.Table, you can use values in a column as keys for grouping.

The following example shows how to query InfluxDB, and then use PyArrow to group the table data and calculate an aggregate value for each group:

# pyarrow-example.py

from influxdb_client_3 import InfluxDBClient3
import pandas

def querySQL():
  
  # Instantiate an InfluxDB client configured for a database
  client = InfluxDBClient3(
    "https://cluster-id.a.influxdb.io",
    database="
DATABASE_NAME
"
,
token="
DATABASE_TOKEN
"
)
# Execute the query to retrieve data # formatted as a PyArrow Table table = client.query( '''SELECT * FROM home WHERE time >= now() - INTERVAL '90 days' ORDER BY time''' ) client.close() return table table = querySQL() # Use PyArrow to aggregate data print(table.group_by('room').aggregate([('temp', 'mean')]))

Replace the following:

  • DATABASE_TOKEN: An InfluxDB database token with read permissions on the databases you want to query.
  • DATABASE_NAME: The name of the InfluxDB database to query.

View example results

For more detail and examples, see the PyArrow documentation and the Apache Arrow Python Cookbook.


Was this page helpful?

Thank you for your feedback!


The future of Flux

Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

Read more

InfluxDB v3 enhancements and InfluxDB Clustered is now generally available

New capabilities, including faster query performance and management tooling advance the InfluxDB v3 product line. InfluxDB Clustered is now generally available.

InfluxDB v3 performance and features

The InfluxDB v3 product line has seen significant enhancements in query performance and has made new management tooling available. These enhancements include an operational dashboard to monitor the health of your InfluxDB cluster, single sign-on (SSO) support in InfluxDB Cloud Dedicated, and new management APIs for tokens and databases.

Learn about the new v3 enhancements


InfluxDB Clustered general availability

InfluxDB Clustered is now generally available and gives you the power of InfluxDB v3 in your self-managed stack.

Talk to us about InfluxDB Clustered