Configure specialized cluster nodes

Optimize performance for specific workloads in your InfluxDB 3 Enterprise cluster by configuring specialized nodes in distributed deployments. Assign specific modes and thread allocations to nodes to maximize cluster efficiency.

• Specialize nodes for specific workloads
• Configure node modes
• Allocate threads by node type
• Configure ingest nodes
• Configure query nodes
• Configure compactor nodes
• Configure a process node
• Multi-mode configurations
• Cluster architecture examples
• Scale your cluster
• Monitor performance
• Troubleshoot node configurations
• Best practices
• Migrate to specialized nodes
• Manage configurations

Specialize nodes for specific workloads

In an InfluxDB 3 Enterprise cluster, you can dedicate nodes to specific tasks:

• Ingest nodes: Optimized for high-throughput data ingestion
• Query nodes: Maximized for complex analytical queries
• Compactor nodes: Dedicated to data compaction and optimization
• Process node: Runs only the Processing Engine. We recommend a standalone process node, which runs only the Processing Engine. See Configure a standalone process node.
• All-in-one nodes: Balanced for mixed workloads (single-node deployments only)

Configure node modes

Pass the --mode parameter when starting the node to specify its capabilities:

# Single mode
influxdb3 serve --mode=ingest

# Multiple modes
influxdb3 serve --mode=ingest,query

# All modes (default, for single-node Enterprise only)
influxdb3 serve --mode=all

Available modes:

• all: All capabilities enabled (single-node Enterprise deployments only)
• ingest: Data ingestion and line protocol parsing
• query: Query execution and data retrieval
• compact: Background compaction and optimization
• process: Runs the Processing Engine, which runs plugins for triggers. Requires --plugin-dir. Setting --plugin-dir adds process mode to a node, whatever its --mode. In a cluster, run process alone on a dedicated node, see Configure a process node.

Allocate threads by node type

Critical concept: Thread pools

Every node has two thread pools that must be properly configured:

1. IO threads: Parse line protocol, handle HTTP requests
2. DataFusion threads: Execute queries, create data snapshots (convert WAL data to Parquet files), perform compaction

Configure ingest nodes

Ingest nodes handle high-volume data writes and require significant IO thread allocation for line protocol parsing.

Example medium ingester (32 cores)

influxdb3 \
  --num-io-threads=12 \
  serve \
  --num-cores=32 \
  --datafusion-num-threads=20 \
  --exec-mem-pool-size=60% \
  --mode=ingest \
  --node-id=ingester-01

Configuration rationale:

• 12 IO threads: Handle multiple concurrent writers (Telegraf agents, applications)
• 20 DataFusion threads: Required for data snapshot operations that convert buffered writes to Parquet files
• 60% memory pool: Balance between write buffers and data snapshot operations

Monitor ingest performance

Key metrics for ingest nodes:

# Monitor IO thread utilization
top -H -p $(pgrep influxdb3) | grep io_worker

# Check write request counts by endpoint
curl -s http://localhost:8181/metrics | grep 'http_requests_total.*write'

# Check overall HTTP request metrics
curl -s http://localhost:8181/metrics | grep 'http_requests_total'

# Monitor WAL size
du -sh /path/to/data/wal/

Configure query nodes

Query nodes execute complex analytical queries and need maximum DataFusion threads.

Analytical query node (64 cores)

influxdb3 \
  --num-io-threads=4 \
  serve \
  --num-cores=64 \
  --datafusion-num-threads=60 \
  --exec-mem-pool-size=90% \
  --file-cache-size=8GB \
  --mode=query \
  --node-id=query-01 \
  --cluster-id=prod-cluster

Configuration rationale:

• 4 IO threads: Minimal, just for HTTP request handling
• 60 DataFusion threads: Maximum parallelism for query execution
• 90% memory pool: Maximize memory for complex aggregations
• 8 GB Parquet cache: Keep frequently accessed data in memory

Real-time query node (32 cores)

influxdb3 \
  --num-io-threads=6 \
  serve \
  --num-cores=32 \
  --datafusion-num-threads=26 \
  --exec-mem-pool-size=80% \
  --file-cache-size=4GB \
  --mode=query \
  --node-id=query-02

Optimize query settings

You can configure datafusion properties for additional tuning of query nodes:

influxdb3 serve \
  --datafusion-config "datafusion.execution.batch_size:16384,datafusion.execution.target_partitions:60" \
  --mode=query

Configure compactor nodes

Compactor nodes optimize stored data through background compaction processes.

Dedicated compactor (32 cores)

influxdb3 \
  --num-io-threads=2 \
  serve \
  --num-cores=32 \
  --datafusion-num-threads=30 \
  --compaction-gen2-duration=24h \
  --compaction-check-interval=5m \
  --mode=compact \
  --node-id=compactor-01 \
  --cluster-id=prod-cluster

# Note: --compaction-row-limit option is not yet released in v3.5.0
# Uncomment when available in a future release:
# --compaction-row-limit=2000000 \

Configuration rationale:

• 2 IO threads: Minimal, compaction is DataFusion-intensive
• 30 DataFusion threads: Maximum threads for sort/merge operations
• 24h gen2 duration: Time-based compaction strategy

Tune compaction parameters

You can adjust compaction strategies to balance performance and resource usage:

# Configure compaction strategy
--compaction-multipliers=4,8,16 \
--compaction-max-num-files-per-plan=100

Distributed compaction

On the upgraded storage engine, you can spread compaction work across every compact node in the cluster instead of running it only on the lease holder. For setup and tuning guidance, see Distributed compaction.

Configure a process node

A process node is a node that runs the Processing Engine (process mode). It runs plugins for triggers. You can run the Processing Engine in one of two architectures:

• Requirements for every process node
• Standalone process node (recommended)
• Mixed process node (not recommended)

Requirements for every process node

The following rules apply to every architecture:

• Set --plugin-dir on the process node: --mode with process and no --plugin-dir fails at startup. Put your plugin files in that directory.
• Don’t set --plugin-dir on other nodes: Setting --plugin-dir or INFLUXDB3_PLUGIN_DIR adds process mode to a node, whatever its --mode. The official Docker image sets INFLUXDB3_PLUGIN_DIR=/plugins; on containers that shouldn’t run the Processing Engine, unset it with -e INFLUXDB3_UNSET_VARS=INFLUXDB3_PLUGIN_DIR.
• Don’t use all mode: In a cluster, list the process node’s modes explicitly.
• Set --internode-bind-addr on the nodes that plugins reach: A plugin’s influxdb3_local.query() call runs on a node with query mode. Its influxdb3_local.write() calls and log messages go to a node with ingest mode. If the process node doesn’t run one of those modes itself, it reaches another node over the internode protocol, and every node with that mode needs --internode-bind-addr. If a node binds to a wildcard address such as 0.0.0.0, set --conn-info to an address that the process node can reach.
• Create triggers on the process node and pin them to it: Set --node-spec nodes:<process-node-id> on every trigger.
• Send Processing Engine requests to the process node: Send trigger creation, package installation, and request trigger calls (/api/v3/engine/<path>) to the process node. Other nodes return 405 Current node mode does not use the processing engine.

Standalone process node (recommended)

A standalone process node runs only in process mode. It doesn’t store writes or answer data queries. Use this architecture for every cluster.

• Start the process node with --mode=process and --plugin-dir.
• Start every ingest node and every query node with --internode-bind-addr.
• Use InfluxDB 3 Enterprise 3.12.0 or later. On earlier versions, WAL triggers on a standalone process node never run.

influxdb3 serve \
  --plugin-dir=/path/to/plugins \
  --mode=process \
  --node-id=process0 \
  --cluster-id=prod-cluster

For the complete setup, see Configure a standalone process node.

Mixed process node (not recommended)

A mixed process node runs process mode together with one or more of ingest, query, or compact.

If you run a mixed process node, follow the requirements for every process node, and:

1. Add process to the node’s --mode list and set --plugin-dir.
2. If the node doesn’t run query, restart every query node with --internode-bind-addr.
3. If the node doesn’t run ingest, restart every ingest node with --internode-bind-addr.
4. Create each trigger on the mixed process node, and set --node-spec to its node ID.

Multi-mode configurations

Some deployments benefit from nodes handling multiple responsibilities.

Ingest + Query node (48 cores)

influxdb3 \
  --num-io-threads=12 \
  serve \
  --num-cores=48 \
  --datafusion-num-threads=36 \
  --exec-mem-pool-size=75% \
  --mode=ingest,query \
  --node-id=hybrid-01

Query + Compact node (32 cores)

influxdb3 \
  --num-io-threads=4 \
  serve \
  --num-cores=32 \
  --datafusion-num-threads=28 \
  --mode=query,compact \
  --node-id=qc-01

Cluster architecture examples

Small cluster (3 nodes)

# Node 1: Ingest, query, and compactor
mode: ingest,query,compact
cores: 32
io_threads: 8
datafusion_threads: 24
---
# Node 2: Ingest and query (no compaction)
mode: ingest,query
cores: 32
io_threads: 8
datafusion_threads: 24
---
# Node 3: Ingest and query (no compaction)
mode: ingest,query
cores: 32
io_threads: 8
datafusion_threads: 24

Medium cluster (6 nodes)

# Nodes 1-2: Ingesters
mode: ingest
cores: 48
io_threads: 16
datafusion_threads: 32
---
# Nodes 3-4: Query nodes
mode: query
cores: 48
io_threads: 4
datafusion_threads: 44
---
# Node 5: Compactor (only one compactor per cluster)
mode: compact
cores: 32
io_threads: 4
datafusion_threads: 28
---
# Node 6: Process node
mode: process
cores: 32
io_threads: 4
datafusion_threads: 28

Large cluster (10+ nodes)

# Nodes 1-4: High-throughput ingesters
mode: ingest
cores: 96
io_threads: 20
datafusion_threads: 76
---
# Nodes 5-8: Query nodes
mode: query
cores: 64
io_threads: 4
datafusion_threads: 60
---
# Node 9: Dedicated compactor (only one compactor per cluster)
mode: compact
cores: 32
io_threads: 2
datafusion_threads: 30
---
# Node 10: Process node
mode: process
cores: 32
io_threads: 6
datafusion_threads: 26

Scale your cluster

Vertical scaling limitations

InfluxDB 3 Enterprise uses a shared-nothing architecture where ingest nodes handle all writes. To maximize ingest performance:

• Scale IO threads with concurrent writers: Each concurrent writer can utilize approximately one IO thread for line protocol parsing
• Use high-core machines: Line protocol parsing is CPU-intensive and benefits from more cores
• Deploy multiple ingest nodes: Run several ingest nodes behind a load balancer to distribute write load
• Optimize batch sizes: Configure clients to send larger batches to reduce per-request overhead

Scale queries horizontally

Query nodes can scale horizontally since they all access the same object store:

# Add query nodes as needed
for i in {1..10}; do
  influxdb3 \
    --num-io-threads=4 \
    serve \
    --num-cores=32 \
    --datafusion-num-threads=28 \
    --mode=query \
    --node-id=query-$i &
done

Monitor performance

Node-specific metrics

Monitor specialized nodes differently based on their role:

Ingest nodes

-- Monitor write activity through parquet file creation
SELECT
  table_name,
  count(*) as files_created,
  sum(row_count) as total_rows,
  sum(size_bytes) as total_bytes
FROM system.parquet_files
WHERE max_time > extract(epoch from now() - INTERVAL '5 minutes') * 1000000000
GROUP BY table_name;

Query nodes

-- Monitor query performance
SELECT
  count(*) as query_count,
  avg(execute_duration) as avg_execute_time,
  max(max_memory) as max_memory_bytes
FROM system.queries
WHERE issue_time > now() - INTERVAL '5 minutes'
  AND success = true;

Compactor nodes

-- Monitor compaction progress
SELECT
  event_type,
  event_status,
  count(*) as event_count,
  avg(event_duration) as avg_duration
FROM system.compaction_events
WHERE event_time > now() - INTERVAL '1 hour'
GROUP BY event_type, event_status
ORDER BY event_count DESC;

Monitor cluster-wide metrics

# Check node health via HTTP endpoints
for node in ingester-01:8181 query-01:8181 compactor-01:8181; do
  echo "Node: $node"
  curl -s "http://$node/health"
done

# Monitor metrics from each node
for node in ingester-01:8181 query-01:8181 compactor-01:8181; do
  echo "=== Metrics from $node ==="
  curl -s "http://$node/metrics" | grep -E "(cpu_usage|memory_usage|http_requests_total)"
done

# Query system tables for cluster-wide monitoring
curl -X POST "http://query-01:8181/api/v3/query_sql" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "q": "SELECT * FROM system.queries WHERE issue_time > now() - INTERVAL '\''5 minutes'\'' ORDER BY issue_time DESC LIMIT 10",
    "db": "sensors"
  }'

Monitor and respond to performance issues

Use the monitoring queries to identify the following patterns and their solutions:

High CPU with low throughput (Ingest nodes)

Detection query:

-- Check for high failed query rate indicating parsing issues
SELECT
  count(*) as total_queries,
  sum(CASE WHEN success = true THEN 1 ELSE 0 END) as successful_queries,
  sum(CASE WHEN success = false THEN 1 ELSE 0 END) as failed_queries
FROM system.queries
WHERE issue_time > now() - INTERVAL '5 minutes';

Symptoms:

• Only 2 CPU cores at 100% on large machines
• High write latency despite available resources
• Failed queries due to parsing timeouts

Solution: Increase IO threads (see Ingest node issues)

Memory pressure alerts (Query nodes)

Detection query:

-- Monitor queries with high memory usage or failures
SELECT
  avg(max_memory) as avg_memory_bytes,
  max(max_memory) as peak_memory_bytes,
  sum(CASE WHEN success = false THEN 1 ELSE 0 END) as failed_queries
FROM system.queries
WHERE issue_time > now() - INTERVAL '5 minutes'
  AND query_type = 'sql';

Symptoms:

• Queries failing with out-of-memory errors
• High memory usage approaching pool limits
• Slow query execution times

Solution: Increase memory pool or optimize queries (see Query node issues)

Compaction falling behind (Compactor nodes)

Detection query:

-- Check compaction event frequency and success rate
SELECT
  event_type,
  count(*) as event_count,
  sum(CASE WHEN event_status = 'success' THEN 1 ELSE 0 END) as successful_events
FROM system.compaction_events
WHERE event_time > now() - INTERVAL '1 hour'
GROUP BY event_type;

Symptoms:

• Decreasing compaction event frequency
• Growing number of small Parquet files
• Increasing query times due to file fragmentation

Solution: For nodes using the Parquet-backed storage engine, increase DataFusion threads on your single compactor node (see Compactor node issues).

The upgraded storage engine does not use DataFusion for compaction—refer to the storage engine configuration reference for tuning guidance.

Troubleshoot node configurations

Ingest node issues

Problem: Low throughput despite available CPU

# Check: Are only 2 cores busy?
top -H -p $(pgrep influxdb3)

# Solution: Increase IO threads
--num-io-threads=16

Problem: Data snapshot creation affecting ingest

# Check: DataFusion threads at 100% during data snapshots to Parquet
# Solution: Reserve more DataFusion threads for snapshot operations
--datafusion-num-threads=40

Query node issues

Problem: Slow queries despite resources

# Check: Memory pressure
free -h

# Solution: Increase memory pool
--exec-mem-pool-size=90%

Problem: Poor cache hit rates

# Solution: Increase Parquet cache
--file-cache-size=10GB

Compactor node issues

Problem: Compaction falling behind

# Check: Compaction queue length
# Solution: Increase threads on the single compactor node (only one compactor is allowed per cluster)
--datafusion-num-threads=30

Best practices

1. Start with monitoring: Understand bottlenecks before specializing nodes
2. Test mode combinations: Some workloads benefit from multi-mode nodes
3. Plan for failure: Ensure redundancy in critical node types
4. Document your topology: Keep clear records of node configurations
5. Regular rebalancing: Adjust thread allocation as workloads evolve
6. Capacity planning: Monitor trends and scale proactively

Migrate to specialized nodes

From single-node to specialized cluster

When scaling a single all node cluster to a multi-node cluster:

• Replace the all node with nodes that have explicit, specialized modes
• Assign compact mode to exactly one node that uses the same node-id as the all node being replaced

# Phase 1: Single-node deployment
influxdb3 serve \
  --node-id=node0 \
  --cluster-id=my-cluster \
  --mode=all \
  --num-io-threads=8

# Phase 2: Scale to multi-node cluster
# Stop the all-mode node and start specialized nodes.
# The compact node MUST use the same --node-id as the replaced all-mode node.

# Compact node: reuses the same node-id as the replaced all-mode node
influxdb3 serve \
  --node-id=node0 \
  --cluster-id=my-cluster \
  --mode=compact

# Ingest and query node
influxdb3 serve \
  --node-id=node1 \
  --cluster-id=my-cluster \
  --mode=ingest,query \
  --num-io-threads=8

# Phase 3: Full specialization (optional)
# Dedicated ingest node
influxdb3 serve \
  --node-id=node1 \
  --cluster-id=my-cluster \
  --mode=ingest \
  --num-io-threads=16

# Dedicated query node
influxdb3 serve \
  --node-id=node2 \
  --cluster-id=my-cluster \
  --mode=query \
  --num-io-threads=4

Manage configurations

Configure using environment variables

# Set environment variables for node type
export INFLUXDB3_MODE=ingest
export INFLUXDB3_NUM_IO_THREADS=20
export INFLUXDB3_DATAFUSION_NUM_THREADS=76

influxdb3 serve --node-id=$HOSTNAME --cluster-id=prod