Deploy a highly available Telegraf Controller cluster

Deploy multiple Telegraf Controller nodes against a shared PostgreSQL database, put them behind a load balancer, and let one node lead while the others stand by. This guide covers configuring the nodes, routing traffic with the health endpoints, and tuning PostgreSQL for quick failover.

• Prerequisites
• Provision shared PostgreSQL and secrets
• Enable high availability on each node
• Apply the license
• Put the cluster behind a load balancer
• Tune PostgreSQL for fast failover
• Verify leadership and failover

Prerequisites

• A valid Telegraf Enterprise license. You apply the license once; every node reads it from the shared database. See Apply a license.
• A PostgreSQL database that every node can reach over the network. You can use self-managed PostgreSQL or a PostgreSQL-compatible managed service. High availability does not support SQLite. See Requirements and constraints.
• Two or more hosts to run the Telegraf Controller binary.
• A load balancer that can route traffic based on an HTTP health check.

Provision shared PostgreSQL and secrets

Every node connects to the same PostgreSQL database and signs sessions with the same secret.

1. Provision a PostgreSQL database and note its connection string. Connect nodes directly to PostgreSQL, or use a connection pooler in session-pooling mode. Transaction-pooling mode breaks leader election.

2. Generate a single SESSION_SECRET to share across all nodes. Reuse the same value on every node so a session stays valid regardless of which node serves the request.

   openssl rand -hex 32

Enable high availability on each node

On every node, set HA_ENABLED=true, point DATABASE_URL at the shared PostgreSQL database, and set the shared SESSION_SECRET. Start the binary the same way you would a single node. You apply the license separately, once, in the next step.

[Service]
Environment=HA_ENABLED=true
Environment=DATABASE_URL=postgresql://
POSTGRES_USER
:
POSTGRES_PASSWORD
@
POSTGRES_HOST
:5432/telegraf_controller
Environment=SESSION_SECRET=
SHARED_SECRET

sudo systemctl daemon-reload
sudo systemctl restart telegraf-controller

export HA_ENABLED=true
export DATABASE_URL="postgresql://
POSTGRES_USER
:
POSTGRES_PASSWORD
@
POSTGRES_HOST
:5432/telegraf_controller"
export SESSION_SECRET="
SHARED_SECRET
"

telegraf_controller --no-interactive

$env:HA_ENABLED="true"
$env:DATABASE_URL="postgresql://
POSTGRES_USER
:
POSTGRES_PASSWORD
@
POSTGRES_HOST
:5432/telegraf_controller"
$env:SESSION_SECRET="
SHARED_SECRET
"

./telegraf_controller.exe --no-interactive

Replace the following:

• POSTGRES_USER and POSTGRES_PASSWORD: the credentials for the shared PostgreSQL database.
• POSTGRES_HOST: the hostname or address of the shared PostgreSQL database, reachable from every node.
• SHARED_SECRET: the shared session secret generated with openssl rand -hex 32, identical on every node.

Optionally, set HA_POLL_INTERVAL_MS to change how quickly settings, token, and license changes propagate between nodes. It defaults to 5000 (5 seconds).

Apply the license

Apply your Telegraf Enterprise license to one of your Telegraf Controller nodes. Use either method:

• Set LICENSE_FILE_PATH on one node at first startup to seed the license into the shared database.
• Apply the license through the user interface or API after the cluster is running.

Every node reads the license from the shared database and becomes licensed within one poll interval, without a restart. For details, see Apply a license.

After a license is present, one node acquires the leader lock and the rest stand by. Confirm the cluster’s state with the health endpoints.

Put the cluster behind a load balancer

Run the cluster behind a load balancer that health-checks each node and routes around any node that fails. Route each class of traffic using the node’s unauthenticated health endpoints:

• Web interface and API traffic: route to nodes that return 200 from GET /health/ready on the API port (default 8888). If you serve the web interface on a separate ui-port, route that port as its own pool, health-checked with GET /.
• Agent heartbeat traffic: route to nodes that return 200 from GET /health on the heartbeat port (default 8000).

Because sessions are validated with the shared SESSION_SECRET, sticky sessions are not required.

For the full list of health endpoints and example configurations for HAProxy, NGINX, and cloud load balancers such as AWS Elastic Load Balancing, see Configure a load balancer.

Tune PostgreSQL for fast failover

When a leader shuts down gracefully, it releases the advisory lock and a standby takes over almost immediately, typically within a second or two. When a leader fails abruptly (a crash, a killed process, or a lost host), PostgreSQL must first notice that the leader’s connection is gone before another node can acquire the lock. With default operating-system keepalive settings, that can take much longer than a graceful handoff.

To bound failover time, shorten PostgreSQL’s TCP keepalive settings so the server detects dropped connections sooner:

ALTER SYSTEM SET tcp_keepalives_idle = 10;
ALTER SYSTEM SET tcp_keepalives_interval = 5;
ALTER SYSTEM SET tcp_keepalives_count = 3;
SELECT pg_reload_conf();

With settings in this range, failover after an abrupt failure completes in well under a minute. Adjust the values to match your availability requirements and network conditions. Throughout either kind of failover, the surviving nodes keep ingesting agent heartbeats.

Verify leadership and failover

1. Identify the current leader by querying each node’s leader endpoint. Exactly one node returns 200:

   curl -s http://node-1.example.com:8888/health/leader
   curl -s http://node-2.example.com:8888/health/leader

2. Confirm that each licensed node reports ready:

   curl -s http://node-1.example.com:8888/health/ready

3. Test failover by stopping the leader (for example, sudo systemctl stop telegraf-controller on the leader host). Within a few seconds, another node’s GET /health/leader returns 200, and the load balancer continues to serve web interface, API, and heartbeat traffic from the surviving nodes.