Configure a load balancer for high availability

Open-source HAProxy performs active HTTP health checks and stops routing to a node that fails them. Configure one backend per traffic class:

frontend fe_web
    bind *:8888
    default_backend be_api

frontend fe_heartbeat
    bind *:8000
    default_backend be_heartbeat

backend be_api
    balance roundrobin
    option httpchk
    http-check send meth GET uri /health/ready
    http-check expect status 200
    server node1 
NODE_1_IP
:8888 check inter 3s fall 3 rise 2
    server node2 
NODE_2_IP
:8888 check inter 3s fall 3 rise 2
    server node3 
NODE_3_IP
:8888 check inter 3s fall 3 rise 2

backend be_heartbeat
    balance roundrobin
    option httpchk
    http-check send meth GET uri /health
    http-check expect status 200
    server node1 
NODE_1_IP
:8000 check inter 3s fall 3 rise 2
    server node2 
NODE_2_IP
:8000 check inter 3s fall 3 rise 2
    server node3 
NODE_3_IP
:8000 check inter 3s fall 3 rise 2

Replace NODE_1_IP, NODE_2_IP, and NODE_3_IP with the addresses of your Telegraf Controller nodes.

Open-source NGINX proxies traffic and passively removes a node after failed responses, but it does not actively probe the health endpoints. For active health checks against /health/ready, use NGINX Plus.

Open-source NGINX (passive checks):

upstream tc_api {
    server 
NODE_1_IP
:8888 max_fails=3 fail_timeout=10s;
    server 
NODE_2_IP
:8888 max_fails=3 fail_timeout=10s;
    server 
NODE_3_IP
:8888 max_fails=3 fail_timeout=10s;
}

upstream tc_heartbeat {
    server 
NODE_1_IP
:8000 max_fails=3 fail_timeout=10s;
    server 
NODE_2_IP
:8000 max_fails=3 fail_timeout=10s;
    server 
NODE_3_IP
:8000 max_fails=3 fail_timeout=10s;
}

server {
    listen 8888;
    location / {
        proxy_pass http://tc_api;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

server {
    listen 8000;
    location / {
        proxy_pass http://tc_heartbeat;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

NGINX Plus (active checks):

upstream tc_api {
    zone tc_api 64k;
    server 
NODE_1_IP
:8888;
    server 
NODE_2_IP
:8888;
    server 
NODE_3_IP
:8888;
}

upstream tc_heartbeat {
    zone tc_heartbeat 64k;
    server 
NODE_1_IP
:8000;
    server 
NODE_2_IP
:8000;
    server 
NODE_3_IP
:8000;
}

match tc_ready {
    status 200;
}

match tc_health {
    status 200;
}

server {
    listen 8888;
    location / {
        proxy_pass http://tc_api;
        health_check uri=/health/ready match=tc_ready interval=3s fails=3 passes=2;
    }
}

server {
    listen 8000;
    location / {
        proxy_pass http://tc_heartbeat;
        health_check uri=/health match=tc_health interval=3s fails=3 passes=2;
    }
}

Replace NODE_1_IP, NODE_2_IP, and NODE_3_IP with the addresses of your Telegraf Controller nodes.

On AWS, create a target group per traffic class and configure each target group’s health check to probe the node’s health endpoint. This applies to both Application Load Balancers (ALB) and Network Load Balancers (NLB).

Create a target group for each traffic class with the AWS CLI:

# Web interface and API target group
aws elbv2 create-target-group \
  --name telegraf-controller-api \
  --protocol HTTP \
  --port 8888 \
  --vpc-id 
VPC_ID
 \
  --health-check-protocol HTTP \
  --health-check-path /health/ready \
  --matcher HttpCode=200

# Agent heartbeat target group
aws elbv2 create-target-group \
  --name telegraf-controller-heartbeat \
  --protocol HTTP \
  --port 8000 \
  --vpc-id 
VPC_ID
 \
  --health-check-protocol HTTP \
  --health-check-path /health \
  --matcher HttpCode=200

Replace VPC_ID with the ID of the VPC that contains your nodes, then register each node with both target groups.

Other cloud load balancers follow the same pattern. Configure the backend service or health probe to send an HTTP GET to each node’s health endpoint and treat only 200 as healthy:

• Web interface and API: HTTP health check on the API port (8888) at path /health/ready, expecting 200.
• Agent heartbeat: HTTP health check on the heartbeat port (8000) at path /health, expecting 200.

For example, use a Google Cloud health check or an Azure Load Balancer health probe with these settings. Consult your provider’s documentation for the exact field names.