Monitor Telegraf Controller

Monitor a Telegraf Controller server with its built-in health endpoints and its log output.

For monitoring the agents themselves, including the fleet summary and per-agent heartbeat data, see Monitor agents.

Health endpoints

Telegraf Controller exposes unauthenticated HTTP health endpoints on the API port and on the heartbeat port. GET /health/live and GET /health/ready on the API port are the general-purpose liveness and readiness probes, and the heartbeat service answers GET /health on its own port. GET /health/leader identifies the leader in a high-availability cluster.

For the full endpoint reference, including status codes, response bodies, and how to choose a probe, see Health endpoints. The endpoints are documented with load balancers in mind, but any monitoring system can poll them.

Heartbeat service health

Telegraf Controller reports the health of its embedded heartbeat service, the listener that receives agent heartbeats, in the web interface and through an authenticated API endpoint. Unlike the health endpoint probes, these report why the service is unhealthy, not only whether it responds.

Health indicator in the web interface

The heartbeat service health indicator appears near the bottom of the left navigation.

Heartbeat service health indicator in the left navigation
StatusIndicator messageAction
OkHeartbeat service is healthy.None.
UnknownHeartbeat service has not reached the database yet.If the state persists after startup, check the server log for database errors.
WarningAgent status check failed. Agent statuses may be outdated.Check the server log for the underlying error.
WarningNo tokens loaded, so every agent heartbeat is rejected. Create an API token with the heartbeat:write permission.Create a token.
ErrorHeartbeat service is not running.Check the server log, then restart Telegraf Controller.
ErrorHeartbeat service cannot connect to the database. Agent heartbeats are rejected and agent statuses are not updated.See Troubleshoot the database.
ErrorToken cache failed to load. Agent heartbeats are rejected with 401.See Agent heartbeats return 401 Invalid token.

Query heartbeat service health with the API

GET /api/heartbeat/health returns the detail behind the indicator. The endpoint requires an API token with read permission on the Heartbeat API.

curl "https://telegraf-controller.example.com/api/heartbeat/health" \
  -H "Authorization: Bearer 
API_TOKEN
"

The response reports:

  • Service: whether the heartbeat service is running (running) and whether heartbeat authentication is enabled (authEnabled).
  • Database: connectivity (dbConnected, null until the service first reaches the database), the last database error, and when it occurred.
  • Token cache: how many API tokens are loaded, the last refresh and last successful refresh times, and the last refresh error.
  • Scheduler: whether the agent status check scheduler is active, its last run and last successful run times, and the last run error.

If the heartbeat endpoint group is listed in disable-auth-endpoints, this endpoint answers without authentication and its response includes database error text. Make sure your firewall restricts who can reach the API port. See Expose the right ports.

Server logs

Telegraf Controller writes log output to the console (standard output and standard error). Where that output ends up depends on how you run the server:

  • Interactive terminal: log output appears in the terminal.
  • systemd: the journal captures console output. Read it with journalctl -u telegraf-controller.
  • LaunchDaemon (macOS): output goes to the file paths configured in the service plist. The LaunchDaemon example uses /var/log/telegraf-controller.log.
  • Windows service: configure your service manager to capture console output to files. See Run Telegraf Controller as a service.

What to watch for

Alert on the following log messages. Each entry links to the page that explains how to fix the problem.


Was this page helpful?

Thank you for your feedback!