AlertNode
The alert
node triggers an event of varying severity levels and passes the
event to event handlers. The criteria for
triggering an alert is specified via a lambda expression.
See AlertNode.Info,
AlertNode.Warn,
and AlertNode.Crit below.
Constructor
Chaining method | Description |
---|---|
alert ( ) | Create an alert node, which can trigger alerts. To dynamically construct a custom HTTP body or URL, use an alert template. For example, httpPost('localhost/?host={{ index .Tags "host"}}&cpu={{ index .Tags "cpu" }}') . |
Property methods
Setters | description |
---|---|
alerta ( ) | Send the alert to Alerta. |
all ( ) | Indicates an alert should trigger only if all points in a batch match the criteria. Does not apply to stream alerts. |
bigPanda ( ) | Send the alert to BigPanda. |
crit ( value ast.LambdaNode ) |
Filter expression for the CRITICAL alert level. An empty value indicates the level is invalid and is skipped. |
critReset ( value ast.LambdaNode ) |
Filter expression for resetting the CRITICAL alert level to lower level. |
details ( value string ) |
Template for constructing a detailed HTML message for the alert. The same template data is available as the AlertNode.Message property, in addition to a Message field that contains the rendered Message value. |
discord ( ) | Send the alert to Discord. |
durationField ( value string ) |
Optional field key to add the alert duration to the data. The duration is always in units of nanoseconds. |
email ( to ...string ) |
Email the alert data. |
exec ( executable string , args ...string ) |
Execute a command whenever an alert is triggered and pass the alert data over STDIN in JSON format. |
flapping ( low float64 , high float64 ) |
Perform flap detection on the alerts. The method used is similar method to Nagios: https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/flapping.html |
hipChat ( ) | Send the alert to HipChat. |
history ( value int64 ) |
Number of previous states to remember when computing flapping levels and checking for state changes. Minimum value is 2 in order to keep track of current and previous states. |
id ( value string ) |
Template for constructing a unique ID for a given alert. |
idField ( value string ) |
Optional field key to add to the data, containing the alert ID as a string. |
idTag ( value string ) |
Optional tag key to use when tagging the data with the alert ID. |
info ( value ast.LambdaNode ) |
Filter expression for the INFO alert level. An empty value indicates the level is invalid and is skipped. |
inhibit ( category string , equalTags ...string ) |
Inhibit other alerts in a category. The equal tags provides a list of tags that must be equal in order for an alert event to be inhibited. |
infoReset ( value ast.LambdaNode ) |
Filter expression for resetting the INFO alert level to lower level. |
kafka ( ) | Send the alert to a Kafka cluster. |
levelField ( value string ) |
Optional field key to add to the data, containing the alert level as a string. |
levelTag ( value string ) |
Optional tag key to use when tagging the data with the alert level. |
log ( filepath string ) |
Log JSON alert data to file. One event per line. Must specify the absolute path to the log file. It will be created if it does not exist. Example: stream |
message ( value string ) |
Template for constructing a meaningful message for the alert. |
messageField ( value string ) |
Optional field key to add to the data, containing the alert message. |
mqtt ( topic string ) |
Send alert to an MQTT broker |
noRecoveries ( ) | Do not send recovery alerts. |
opsGenie ( ) | Send alert to OpsGenie using OpsGenie’s v1 API. (Deprecated) |
opsGenie2 ( ) | Send alert to OpsGenie using OpsGenie’s v2 API. |
pagerDuty ( ) | Send the alert to PagerDuty using PagerDuty’s v1 API. (Deprecated) |
pagerDuty2 ( ) | Send the alert to PagerDuty using PagerDuty’s v2 API. |
post ( urls ...string ) |
HTTP POST JSON alert data to a specified URL. |
pushover ( ) | Send the alert to Pushover. |
quiet ( ) | Suppresses all error logging events from this node. |
sensu ( ) | Send the alert to Sensu. |
slack ( ) | Send the alert to Slack. |
snmpTrap ( trapOid string ) |
Send the alert using SNMP traps. |
stateChangesOnly ( maxInterval ...time.Duration ) |
Only sends events where the state changed. Each different alert level OK, INFO, WARNING, and CRITICAL are considered different states. |
talk ( ) | Send the alert to Talk. |
tcp ( address string ) |
Send JSON alert data to a specified address over TCP. |
telegram ( ) | Send the alert to Telegram. |
topic ( value string ) |
Topic specifies the name of an alert topic to which, alerts will be published. Alert handlers can be configured per topic, see the API documentation. |
victorOps ( ) | Send alert to VictorOps. |
warn ( value ast.LambdaNode ) |
Filter expression for the WARNING alert level. An empty value indicates the level is invalid and is skipped. |
warnReset ( value ast.LambdaNode ) |
Filter expression for resetting the WARNING alert level to lower level. |
Chaining methods
Alert, Barrier, Bottom, ChangeDetect, Combine, Count, CumulativeSum, Deadman, Default, Delete, Derivative, Difference, Distinct, Ec2Autoscale, Elapsed, Eval, First, Flatten, GroupBy, HoltWinters, HoltWintersWithFit, HttpOut, HttpPost, InfluxDBOut, Join, K8sAutoscale, KapacitorLoopback, Last, Max, Mean, Median, Min, Mode, MovingAverage, Percentile, Sample, Shift, Sideload, Spread, StateCount, StateDuration, Stats, Stddev, Sum, SwarmAutoscale, Top, Trickle, Union, Where, Window
Available event handlers
Different event handlers can be configured for each AlertNode.
Some handlers like Email, HipChat, Sensu, Slack, OpsGenie, VictorOps, PagerDuty, Telegram and Talk have a configuration
option, global
, that indicates that all alerts implicitly use the handler.
Handler | Description |
---|---|
Alerta | Post alert message to Alerta. |
BigPanda | Post alert message to BigPanda. |
Discord | Post alert message to Discord channel. |
Send and email with alert data. | |
exec | Execute a command passing alert data over STDIN. |
HipChat | Post alert message to HipChat room. |
Kafka | Send alert to a Apache Kafka cluster. |
log | Log alert data to file. |
MQTT | Post alert message to MQTT. |
OpsGenie v1 | Send alert to OpsGenie using their v1 API. (Deprecated) |
OpsGenie v2 | Send alert to OpsGenie using their v2 API. |
PagerDuty v1 | Send alert to PagerDuty using their v1 API. (Deprecated) |
PagerDuty v2 | Send alert to PagerDuty using their v2 API. |
post | HTTP POST data to a specified URL. |
Pushover | Send alert to Pushover. |
Sensu | Post alert message to Sensu client. |
Slack | Post alert message to Slack channel. |
SNMPTrap | Trigger SNMP traps. |
Talk | Post alert message to Talk client. |
tcp | Send data to a specified address via raw TCP. |
Telegram | Post alert message to Telegram client. |
VictorOps | Send alert to VictorOps. |
Alert event data
Each event that gets sent to a handler contains the following alert data:
Name | Description |
---|---|
ID | The ID of the alert, user defined. |
Message | The alert message, user defined. |
Details | The alert details, user defined HTML content. |
Time | The time the alert occurred. |
Duration | The duration of the alert in nanoseconds. |
Level | One of OK, INFO, WARNING or CRITICAL. |
Data | influxql.Result containing the data that triggered the alert. |
Recoverable | Indicates whether the alert is auto-recoverable. Determined by the .noRecoveries() property. |
Events are sent to handlers if the alert is in a state other than ‘OK’ or the alert just changed to the ‘OK’ state from a non ‘OK’ state (a.k.a. the alert recovered). Using the AlertNode.StateChangesOnly property events will only be sent to handlers if the alert changed state.
It is valid to configure multiple alert handlers, even with the same type.
Example:
stream
.groupBy('service')
|alert()
.id('kapacitor/{{ index .Tags "service" }}')
.message('{{ .ID }} is {{ .Level }} value:{{ index .Fields "value" }}')
.info(lambda: "value" > 10)
.warn(lambda: "value" > 20)
.crit(lambda: "value" > 30)
.post("http://example.com/api/alert")
.post("http://another.example.com/api/alert")
.tcp("exampleendpoint.com:5678")
.email('oncall@example.com')
Each expression maintains its own state. The order of execution for the expressions is not considered to be deterministic. For each point an expression may or may not be evaluated. If no expression is true then the alert is considered to be in the OK state.
Kapacitor supports alert reset expressions. This way when an alert enters a state, it can only be lowered in severity if its reset expression evaluates to true.
Example:
stream
|from()
.measurement('cpu')
.where(lambda: "host" == 'serverA')
.groupBy('host')
|alert()
.info(lambda: "value" > 60)
.infoReset(lambda: "value" < 50)
.warn(lambda: "value" > 70)
.warnReset(lambda: "value" < 60)
.crit(lambda: "value" > 80)
.critReset(lambda: "value" < 70)
For example, given the following values, the corresponding alert states are:
Value | Alert State |
---|---|
61 | INFO |
73 | WARNING |
64 | WARNING |
85 | CRITICAL |
62 | INFO |
56 | INFO |
47 | OK |
Available Statistics:
alerts_triggered
: Total number of alerts triggeredoks_triggered
: Number of OK alerts triggeredinfos_triggered
: Number of Info alerts triggeredwarns_triggered
: Number of Warn alerts triggeredcrits_triggered
: Number of Crit alerts triggered
Properties
Property methods modify state on the calling node.
They do not add another node to the pipeline, and always return a reference to the calling node.
Property methods are marked using the .
operator.
Alerta
Send the alert to Alerta. Detailed configuration options and setup instructions are provided in the Alerta Event Handler article.
Example kapacitor.conf
[alerta]
enabled = true
url = "https://alerta.yourdomain"
token = "9hiWoDOZ9IbmHsOTeST123ABciWTIqXQVFDo63h9"
environment = "Production"
origin = "Kapacitor"
Example TICKscript
stream
|alert()
.alerta()
.resource('Hostname or service')
.event('Something went wrong')
All
Indicates an alert should trigger only if all points in a batch match the criteria. Does not apply to stream alerts.
alert.all()
BigPanda
Send the alert to BigPanda. Detailed configuration options and setup instructions are provided in the BigPanda Event Handler article.
Example kapacitor.conf
[bigpanda]
enabled = true
url = "https://api.bigpanda.io/data/v2/alerts"
token = "BigPanda-API-auth-bearer-token"
app-key = "BigPanda-integration-app-key"
Example TICKscript
stream
|alert()
.bigPanda()
.appKey('fc39458f98e91eb0310258c3b725d643')
.primaryProperty('device')
.secondaryProperty('sensor_name')
.topic('bigpanda-topic')
Category
Category places this alert in a named category. Categories are used to inhibit alerts.
alert.category(value string)
Crit
Filter expression for the CRITICAL alert level. An empty value indicates the level is invalid and is skipped.
alert.crit(value ast.LambdaNode)
CritReset
Filter expression for resetting the CRITICAL alert level to lower level.
alert.critReset(value ast.LambdaNode)
Details
Template for constructing a detailed HTML message for the alert. The same template data is available as the AlertNode.Message property, in addition to a Message field that contains the rendered Message value.
The intent is that the Message property be a single line summary while the Details property is a more detailed message possibly spanning multiple lines, and containing HTML formatting.
This template is rendered using the html/template package in Go so safe and valid HTML is generated.
The json
method is available within the template to convert any variable to a valid
JSON string.
xExample:
|alert()
.id('{{ .Name }}')
.details('''
<h1>{{ .ID }}</h1>
<b>{{ .Message }}</b>
Value: {{ index .Fields "value" }}
''')
.email()
Default: {{ json . }}
alert.details(value string)
Discord
Send the alert to Discord. Detailed configuration options and setup instructions are provided in the Discord Event Handler article.
Example kapacitor.conf
[discord]
enabled = true
url = "https://discordapp.com/api/webhooks/xxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
Example TICKscript
stream
|alert()
.discord()
.embedTitle('Alert!')
DurationField
Optional field key to add the alert duration to the data. The duration is always in units of nanoseconds.
alert.durationField(value string)
Email the alert data to specified “To” email addresses. Detailed configuration options and setup instructions are provided in the Email Event Handler article.
Example kapacitor.conf
[smtp]
enabled = true
host = "localhost"
port = 25
username = "xxxx"
password = "xxxx"
from = "kapacitor@example.com"
to = ["oncall@example.com"]
global = true
state-changes-only = true
Example TICKscript
|alert()
.id('{{ .Name }}')
// Email subject
.message('{{ .ID }}:{{ .Level }}')
// Email body as HTML
.details('''
<h1>{{ .ID }}</h1>
<b>{{ .Message }}</b>
Value: {{ index .Fields "value" }}
''')
.email()
Exec
Execute a command whenever an alert is triggered and pass the alert data over STDIN in JSON format. Detailed usage instructions and examples are provided in the Exec Event Handler article.
// Pattern
alert.exec(executable string, args ...string)
// Example
alert.exec('/usr/bin/python', 'myscript.py')
Flapping
Perform flap detection on the alerts. The method used is similar method to Nagios flapping
Each different alerting level is considered a different state.
The low and high thresholds are inverted thresholds of a percentage of state changes.
Meaning that if the percentage of state changes goes above the high
threshold, the alert enters a flapping state. The alert remains in the flapping state
until the percentage of state changes goes below the low
threshold.
Typical values are low: 0.25 and high: 0.5. The percentage values represent the number state changes
over the total possible number of state changes. A percentage change of 0.5 means that the alert changed
state in half of the recorded history, and remained the same in the other half of the history.
// Pattern
alert.flapping(low float64, high float64)
// Example
alert.flapping(0.25, 0.5)
Kafka
Send the alert to an Apache Kafka cluster. Detailed configuration options and setup instructions are provided in the Kafka Event Handler article.
Example kapacitor.conf
[[kafka]]
enabled = true
id = "localhost"
timeout = 10s
Example TICKscript
stream
|alert()
.kafka()
.cluster('kafka-cluster')
.kafkaTopic('kafka-topic-name')
History
Number of previous states to remember when computing flapping levels and checking for state changes. Minimum value is 2 in order to keep track of current and previous states.
Default: 21
// Pattern
alert.history(value int64)
// Example
alert.history(21)
HipChat
Send the alert to HipChat. Detailed configuration options and setup instructions are provided in the HipChat Event Handler article.
Example kapacitor.conf
[hipchat]
enabled = true
url = "https://orgname.hipchat.com/v2/room"
room = "4189212"
token = "9hiWoDOZ9IbmHsOTeST123ABciWTIqXQVFDo63h9"
Example TICKscript
stream
|alert()
.hipChat()
.room('Kapacitor')
Id
Template for constructing a unique ID for a given alert.
Available template data
Data Name | Description |
---|---|
Name | Measurement name. |
TaskName | The name of the task |
Group | Concatenation of all group-by tags of the form [key=value,]+ . If no groupBy is performed equal to literal ’nil’. |
Tags | Map of tags. Use `{{ index .Tags “key” }}`` to get a specific tag value. |
ServerInfo | Information about the running server. Available nested fields are Hostname , ClusterID and ServerID . |
Default: {{ .Name }}:{{ .Group }}
// Pattern
alert.id(value string)
// Example
alert.id('{{ .Name }}:{{ .Group }}')
Example: ID template using name and group
stream
|from()
.measurement('cpu')
.groupBy('cpu')
|alert()
.id('kapacitor/{{ .Name }}/{{ .Group }}')
Resulting ID: kapacitor/cpu/cpu=cpu0
Example: ID template using a tag
stream
|from()
.measurement('cpu')
.groupBy('service')
|alert()
.id('kapacitor/{{ index .Tags "service" }}')
Resulting ID: kapacitor/authentication
Example: ID template using multiple tags
stream
|from()
.measurement('cpu')
.groupBy('service', 'host')
|alert()
.id('kapacitor/{{ index .Tags "service" }}/{{ index .Tags "host" }}')
Resulting ID: kapacitor/authentication/auth001.example.com
IdField
Optional field key to add to the data, containing the alert ID as a string.
// Pattern
alert.idField(value string)
// Example
alert.idField('id')
IdTag
Optional tag key to use when tagging the data with the alert ID.
// Pattern
alert.idTag(value string)
// Example
alert.idTag('alertID')
Info
Filter expression for the INFO alert level. An empty value indicates the level is invalid and is skipped.
// Pattern
alert.info(value ast.LambdaNode)
// Example
alert.info(lambda: 'usage_idle' < 60)
InfoReset
Filter expression for resetting the INFO alert level to lower level.
// Pattern
alert.infoReset(value ast.LambdaNode)
// Example
alert.infoReset(lambda: 'usage_idle' > 60)
Inhibit
Inhibit other alerts in a category. The equal tags provides a list of tags that must be equal in order for an alert event to be inhibited.
The following two TICKscripts demonstrate how to use the inhibit feature:
//cpu_alert.tick
stream
|from()
.measurement('cpu')
.groupBy('host')
|alert()
.category('system_alerts')
.crit(lambda: "usage_idle" < 10.0)
//host_alert.tick
stream
|from()
.measurement('uptime')
.groupBy('host')
|deadman(0.0, 1m)
.inhibit('system_alerts', 'host')
The deadman is a type of alert node and can be used to inhibit all alerts in the system_alerts
category when triggered.
The host
argument to the inhibit function says that the host tag must be equal between the CPU alert and the host alert in order for it to be inhibited.
This has the effect of the deadman alerts only inhibiting CPU alerts for hosts that are currently dead.
alert.inhibit(category string, equalTags ...string)
Kafka
Send the alert to a Kafka topic. Detailed setup and usage instructions are provided in the Kafka Event Handler article.
Example: kapacitor.conf
[[kafka]]
enabled = true
id = "default"
brokers = ["localhost:9092"]
Example: TICKscript
stream
|alert()
.kafka()
.cluster('default')
.kafkaTopic('alerts')
LevelField
Optional field key to add to the data, containing the alert level as a string.
// Pattern
alert.levelField(value string)
// Example
alert.levelField('INFO')
LevelTag
Optional tag key to use when tagging the data with the alert level.
// Pattern
alert.levelTag(value string)
// Example
alert.levelTag('level')
Log
Log JSON alert data to file. Detailed setup and usage instructions are provided in the Log Event Handler article.
Example TICKscript
stream
|alert()
.log('/tmp/alert')
.mode(0644)
Message
Template for constructing a meaningful message for the alert.
Available template data
Data Name | Description |
---|---|
ID | The ID of the alert. |
Name | Measurement name. |
TaskName | The name of the task |
Group | Concatenation of all group-by tags of the form 1. If no groupBy is performed equal to literal ’nil’. |
Tags | Map of tags. Use {{ index .Tags "key" }} to get a specific tag value. |
Level | Alert Level - INFO , WARNING , CRITICAL . |
Fields | Map of fields. Use {{ index .Fields "key" }} to get a specific field value. |
Time | The time of the point that triggered the event. Default format is YYYY-MM-DD 00:00:00 +0000 UTC |
Duration | The duration of the alert. |
Default: {{ .ID }} is {{ .Level }}
Example:
stream
|from()
.measurement('cpu')
.groupBy('service', 'host')
|alert()
.id('{{ index .Tags "service" }}/{{ index .Tags "host" }}')
.message('{{ .ID }} is {{ .Level}} value: {{ index .Fields "value" }}')
Resulting Message: authentication/auth001.example.com is CRITICAL value:42
MessageField
Optional field key to add to the data containing the alert message.
// Pattern
alert.messageField(value string)
// Example
alert.messageField('message')
MQTT
Send alert to an MQTT broker. Detailed configuration options and usage instructions are provided in the MQTT Event Handler article.
Example kapacitor.conf
[[mqtt]]
enabled = true
name = "localhost"
default = true
url = "tcp://localhost:1883"
client-id = "kapacitor"
username = "myusername"
password = "mysupersecretpassw0rd"
Example TICKscript
stream
|alert()
.mqtt('topic')
.brokerName('localhost')
NoRecoveries
Do not send recovery alerts. Sets recoverable
alert data field to false
.
alert.noRecoveries()
OpsGenie v1
Send alert to OpsGenie using OpsGenie’s v1 API. Detailed configuration options and setup instructions are provided in the OpsGenie v1 Event Handler article.
Example kapacitor.conf
[opsgenie]
enabled = true
api-key = "xxxxx"
teams = ["everyone"]
recipients = ["jim", "bob"]
Example TICKscript
stream
|alert()
.opsGenie()
OpsGenie v2
Send alert to OpsGenie using OpsGenie’s v2 API. Detailed configuration options and setup instructions are provided in the OpsGenie v2 Event Handler article.
Example kapacitor.conf
[opsgenie2]
enabled = true
api-key = "xxxxx"
teams = ["everyone"]
recipients = ["jim", "bob"]
Example TICKscript
stream
|alert()
.opsGenie2()
PagerDuty v1
Send the alert to PagerDuty using PagerDuty’s v1 API. Detailed configuration options and setup instructions are provided in the PagerDuty v1 Event Handler article.
Example kapacitor.conf
[pagerduty]
enabled = true
service-key = "xxxx"
url = "https://events.pagerduty.com/generic/2010-04-15/create_event.json"
Example TICKscript
stream
|alert()
.pagerDuty()
PagerDuty v2
Send the alert to PagerDuty using PagerDuty’s v2 API. Detailed configuration options and setup instructions are provided in the PagerDuty v2 Event Handler article.
Example kapacitor.conf
[pagerduty2]
enabled = true
routing-key = "xxxx"
url = "https://events.pagerduty.com/v2/enqueue"
Example TICKscript
stream
|alert()
.pagerDuty2()
Post
HTTP POST JSON alert data to a specified URL. Detailed configuration options and setup instructions are provided in the Post Event Handler article.
Example TICKscript
stream
|alert()
.post('http://example.com')
.captureResponse()
Pushover
Send the alert to Pushover. Detailed configuration options and setup instructions are provided in the Pushover Event Handler article.
Example kapacitor.conf
[pushover]
enabled = true
token = "9hiWoDOZ9IbmHsOTeST123ABciWTIqXQVFDo63h9"
user_key = "Pushover"
Example TICKscript
stream
|alert()
.pushover()
.sound('siren')
.user_key('other user')
.device('mydev')
.title('mytitle')
.uRL('myurl')
.URLTitle('mytitle')
Quiet
Suppress all error logging events from this node.
alert.quiet()
Sensu
Send the alert to Sensu. Detailed configuration options and setup instructions are provided in the Sensu Event Handler article.
*Example kapacitor.conf
[sensu]
enabled = true
url = "http://sensu:3030"
source = "Kapacitor"
handlers = ["sns","slack"]
Example TICKscript
stream
|alert()
.sensu()
.handlers('sns','slack')
Slack
Send the alert to Slack. Detailed configuration options and setup instructions are provided in the Slack Event Handler article.
Example kapacitor.conf
[slack]
enabled = true
url = "https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxx"
channel = "#general"
Example TICKscript
stream
|alert()
.slack()
.channel('#alerts')
.channel('@jsmith')
SnmpTrap
Send the alert using SNMP traps. Detailed configuration options and setup instructions are provided in the SNMP Trap Event Handler article.
Example kapacitor.conf
[snmptrap]
enabled = true
addr = "127.0.0.1:9162"
community = "public"
Example TICKscript
stream
|alert()
.snmpTrap('1.3.6.1.2.1.1')
.data('1.3.6.1.2.1.1.6', 'i', '{{ index .Field "value" }}')
.data('1.3.6.1.2.1.1.7', 's', '{{ .Message }}')
StateChangesOnly
Only sends events where the state changed. Each different alert level OK, INFO, WARNING, and CRITICAL are considered different states.
Example
stream
|from()
.measurement('cpu')
|window()
.period(10s)
.every(10s)
|alert()
.crit(lambda: "value" > 10)
.stateChangesOnly()
.slack()
If the “value” is greater than 10 for a total of 60s, then only two events will be sent. First, when the value crosses the threshold, and second, when it falls back into an OK state. Without stateChangesOnly, the alert would have triggered 7 times: 6 times for each 10s period where the condition was met and once more for the recovery.
An optional maximum interval duration can be provided. An event will not be ignored (aka trigger an alert) if more than the maximum interval has elapsed since the last alert.
Example
stream
// ...
|alert()
.crit(lambda: "value" > 10)
.stateChangesOnly(10m)
.slack()
TCP
Send JSON alert data to a specified address over TCP. Detailed usage instructions are provided in the TCPEvent Handler article.
// Pattern
alert.tcp(address string)
// Example
alert.tcp('127.0.0.1:7777')
Telegram
Send the alert to Telegram. Detailed configuration options and setup instructions are provided in the Telegram Event Handler article.
Example kapacitor.conf
[telegram]
enabled = true
token = "123456789:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
chat-id = "xxxxxxxxx"
parse-mode = "Markdown"
disable-web-page-preview = true
disable-notification = false
Example TICKscript
stream
|alert()
.telegram()
.chatId('xxxxxxx')
Topic
Topic specifies the name of an alert topic to which alerts will be published. Alert handlers can be configured per topic. See the API documentation.
// Pattern
alert.topic(value string)
// Example
alert.topic('cpu')
VictorOps
Send alert to VictorOps. Detailed configuration options and setup instructions are provided in the VictorOps Event Handler article.
Example kapacitor.conf
[victorops]
enabled = true
api-key = "xxxxx"
routing-key = "everyone"
Example TICKscript
stream
|alert()
.victorOps()
.routingKey('team_rocket')
Warn
Filter expression for the WARNING alert level. An empty value indicates the level is invalid and is skipped.
// Pattern
alert.warn(value ast.LambdaNode)
// Example
alert.warn(lambda: 'usage_idle' < 20)
WarnReset
Filter expression for resetting the WARNING alert level to lower level.
// Pattern
alert.warnReset(value ast.LambdaNode)
// Example
alert.warnReset(lambda: 'usage_idle' > 20)
Chaining Methods
Chaining methods create a new node in the pipeline as a child of the calling node.
They do not modify the calling node.
Chaining methods are marked using the |
operator.
Alert
Create an alert node, which can trigger alerts.
alert|alert()
Returns: AlertNode
Barrier
Create a new Barrier node that emits a BarrierMessage periodically.
One BarrierMessage will be emitted every period duration.
alert|barrier()
Returns: BarrierNode
Bottom
Select the bottom num
points for field
and sort by any extra tags or fields.
alert|bottom(num int64, field string, fieldsAndTags ...string)
Returns: InfluxQLNode
ChangeDetect
Create a new node that only emits new points if different from the previous point.
alert|changeDetect(field string)
Returns: ChangeDetectNode
Combine
Combine this node with itself. The data is combined on timestamp.
alert|combine(expressions ...ast.LambdaNode)
Returns: CombineNode
Count
Count the number of points.
alert|count(field string)
Returns: InfluxQLNode
CumulativeSum
Compute a cumulative sum of each point that is received. A point is emitted for every point collected.
alert|cumulativeSum(field string)
Returns: InfluxQLNode
Deadman
Helper function for creating an alert on low throughput, a.k.a. deadman’s switch.
- Threshold: trigger alert if throughput drops below threshold in points/interval.
- Interval: how often to check the throughput.
- Expressions: optional list of expressions to also evaluate. Useful for time of day alerting.
Example:
var data = stream
|from()...
// Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
data
|deadman(100.0, 10s)
//Do normal processing of data
data...
The above is equivalent to this example:
var data = stream
|from()...
// Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
data
|stats(10s)
.align()
|derivative('emitted')
.unit(10s)
.nonNegative()
|alert()
.id('node \'stream0\' in task \'{{ .TaskName }}\'')
.message('{{ .ID }} is {{ if eq .Level "OK" }}alive{{ else }}dead{{ end }}: {{ index .Fields "emitted" | printf "%0.3f" }} points/10s.')
.crit(lambda: "emitted" <= 100.0)
//Do normal processing of data
data...
The id
and message
alert properties can be configured globally via the ‘deadman’ configuration section.
Since the AlertNode is the last piece it can be further modified as usual. Example:
var data = stream
|from()...
// Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
data
|deadman(100.0, 10s)
.slack()
.channel('#dead_tasks')
//Do normal processing of data
data...
You can specify additional lambda expressions to further constrain when the deadman’s switch is triggered. Example:
var data = stream
|from()...
// Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
// Only trigger the alert if the time of day is between 8am-5pm.
data
|deadman(100.0, 10s, lambda: hour("time") >= 8 AND hour("time") <= 17)
//Do normal processing of data
data...
alert|deadman(threshold float64, interval time.Duration, expr ...ast.LambdaNode)
Returns: AlertNode
Default
Create a node that can set defaults for missing tags or fields.
alert|default()
Returns: DefaultNode
Delete
Create a node that can delete tags or fields.
alert|delete()
Returns: DeleteNode
Derivative
Create a new node that computes the derivative of adjacent points.
alert|derivative(field string)
Returns: DerivativeNode
Difference
Compute the difference between points independent of elapsed time.
alert|difference(field string)
Returns: InfluxQLNode
Distinct
Produce batch of only the distinct points.
alert|distinct(field string)
Returns: InfluxQLNode
Ec2Autoscale
Create a node that can trigger autoscale events for a ec2 autoscalegroup.
alert|ec2Autoscale()
Returns: Ec2AutoscaleNode
Elapsed
Compute the elapsed time between points.
alert|elapsed(field string, unit time.Duration)
Returns: InfluxQLNode
Eval
Create an eval node that will evaluate the given transformation function to each data point. A list of expressions may be provided and will be evaluated in the order they are given. The results are available to later expressions.
alert|eval(expressions ...ast.LambdaNode)
Returns: EvalNode
First
Select the first point.
alert|first(field string)
Returns: InfluxQLNode
Flatten
Flatten points with similar times into a single point.
alert|flatten()
Returns: FlattenNode
GroupBy
Group the data by a set of tags.
Can pass literal * to group by all dimensions. Example:
|groupBy(*)
alert|groupBy(tag ...interface{})
Returns: GroupByNode
HoltWinters
Compute the Holt-Winters (/influxdb/v1/query_language/functions/#holt-winters) forecast of a data set.
alert|holtWinters(field string, h int64, m int64, interval time.Duration)
Returns: InfluxQLNode
HoltWintersWithFit
Compute the Holt-Winters (/influxdb/v1/query_language/functions/#holt-winters) forecast of a data set. This method also outputs all the points used to fit the data in addition to the forecasted data.
alert|holtWintersWithFit(field string, h int64, m int64, interval time.Duration)
Returns: InfluxQLNode
HttpOut
Create an HTTP output node that caches the most recent data it has received.
The cached data is available at the given endpoint.
The endpoint is the relative path from the API endpoint of the running task.
For example, if the task endpoint is at /kapacitor/v1/tasks/<task_id>
and endpoint is
top10
, then the data can be requested from /kapacitor/v1/tasks/<task_id>/top10
.
alert|httpOut(endpoint string)
Returns: HTTPOutNode
HttpPost
Creates an HTTP Post node that POSTS received data to the provided HTTP endpoint. HttpPost expects 0 or 1 arguments. If 0 arguments are provided, you must specify an endpoint property method.
alert|httpPost(url ...string)
Returns: HTTPPostNode
InfluxDBOut
Create an influxdb output node that will store the incoming data into InfluxDB.
alert|influxDBOut()
Returns: InfluxDBOutNode
Join
Join this node with other nodes. The data is joined on timestamp.
alert|join(others ...Node)
Returns: JoinNode
K8sAutoscale
Create a node that can trigger autoscale events for a kubernetes cluster.
alert|k8sAutoscale()
Returns: K8sAutoscaleNode
KapacitorLoopback
Create an kapacitor loopback node that will send data back into Kapacitor as a stream.
alert|kapacitorLoopback()
Returns: KapacitorLoopbackNode
Last
Select the last point.
alert|last(field string)
Returns: InfluxQLNode
Max
Select the maximum point.
alert|max(field string)
Returns: InfluxQLNode
Mean
Compute the mean of the data.
alert|mean(field string)
Returns: InfluxQLNode
Median
Compute the median of the data.
Note: This method is not a selector. If you want the median point, use
.percentile(field, 50.0)
.
alert|median(field string)
Returns: InfluxQLNode
Min
Select the minimum point.
alert|min(field string)
Returns: InfluxQLNode
Mode
Compute the mode of the data.
alert|mode(field string)
Returns: InfluxQLNode
MovingAverage
Compute a moving average of the last window points. No points are emitted until the window is full.
alert|movingAverage(field string, window int64)
Returns: InfluxQLNode
Percentile
Select a point at the given percentile. This is a selector function, no interpolation between points is performed.
alert|percentile(field string, percentile float64)
Returns: InfluxQLNode
Sample
Create a new node that samples the incoming points or batches.
One point will be emitted every count or duration specified.
alert|sample(rate interface{})
Returns: SampleNode
Shift
Create a new node that shifts the incoming points or batches in time.
alert|shift(shift time.Duration)
Returns: ShiftNode
Sideload
Create a node that can load data from external sources.
alert|sideload()
Returns: SideloadNode
Spread
Compute the difference between min
and max
points.
alert|spread(field string)
Returns: InfluxQLNode
StateCount
Create a node that tracks number of consecutive points in a given state.
alert|stateCount(expression ast.LambdaNode)
Returns: StateCountNode
StateDuration
Create a node that tracks duration in a given state.
alert|stateDuration(expression ast.LambdaNode)
Returns: StateDurationNode
Stats
Create a new stream of data that contains the internal statistics of the node. The interval represents how often to emit the statistics based on real time. This means the interval time is independent of the times of the data points the source node is receiving.
alert|stats(interval time.Duration)
Returns: StatsNode
Stddev
Compute the standard deviation.
alert|stddev(field string)
Returns: InfluxQLNode
Sum
Compute the sum of all values.
alert|sum(field string)
Returns: InfluxQLNode
SwarmAutoscale
Create a node that can trigger autoscale events for a Docker swarm cluster.
alert|swarmAutoscale()
Returns: SwarmAutoscaleNode
Top
Select the top num
points for field
and sort by any extra tags or fields.
alert|top(num int64, field string, fieldsAndTags ...string)
Returns: InfluxQLNode
Trickle
Create a new node that converts batch data to stream data.
alert|trickle()
Returns: TrickleNode
Union
Perform the union of this node and all other given nodes.
alert|union(node ...Node)
Returns: UnionNode
Where
Create a new node that filters the data stream by a given expression.
alert|where(expression ast.LambdaNode)
Returns: WhereNode
Window
Create a new node that windows the stream by time.
NOTE: Window can only be applied to stream edges.
alert|window()
Returns: WindowNode
Was this page helpful?
Thank you for your feedback!
Support and feedback
Thank you for being part of our community! We welcome and encourage your feedback and bug reports for Kapacitor and this documentation. To find support, use the following resources: