Kapacitor’s alert system follows a publish subscribe design pattern.
Alerts are published to a
handlers subscribe to a topic.
This example will walk you through setting up a simple cpu threshold alert that sends alerts to Slack.
It is expected that you have a working Telegraf and Kapacitor install to walk through this example. If you do not please take a second to setup both.
We are going to demonstrate how to setup a
cpu alert topic and send alerts to that topic.
First let’s define our simple cpu alert.
stream |from() .measurement('cpu') .groupBy(*) |alert() .warn(lambda: "usage_idle" < 20) .crit(lambda: "usage_idle" < 10) // Send alerts to the `cpu` topic .topic('cpu')
The above TICKscript creates a threshold alert for cpu usage and sends the alerts to the
Save the above script as
Create and start the task by running the following commands:
$ kapacitor define cpu_alert -type stream -tick cpu_alert.tick -dbrp telegraf.autogen $ kapacitor enable cpu_alert
The Slack handler
At this point we have a Kapacitor task which is generating alerts and sending them to the
cpu topic, but since the topic does not have any handlers nothing happens with the alerts.
We can confirm there are no handlers by checking the topic:
$ kapacitor show-topic cpu
The output should look something like:
ID: cpu Level: OK Collected: 27 Handlers:  Events: Event Level Message Date cpu:cpu=cpu3,host=localhost OK cpu:cpu=cpu3,host=localhost is OK 23 Jan 17 14:04 MST
NOTE: Topics are created only when needed, as such if the task has not triggered an alert yet, the topic will not exist. If you get an error about the topic not existing, cause an alert to be triggered. Either change the thresholds on the task or create some cpu load.
To configure a handler we must first define the handler. A handler definition has a few parts:
- ID - The unique ID of the handler.
- Kind - The kind of handler, in this case it will be a
- Match - A lambda expression to filter matching alerts. By default all alerts match.
- Options - A map of values, differs by kind.
The slack handler can be defined as either yaml or json, here we use yaml:
kind: slack options: channel: '#alerts'
The above handler definition defines a handler that sends alerts to the slack channel
Save the above text as
Now we can define our new handler via the
To do this we use the
define-topic-handler command which takes three arguments.
$ kapacitor define-topic-handler Usage: kapacitor define-topic-handler <topic id> <handler id> <path to handler spec file>
$ kapacitor define-topic-handler cpu slack ./slack.yaml
Validate the handler was defined as expected:
$ kapacitor show-topic-handler cpu slack
Finally confirm the topic is configured as expected:
$ kapacitor show-topic slack
The output should look something like:
ID: cpu Level: OK Collected: 27 Handlers: [slack] Events: Event Level Message Date cpu:cpu=cpu3,host=localhost OK cpu:cpu=cpu3,host=localhost is OK 23 Jan 17 14:04 MST
We are done, future alerts triggered by the
cpu_alert task will be send to Slack via the
While it is simple to define alert handlers directly in the TICkscript it can become burdensome once you have many tasks. Using topics decouples the definition of the alert from the handling of the alert. Now to change the slack channel is a single API call to update the slack handler and no TICKscripts have to change.
Topics can be chained together using the
This allows you to further group your alerts into various topics.
For example the above task could be modified to send alerts to the
system topic instead of the
This way all system related alerts can be handled in a consitent manner.
The new TICKscript:
stream |from() .measurement('cpu') .groupBy(*) |alert() .warn(lambda: "usage_idle" < 20) .crit(lambda: "usage_idle" < 10) // Send alerts to the `system` topic .topic('system')
To send all system alerts to Slack, create a new handler for the system topic.
kind: publish options: topics: - ops_team
kapacitor define-topic-handler system publish-to-ops_team ./publish-to-ops_team.yaml
Since the operations team has a on-call rotation you can setup handling of alerts on the
ops_team topic accordingly.
kind: victorops options: routing-key: ops_team
kapacitor define-topic-handler ops_team victorops ./victorops.yaml
system related alerts get sent to the
ops_team topic which in turn get handled in Victor Ops.
Match conditions can be applied to handlers. Only alerts matching the conditions will be handled by that handler.
For example it is typical to only send Slack messages when alerts change state instead of every time an alert is evaluated. Modifing the slack handler definition from the first example we get:
kind: slack match: changed() == TRUE options: channel: '#alerts'
Now update the handler and only alerts that changed state will be sent to Slack.
kapacitor define-topic-handler cpu slack ./slack.yaml