Documentation

Kapacitor alerts overview

This page documents an earlier version of Kapacitor. Kapacitor v1.5 is the latest stable version. View this page in the v1.5 documentation.

Kapacitor makes it possible to handle alert messages in two different ways.

  • The messages can be pushed directly to an event handler exposed through the Alert node.
  • The messages can be published to a topic namespace to which one or more alert handlers can subscribe.

No matter which approach is used, the handlers need to be enabled and configured in the configuration file. If the handler requires sensitive information such as tokens and passwords, it can also be configured using the Kapacitor HTTP API.

Push to handler

Pushing messages to a handler is the basic approach presented in the Getting started with Kapacitor guide. This involves simply calling the relevant chaining method made available through the alert node. Messages can be pushed to log() files, the email() service, the httpOut() cache and many third party services.

Publish and subscribe

An alert topic is simply a namespace where alerts are grouped. When an alert event fires it can be published to a topic. Multiple handlers can subscribe (can be bound) to that topic and all handlers process each alert event for the topic. Handlers get bound to topics through the kapacitor command line client and handler binding files. Handler binding files can be written in yaml or json. They contain four key fields and one optional one.

  • topic: declares the topic to which the handler will subscribe.
  • id: declares the identity of the binding.
  • kind: declares the type of event handler to be used. Note that this needs to be enabled in the kapacitord configuration.
  • match: (optional) declares a match expression used to filter which alert events will be processed. See the section Match Expressions below.
  • options: options specific to the handler in question. These are listed below in the section List of handlers

Example 1: A handler binding file for the slack handler and cpu topic

topic: cpu
id: slack
kind: slack
options:
  channel: '#kapacitor'

Example 1 could be saved into a file named slack_cpu_handler.yaml.

This can then be generated into a Kapacitor topic handler through the command line client.

$ kapacitor define-topic-handler slack_cpu_handler.yaml

Handler bindings can also be created over the HTTP API. See the Create a Handler section of the HTTP API document.

For a walk through on defining and using alert topics see the Using Alert Topics walk-through.

Handlers

A handler takes action on incoming alert events for a specific topic. Each handler operates on exactly one topic.

List of Handlers

The following is a list of available alert handlers and their options.

Aggregate

Aggreate multiple events into a single event.

Options:

NameTypeDescription
intervalduration stringHow often to aggregate events.
topicstringA topic into which to publish the aggregate events.
messagestringA template string where {{.Interval}} and {{.Count}} are available for constructing a meaning full message.

Example:

kind: aggregate
options:
    interval: 5m
    topic: agg_5m

Send aggregate events of the past 5m to the agg_5m topic. Further handling of the aggregated events can be configured on the agg_5m topic.

Alerta

Send alert events to an Alerta instance.

Options:

NameTypeDescription
tokenstringAlerta authentication token. If empty uses the token from the configuration.
token-prefixstringAlerta authentication token prefix. If empty uses Bearer.
resourcestringAlerta resource. Can be a template and has access to the same data as the AlertNode.Details property. Default: {{ .Name }}
eventstringAlerta event. Can be a template and has access to the same data as the idInfo property. Default: {{ .ID }}.
environmentstringAlerta environment. Can be a template and has access to the same data as the AlertNode.Details property. Defaut is set from the configuration.
groupstringAlerta group. Can be a template and has access to the same data as the AlertNode.Details property. Default: {{ .Group }}.
valuestringAlerta value. Can be a template and has access to the same data as the AlertNode.Details property. Default is an empty string.
originstringAlerta origin. If empty uses the origin from the configuration.
servicelist of stringList of effected Services.

Example:

kind: alerta
options:
    resource: system

Exec

Execute an external program, the alert data is passed over STDIN to the process.

Options:

NameTypeDescription
progstringPath to program to execute.
argslist of stringList of arguments to the program.

Example:

kind: exec
options:
    prog: /path/to/executable

Hipchat

Send alert events to a Hipchat room.

Options:

NameTypeDescription
roomstringHipChat room in which to post messages. If empty uses the channel from the configuration.
tokenstringHipChat authentication token. If empty uses the token from the configuration.

Example:

kind: hipchat
options:
    room: '#alerts'

Log

Log alert events to a file.

Options:

NameTypeDescription
pathstringPath to the log file.
modeintFile mode to use when creating the file.

Example:

kind: log
options:
    path: '/tmp/alerts.log'

Opsgenie

Send alert events to OpsGenie.

Options:

NameTypeDescription
teams-listlist of stringList of teams.
recipients-listList of recipients.

Example:

kind: opsgenie
options:
    teams:
        - rocket

Pagerduty

Send alert events to PagerDuty.

Options:

NameTypeDescription
service-keystringThe service key.

Example:

kind: pageduty

Pushover

Send alert events to Pushover.

Options:

NameTypeDescription
devicestringSpecific list of user’devices rather than all of a user’s devices (multiple device names may be separated by a comma)
titlestringThe message title, otherwise the apps name is used.
urlstringA supplementary URL to show with the message.
url-titlestringA title for a supplementary URL, otherwise just the URL is shown.
soundstringThe name of one of the sounds supported by the device clients to override the user’s default sound choice.

Example:

kind: pushover
options:
    title: Alert from Kapacitor

Post

Post JSON encoded alert data to an HTTP endpoint.

Options:

NameTypeDescription
urlstringThe URL to which the alert data will be posted.
endpointstringName of a configured httppost endpoint, cannot be specified in conjunciton with URL.
headersmap of string to stringSet of extra header values to set on the POST request.

Example:

kind: post
options:
    url: http://example.com

Publish

Publish events to another topic.

Options:

NameTypeDescription
topicslist of stringList of topic names to publish events.

Example:

kind: publish
options:
    topics:
        - system
        - ops_team

Sensu

Send alert events to Sensu.

Options:

NameTypeDescription
sourcestringSensu source for which to post messages. If empty uses the source from the configuration.
handlerslist of stringSensu handler list. If empty uses the handler list from the configuration.

Example:

kind: sensu

Slack

Send alert events to Slack.

Options:

NameTypeDescription
channelstringSlack channel in which to post messages. If empty uses the channel from the configuration.
usernamestringUsername of the Slack bot. If empty uses the username from the configuration.
icon-emojistringIconEmoji is an emoji name surrounded in ‘:’ characters. The emoji image will replace the normal user icon for the slack bot.

Example:

kind: slack
options:
    channel: '#alerts'

SMTP

Send alert events via email.

Options:

NameTypeDescription
tolist of stringList of email addresses.

Example:

kind: smtp
options:
    to:
        - oncall1@example.com
        - oncall2@example.com

Snmptrap

Trigger SNMP traps for alert events.

Options:

NameTypeDescription
trap-oidstringOID of the trap.
data-listobjectEach data object has oid, type, and value fields. Each field is a string.

Example:

kind: snmptrap
options:
    trap-oid: '1.1.1.1'
    data-list:
        oid: '1.3.6.1.2.1.1.7'
        type: i
        value: '{{ index .Field "value" }}'

Talk

Send alert events to a Talk instance. No options are available. See the Talk service configuration.

Example:

kind: talk

TCP

Send JSON encoded alert data to a TCP endpoint.

Options:

NameTypeDescription
addressstringAddress of TCP endpoint.

Example:

kind: tcp
options:
    address: 127.0.0.1:7777

Telegram

Send alert events to a Telegram instance.

Options:

NameTypeDescription
chat-idstringTelegram user/group ID to post messages to. If empty uses the chati-d from the configuration.
parse-modestringParse node, defaults to Mardown. If empty uses the parse-mode from the configuration.
disable-web-page-previewboolWeb Page preview. If empty uses the disable-web-page-preview from the configuration.
disable-notificationboolDisables Notification. If empty uses the disable-notification from the configuration.

Example:

kind: telegram

Victorops

Send alert events to a VictorOps instance.

Options:

NameTypeDescription
routing-keystringThe routing key of the alert event.

Example:

kind: victorops
options:
    routing-key: ops_team

Match expressions

Alert handlers support match expressions that filter which alert events the handler processes.

A match expression is a TICKscript lambda expression. The data that triggered the alert is available to the match expression, including all fields and tags.

In addition to the data that triggered the alert metadata about the alert is available. This alert metadata is available via various functions.

NameTypeDescription
levelintThe alert level of the event, one of ‘0’, ‘1’, ‘2’, or ‘3’ corresponding to ‘OK’, ‘INFO’, ‘WARNING’, and ‘CRITICAL’.
changedboolIndicates whether the alert level changed with this event.
namestringReturns the measurement name of the triggering data.
taskNamestringReturns the task name that generated the alert event.
durationdurationReturns the duration of the event in a non OK state.

Additionally the vars OK, INFO, WARNING, and CRITICAL have been defined to correspond with the return value of the level function.

For example to send only critical alerts to a handler, use this match expression:

match: level() == CRITICAL

Examples

Send only changed events to the handler:

match: changed() == TRUE

Send only WARNING and CRITICAL events to the handler:

match: level() >= WARNING

Send events with the tag “host” equal to s001.example.com to the handler:

match: "host" == 's001.example.com'

New! Cloud or OSS?

InfluxDB OSS 2.0 release candidate

InfluxDB OSS v2.0.rc includes breaking changes that require a manual upgrade from all alpha and beta versions. For information, see:

Upgrade to InfluxDB OSS v2.0.rc