Documentation

AlertNode

The alert node triggers an event of varying severity levels and passes the event to event handlers. The criteria for triggering an alert is specified via a lambda expression. See AlertNode.Info, AlertNode.Warn, and AlertNode.Crit below.

Constructor

Chaining methodDescription
alert ( )Create an alert node, which can trigger alerts.

To dynamically construct a custom HTTP body or URL, use an alert template. For example, httpPost('localhost/?host={{ index .Tags "host"}}&cpu={{ index .Tags "cpu" }}').

Property methods

Settersdescription
alerta ( )Send the alert to Alerta.
all ( )Indicates an alert should trigger only if all points in a batch match the criteria. Does not apply to stream alerts.
bigPanda ( )Send the alert to BigPanda.
crit ( value ast.LambdaNode)Filter expression for the CRITICAL alert level. An empty value indicates the level is invalid and is skipped.
critReset ( value ast.LambdaNode)Filter expression for resetting the CRITICAL alert level to lower level.
details ( value string)Template for constructing a detailed HTML message for the alert. The same template data is available as the AlertNode.Message property, in addition to a Message field that contains the rendered Message value.
discord ( )Send the alert to Discord.
durationField ( value string)Optional field key to add the alert duration to the data. The duration is always in units of nanoseconds.
email ( to ...string)Email the alert data.
exec ( executable stringargs ...string)Execute a command whenever an alert is triggered and pass the alert data over STDIN in JSON format.
flapping ( low float64high float64)Perform flap detection on the alerts. The method used is similar method to Nagios: https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/flapping.html
hipChat ( )Send the alert to HipChat.
history ( value int64)Number of previous states to remember when computing flapping levels and checking for state changes. Minimum value is 2 in order to keep track of current and previous states.
id ( value string)Template for constructing a unique ID for a given alert.
idField ( value string)Optional field key to add to the data, containing the alert ID as a string.
idTag ( value string)Optional tag key to use when tagging the data with the alert ID.
info ( value ast.LambdaNode)Filter expression for the INFO alert level. An empty value indicates the level is invalid and is skipped.
inhibit ( category stringequalTags ...string)Inhibit other alerts in a category. The equal tags provides a list of tags that must be equal in order for an alert event to be inhibited.
infoReset ( value ast.LambdaNode)Filter expression for resetting the INFO alert level to lower level.
kafka ( )Send the alert to a Kafka cluster.
levelField ( value string)Optional field key to add to the data, containing the alert level as a string.
levelTag ( value string)Optional tag key to use when tagging the data with the alert level.
log ( filepath string)Log JSON alert data to file. One event per line. Must specify the absolute path to the log file. It will be created if it does not exist. Example: stream
message ( value string)Template for constructing a meaningful message for the alert.
messageField ( value string)Optional field key to add to the data, containing the alert message.
mqtt ( topic string)Send alert to an MQTT broker
noRecoveries ( )Do not send recovery alerts.
opsGenie ( )Send alert to OpsGenie using OpsGenie’s v1 API. (Deprecated)
opsGenie2 ( )Send alert to OpsGenie using OpsGenie’s v2 API.
pagerDuty ( )Send the alert to PagerDuty using PagerDuty’s v1 API. (Deprecated)
pagerDuty2 ( )Send the alert to PagerDuty using PagerDuty’s v2 API.
post ( urls ...string)HTTP POST JSON alert data to a specified URL.
pushover ( )Send the alert to Pushover.
quiet ( )Suppresses all error logging events from this node.
sensu ( )Send the alert to Sensu.
slack ( )Send the alert to Slack.
snmpTrap ( trapOid string)Send the alert using SNMP traps.
stateChangesOnly ( maxInterval ...time.Duration)Only sends events where the state changed. Each different alert level OK, INFO, WARNING, and CRITICAL are considered different states.
talk ( )Send the alert to Talk.
tcp ( address string)Send JSON alert data to a specified address over TCP.
telegram ( )Send the alert to Telegram.
topic ( value string)Topic specifies the name of an alert topic to which, alerts will be published. Alert handlers can be configured per topic, see the API documentation.
victorOps ( )Send alert to VictorOps.
warn ( value ast.LambdaNode)Filter expression for the WARNING alert level. An empty value indicates the level is invalid and is skipped.
warnReset ( value ast.LambdaNode)Filter expression for resetting the WARNING alert level to lower level.

Chaining methods

Alert, Barrier, Bottom, ChangeDetect, Combine, Count, CumulativeSum, Deadman, Default, Delete, Derivative, Difference, Distinct, Ec2Autoscale, Elapsed, Eval, First, Flatten, GroupBy, HoltWinters, HoltWintersWithFit, HttpOut, HttpPost, InfluxDBOut, Join, K8sAutoscale, KapacitorLoopback, Last, Max, Mean, Median, Min, Mode, MovingAverage, Percentile, Sample, Shift, Sideload, Spread, StateCount, StateDuration, Stats, Stddev, Sum, SwarmAutoscale, Top, Trickle, Union, Where, Window


Available event handlers

Different event handlers can be configured for each AlertNode. Some handlers like Email, HipChat, Sensu, Slack, OpsGenie, VictorOps, PagerDuty, Telegram and Talk have a configuration option, global, that indicates that all alerts implicitly use the handler.

HandlerDescription
AlertaPost alert message to Alerta.
BigPandaPost alert message to BigPanda.
DiscordPost alert message to Discord channel.
emailSend and email with alert data.
execExecute a command passing alert data over STDIN.
HipChatPost alert message to HipChat room.
KafkaSend alert to a Apache Kafka cluster.
logLog alert data to file.
MQTTPost alert message to MQTT.
OpsGenie v1Send alert to OpsGenie using their v1 API. (Deprecated)
OpsGenie v2Send alert to OpsGenie using their v2 API.
PagerDuty v1Send alert to PagerDuty using their v1 API. (Deprecated)
PagerDuty v2Send alert to PagerDuty using their v2 API.
postHTTP POST data to a specified URL.
PushoverSend alert to Pushover.
SensuPost alert message to Sensu client.
SlackPost alert message to Slack channel.
SNMPTrapTrigger SNMP traps.
TalkPost alert message to Talk client.
tcpSend data to a specified address via raw TCP.
TelegramPost alert message to Telegram client.
VictorOpsSend alert to VictorOps.

Alert event data

Each event that gets sent to a handler contains the following alert data:

NameDescription
IDThe ID of the alert, user defined.
MessageThe alert message, user defined.
DetailsThe alert details, user defined HTML content.
TimeThe time the alert occurred.
DurationThe duration of the alert in nanoseconds.
LevelOne of OK, INFO, WARNING or CRITICAL.
Datainfluxql.Result containing the data that triggered the alert.
RecoverableIndicates whether the alert is auto-recoverable. Determined by the .noRecoveries() property.

Events are sent to handlers if the alert is in a state other than ‘OK’ or the alert just changed to the ‘OK’ state from a non ‘OK’ state (a.k.a. the alert recovered). Using the AlertNode.StateChangesOnly property events will only be sent to handlers if the alert changed state.

It is valid to configure multiple alert handlers, even with the same type.

Example:

stream
    .groupBy('service')
  |alert()
    .id('kapacitor/{{ index .Tags "service" }}')
    .message('{{ .ID }} is {{ .Level }} value:{{ index .Fields "value" }}')
    .info(lambda: "value" > 10)
    .warn(lambda: "value" > 20)
    .crit(lambda: "value" > 30)
    .post("http://example.com/api/alert")
    .post("http://another.example.com/api/alert")
    .tcp("exampleendpoint.com:5678")
    .email('oncall@example.com')
  • Copy
  • Fill window

Each expression maintains its own state. The order of execution for the expressions is not considered to be deterministic. For each point an expression may or may not be evaluated. If no expression is true then the alert is considered to be in the OK state.

Kapacitor supports alert reset expressions. This way when an alert enters a state, it can only be lowered in severity if its reset expression evaluates to true.

Example:

stream
  |from()
    .measurement('cpu')
    .where(lambda: "host" == 'serverA')
    .groupBy('host')
  |alert()
    .info(lambda: "value" > 60)
    .infoReset(lambda: "value" < 50)
    .warn(lambda: "value" > 70)
    .warnReset(lambda: "value" < 60)
    .crit(lambda: "value" > 80)
    .critReset(lambda: "value" < 70)
  • Copy
  • Fill window

For example, given the following values, the corresponding alert states are:

ValueAlert State
61INFO
73WARNING
64WARNING
85CRITICAL
62INFO
56INFO
47OK

Available Statistics:

  • alerts_triggered: Total number of alerts triggered
  • oks_triggered: Number of OK alerts triggered
  • infos_triggered: Number of Info alerts triggered
  • warns_triggered: Number of Warn alerts triggered
  • crits_triggered: Number of Crit alerts triggered

Properties

Property methods modify state on the calling node. They do not add another node to the pipeline, and always return a reference to the calling node. Property methods are marked using the . operator.

Alerta

Send the alert to Alerta. Detailed configuration options and setup instructions are provided in the Alerta Event Handler article.

Example kapacitor.conf

[alerta]
  enabled = true
  url = "https://alerta.yourdomain"
  token = "9hiWoDOZ9IbmHsOTeST123ABciWTIqXQVFDo63h9"
  environment = "Production"
  origin = "Kapacitor"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .alerta()
      .resource('Hostname or service')
      .event('Something went wrong')
  • Copy
  • Fill window

All

Indicates an alert should trigger only if all points in a batch match the criteria. Does not apply to stream alerts.

alert.all()
  • Copy
  • Fill window

BigPanda

Send the alert to BigPanda. Detailed configuration options and setup instructions are provided in the BigPanda Event Handler article.

Example kapacitor.conf

[bigpanda]
  enabled = true
  url = "https://api.bigpanda.io/data/v2/alerts"
  token = "BigPanda-API-auth-bearer-token"
  app-key = "BigPanda-integration-app-key"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .bigPanda()
      .appKey('fc39458f98e91eb0310258c3b725d643')
      .primaryProperty('device')
      .secondaryProperty('sensor_name')
      .topic('bigpanda-topic')
  • Copy
  • Fill window

Category

Category places this alert in a named category. Categories are used to inhibit alerts.

alert.category(value string)
  • Copy
  • Fill window

Crit

Filter expression for the CRITICAL alert level. An empty value indicates the level is invalid and is skipped.

alert.crit(value ast.LambdaNode)
  • Copy
  • Fill window

CritReset

Filter expression for resetting the CRITICAL alert level to lower level.

alert.critReset(value ast.LambdaNode)
  • Copy
  • Fill window

Details

Template for constructing a detailed HTML message for the alert. The same template data is available as the AlertNode.Message property, in addition to a Message field that contains the rendered Message value.

The intent is that the Message property be a single line summary while the Details property is a more detailed message possibly spanning multiple lines, and containing HTML formatting.

This template is rendered using the html/template package in Go so safe and valid HTML is generated.

The json method is available within the template to convert any variable to a valid JSON string.

xExample:

|alert()
  .id('{{ .Name }}')
  .details('''
<h1>{{ .ID }}</h1>
<b>{{ .Message }}</b>
Value: {{ index .Fields "value" }}
''')
  .email()
  • Copy
  • Fill window

Default: {{ json . }}

alert.details(value string)
  • Copy
  • Fill window

Discord

Send the alert to Discord. Detailed configuration options and setup instructions are provided in the Discord Event Handler article.

Example kapacitor.conf

[discord]
  enabled = true
  url = "https://discordapp.com/api/webhooks/xxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .discord()
      .embedTitle('Alert!')
  • Copy
  • Fill window

DurationField

Optional field key to add the alert duration to the data. The duration is always in units of nanoseconds.

alert.durationField(value string)
  • Copy
  • Fill window

Email

Email the alert data to specified “To” email addresses. Detailed configuration options and setup instructions are provided in the Email Event Handler article.

Example kapacitor.conf

[smtp]
  enabled = true
  host = "localhost"
  port = 25
  username = "xxxx"
  password = "xxxx"
  from = "kapacitor@example.com"
  to = ["oncall@example.com"]
  global = true
  state-changes-only = true
  • Copy
  • Fill window

Example TICKscript

|alert()
  .id('{{ .Name }}')
  // Email subject
  .message('{{ .ID }}:{{ .Level }}')
  // Email body as HTML
  .details('''
<h1>{{ .ID }}</h1>
<b>{{ .Message }}</b>
Value: {{ index .Fields "value" }}
''')
  .email()
  • Copy
  • Fill window

Exec

Execute a command whenever an alert is triggered and pass the alert data over STDIN in JSON format. Detailed usage instructions and examples are provided in the Exec Event Handler article.

// Pattern
alert.exec(executable string, args ...string)

// Example
alert.exec('/usr/bin/python', 'myscript.py')
  • Copy
  • Fill window

Flapping

Perform flap detection on the alerts. The method used is similar method to Nagios flapping

Each different alerting level is considered a different state. The low and high thresholds are inverted thresholds of a percentage of state changes. Meaning that if the percentage of state changes goes above the high threshold, the alert enters a flapping state. The alert remains in the flapping state until the percentage of state changes goes below the low threshold. Typical values are low: 0.25 and high: 0.5. The percentage values represent the number state changes over the total possible number of state changes. A percentage change of 0.5 means that the alert changed state in half of the recorded history, and remained the same in the other half of the history.

// Pattern
alert.flapping(low float64, high float64)

// Example
alert.flapping(0.25, 0.5)
  • Copy
  • Fill window

Kafka

Send the alert to an Apache Kafka cluster. Detailed configuration options and setup instructions are provided in the Kafka Event Handler article.

Example kapacitor.conf

[[kafka]]
  enabled = true
  id = "localhost"
  timeout = 10s
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .kafka()
      .cluster('kafka-cluster')
      .kafkaTopic('kafka-topic-name')
  • Copy
  • Fill window

History

Number of previous states to remember when computing flapping levels and checking for state changes. Minimum value is 2 in order to keep track of current and previous states.

Default: 21

// Pattern
alert.history(value int64)

// Example
alert.history(21)
  • Copy
  • Fill window

HipChat

Send the alert to HipChat. Detailed configuration options and setup instructions are provided in the HipChat Event Handler article.

Example kapacitor.conf

[hipchat]
  enabled = true
  url = "https://orgname.hipchat.com/v2/room"
  room = "4189212"
  token = "9hiWoDOZ9IbmHsOTeST123ABciWTIqXQVFDo63h9"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .hipChat()
      .room('Kapacitor')
  • Copy
  • Fill window

Id

Template for constructing a unique ID for a given alert.

Available template data

Data NameDescription
NameMeasurement name.
TaskNameThe name of the task
GroupConcatenation of all group-by tags of the form [key=value,]+. If no groupBy is performed equal to literal ’nil’.
TagsMap of tags. Use `{{ index .Tags “key” }}`` to get a specific tag value.
ServerInfoInformation about the running server. Available nested fields are Hostname, ClusterID and ServerID.

Default: {{ .Name }}:{{ .Group }}

// Pattern
alert.id(value string)

// Example
alert.id('{{ .Name }}:{{ .Group }}')
  • Copy
  • Fill window

Example: ID template using name and group

stream
  |from()
    .measurement('cpu')
    .groupBy('cpu')
  |alert()
    .id('kapacitor/{{ .Name }}/{{ .Group }}')
  • Copy
  • Fill window

Resulting ID: kapacitor/cpu/cpu=cpu0

Example: ID template using a tag

stream
  |from()
    .measurement('cpu')
    .groupBy('service')
  |alert()
    .id('kapacitor/{{ index .Tags "service" }}')
  • Copy
  • Fill window

Resulting ID: kapacitor/authentication

Example: ID template using multiple tags

stream
  |from()
    .measurement('cpu')
    .groupBy('service', 'host')
  |alert()
    .id('kapacitor/{{ index .Tags "service" }}/{{ index .Tags "host" }}')
  • Copy
  • Fill window

Resulting ID: kapacitor/authentication/auth001.example.com

IdField

Optional field key to add to the data, containing the alert ID as a string.

// Pattern
alert.idField(value string)

// Example
alert.idField('id')
  • Copy
  • Fill window

IdTag

Optional tag key to use when tagging the data with the alert ID.

// Pattern
alert.idTag(value string)

// Example
alert.idTag('alertID')
  • Copy
  • Fill window

Info

Filter expression for the INFO alert level. An empty value indicates the level is invalid and is skipped.

// Pattern
alert.info(value ast.LambdaNode)

// Example
alert.info(lambda: 'usage_idle' < 60)
  • Copy
  • Fill window

InfoReset

Filter expression for resetting the INFO alert level to lower level.

// Pattern
alert.infoReset(value ast.LambdaNode)

// Example
alert.infoReset(lambda: 'usage_idle' > 60)
  • Copy
  • Fill window

Inhibit

Inhibit other alerts in a category. The equal tags provides a list of tags that must be equal in order for an alert event to be inhibited.

The following two TICKscripts demonstrate how to use the inhibit feature:

    //cpu_alert.tick
stream
  |from()
    .measurement('cpu')
    .groupBy('host')
  |alert()
    .category('system_alerts')
    .crit(lambda: "usage_idle" < 10.0)
  • Copy
  • Fill window
//host_alert.tick
stream
  |from()
    .measurement('uptime')
    .groupBy('host')
  |deadman(0.0, 1m)
    .inhibit('system_alerts', 'host')
  • Copy
  • Fill window

The deadman is a type of alert node and can be used to inhibit all alerts in the system_alerts category when triggered. The host argument to the inhibit function says that the host tag must be equal between the CPU alert and the host alert in order for it to be inhibited. This has the effect of the deadman alerts only inhibiting CPU alerts for hosts that are currently dead.

alert.inhibit(category string, equalTags ...string)
  • Copy
  • Fill window

Kafka

Send the alert to a Kafka topic. Detailed setup and usage instructions are provided in the Kafka Event Handler article.

Example: kapacitor.conf

[[kafka]]
  enabled = true
  id = "default"
  brokers = ["localhost:9092"]
  • Copy
  • Fill window

Example: TICKscript

stream
 |alert()
    .kafka()
      .cluster('default')
      .kafkaTopic('alerts')
  • Copy
  • Fill window

LevelField

Optional field key to add to the data, containing the alert level as a string.

// Pattern
alert.levelField(value string)

// Example
alert.levelField('INFO')
  • Copy
  • Fill window

LevelTag

Optional tag key to use when tagging the data with the alert level.

// Pattern
alert.levelTag(value string)

// Example
alert.levelTag('level')
  • Copy
  • Fill window

Log

Log JSON alert data to file. Detailed setup and usage instructions are provided in the Log Event Handler article.

Example TICKscript

stream
  |alert()
    .log('/tmp/alert')
      .mode(0644)
  • Copy
  • Fill window

Message

Template for constructing a meaningful message for the alert.

Available template data

Data NameDescription
IDThe ID of the alert.
NameMeasurement name.
TaskNameThe name of the task
GroupConcatenation of all group-by tags of the form 1. If no groupBy is performed equal to literal ’nil’.
TagsMap of tags. Use {{ index .Tags "key" }} to get a specific tag value.
LevelAlert Level - INFO, WARNING, CRITICAL.
FieldsMap of fields. Use {{ index .Fields "key" }} to get a specific field value.
TimeThe time of the point that triggered the event. Default format is YYYY-MM-DD 00:00:00 +0000 UTC
DurationThe duration of the alert.

Default: {{ .ID }} is {{ .Level }}

Example:

stream
  |from()
    .measurement('cpu')
    .groupBy('service', 'host')
  |alert()
    .id('{{ index .Tags "service" }}/{{ index .Tags "host" }}')
    .message('{{ .ID }} is {{ .Level}} value: {{ index .Fields "value" }}')
  • Copy
  • Fill window

Resulting Message: authentication/auth001.example.com is CRITICAL value:42

MessageField

Optional field key to add to the data containing the alert message.

// Pattern
alert.messageField(value string)

// Example
alert.messageField('message')
  • Copy
  • Fill window

MQTT

Send alert to an MQTT broker. Detailed configuration options and usage instructions are provided in the MQTT Event Handler article.

Example kapacitor.conf

[[mqtt]]
  enabled = true
  name = "localhost"
  default = true
  url = "tcp://localhost:1883"
  client-id = "kapacitor"
  username = "myusername"
  password = "mysupersecretpassw0rd"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .mqtt('topic')
      .brokerName('localhost')
  • Copy
  • Fill window

NoRecoveries

Do not send recovery alerts. Sets recoverable alert data field to false.

alert.noRecoveries()
  • Copy
  • Fill window

OpsGenie v1

Send alert to OpsGenie using OpsGenie’s v1 API. Detailed configuration options and setup instructions are provided in the OpsGenie v1 Event Handler article.

Example kapacitor.conf

[opsgenie]
  enabled = true
  api-key = "xxxxx"
  teams = ["everyone"]
  recipients = ["jim", "bob"]
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .opsGenie()
  • Copy
  • Fill window

OpsGenie v2

Send alert to OpsGenie using OpsGenie’s v2 API. Detailed configuration options and setup instructions are provided in the OpsGenie v2 Event Handler article.

Example kapacitor.conf

[opsgenie2]
  enabled = true
  api-key = "xxxxx"
  teams = ["everyone"]
  recipients = ["jim", "bob"]
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .opsGenie2()
  • Copy
  • Fill window

PagerDuty v1

Send the alert to PagerDuty using PagerDuty’s v1 API. Detailed configuration options and setup instructions are provided in the PagerDuty v1 Event Handler article.

Example kapacitor.conf

[pagerduty]
  enabled = true
  service-key = "xxxx"
  url = "https://events.pagerduty.com/generic/2010-04-15/create_event.json"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .pagerDuty()
  • Copy
  • Fill window

PagerDuty v2

Send the alert to PagerDuty using PagerDuty’s v2 API. Detailed configuration options and setup instructions are provided in the PagerDuty v2 Event Handler article.

Example kapacitor.conf

[pagerduty2]
  enabled = true
  routing-key = "xxxx"
  url = "https://events.pagerduty.com/v2/enqueue"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .pagerDuty2()
  • Copy
  • Fill window

Post

HTTP POST JSON alert data to a specified URL. Detailed configuration options and setup instructions are provided in the Post Event Handler article.

Example TICKscript

stream
  |alert()
    .post('http://example.com')
      .captureResponse()
  • Copy
  • Fill window

Pushover

Send the alert to Pushover. Detailed configuration options and setup instructions are provided in the Pushover Event Handler article.

Example kapacitor.conf

[pushover]
  enabled = true
  token = "9hiWoDOZ9IbmHsOTeST123ABciWTIqXQVFDo63h9"
  user_key = "Pushover"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .pushover()
      .sound('siren')
      .user_key('other user')
      .device('mydev')
      .title('mytitle')
      .uRL('myurl')
      .URLTitle('mytitle')
  • Copy
  • Fill window

Quiet

Suppress all error logging events from this node.

alert.quiet()
  • Copy
  • Fill window

Sensu

Send the alert to Sensu. Detailed configuration options and setup instructions are provided in the Sensu Event Handler article.

*Example kapacitor.conf

[sensu]
  enabled = true
  url = "http://sensu:3030"
  source = "Kapacitor"
  handlers = ["sns","slack"]
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .sensu()
      .handlers('sns','slack')
  • Copy
  • Fill window

Slack

Send the alert to Slack. Detailed configuration options and setup instructions are provided in the Slack Event Handler article.

Example kapacitor.conf

[slack]
  enabled = true
  url = "https://hooks.slack.com/services/xxxxxxxxx/xxxxxxxxx/xxxxxxxxxxxxxxxxxxxxxxxx"
  channel = "#general"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .slack()
      .channel('#alerts')
      .channel('@jsmith')
  • Copy
  • Fill window

SnmpTrap

Send the alert using SNMP traps. Detailed configuration options and setup instructions are provided in the SNMP Trap Event Handler article.

Example kapacitor.conf

[snmptrap]
  enabled = true
  addr = "127.0.0.1:9162"
  community = "public"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .snmpTrap('1.3.6.1.2.1.1')
      .data('1.3.6.1.2.1.1.6', 'i', '{{ index .Field "value" }}')
      .data('1.3.6.1.2.1.1.7', 's', '{{ .Message }}')
  • Copy
  • Fill window

StateChangesOnly

Only sends events where the state changed. Each different alert level OK, INFO, WARNING, and CRITICAL are considered different states.

Example

stream
  |from()
    .measurement('cpu')
  |window()
    .period(10s)
    .every(10s)
  |alert()
    .crit(lambda: "value" > 10)
    .stateChangesOnly()
    .slack()
  • Copy
  • Fill window

If the “value” is greater than 10 for a total of 60s, then only two events will be sent. First, when the value crosses the threshold, and second, when it falls back into an OK state. Without stateChangesOnly, the alert would have triggered 7 times: 6 times for each 10s period where the condition was met and once more for the recovery.

An optional maximum interval duration can be provided. An event will not be ignored (aka trigger an alert) if more than the maximum interval has elapsed since the last alert.

Example

stream
  // ...
  |alert()
    .crit(lambda: "value" > 10)
    .stateChangesOnly(10m)
    .slack()
  • Copy
  • Fill window

TCP

Send JSON alert data to a specified address over TCP. Detailed usage instructions are provided in the TCPEvent Handler article.

// Pattern
alert.tcp(address string)

// Example
alert.tcp('127.0.0.1:7777')
  • Copy
  • Fill window

Telegram

Send the alert to Telegram. Detailed configuration options and setup instructions are provided in the Telegram Event Handler article.

Example kapacitor.conf

[telegram]
  enabled = true
  token = "123456789:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
  chat-id = "xxxxxxxxx"
  parse-mode = "Markdown"
	disable-web-page-preview = true
	disable-notification = false
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .telegram()
      .chatId('xxxxxxx')
  • Copy
  • Fill window

Topic

Topic specifies the name of an alert topic to which alerts will be published. Alert handlers can be configured per topic. See the API documentation.

// Pattern
alert.topic(value string)

// Example
alert.topic('cpu')
  • Copy
  • Fill window

VictorOps

Send alert to VictorOps. Detailed configuration options and setup instructions are provided in the VictorOps Event Handler article.

Example kapacitor.conf

[victorops]
  enabled = true
  api-key = "xxxxx"
  routing-key = "everyone"
  • Copy
  • Fill window

Example TICKscript

stream
  |alert()
    .victorOps()
      .routingKey('team_rocket')
  • Copy
  • Fill window

Warn

Filter expression for the WARNING alert level. An empty value indicates the level is invalid and is skipped.

// Pattern
alert.warn(value ast.LambdaNode)

// Example
alert.warn(lambda: 'usage_idle' < 20)
  • Copy
  • Fill window

WarnReset

Filter expression for resetting the WARNING alert level to lower level.

// Pattern
alert.warnReset(value ast.LambdaNode)

// Example
alert.warnReset(lambda: 'usage_idle' > 20)
  • Copy
  • Fill window

Chaining Methods

Chaining methods create a new node in the pipeline as a child of the calling node. They do not modify the calling node. Chaining methods are marked using the | operator.

Alert

Create an alert node, which can trigger alerts.

alert|alert()
  • Copy
  • Fill window

Returns: AlertNode

Barrier

Create a new Barrier node that emits a BarrierMessage periodically.

One BarrierMessage will be emitted every period duration.

alert|barrier()
  • Copy
  • Fill window

Returns: BarrierNode

Bottom

Select the bottom num points for field and sort by any extra tags or fields.

alert|bottom(num int64, field string, fieldsAndTags ...string)
  • Copy
  • Fill window

Returns: InfluxQLNode

ChangeDetect

Create a new node that only emits new points if different from the previous point.

alert|changeDetect(field string)
  • Copy
  • Fill window

Returns: ChangeDetectNode

Combine

Combine this node with itself. The data is combined on timestamp.

alert|combine(expressions ...ast.LambdaNode)
  • Copy
  • Fill window

Returns: CombineNode

Count

Count the number of points.

alert|count(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

CumulativeSum

Compute a cumulative sum of each point that is received. A point is emitted for every point collected.

alert|cumulativeSum(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Deadman

Helper function for creating an alert on low throughput, a.k.a. deadman’s switch.

  • Threshold: trigger alert if throughput drops below threshold in points/interval.
  • Interval: how often to check the throughput.
  • Expressions: optional list of expressions to also evaluate. Useful for time of day alerting.

Example:

    var data = stream
        |from()...
    // Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
    data
        |deadman(100.0, 10s)
    //Do normal processing of data
    data...
  • Copy
  • Fill window

The above is equivalent to this example:

    var data = stream
        |from()...
    // Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
    data
        |stats(10s)
            .align()
        |derivative('emitted')
            .unit(10s)
            .nonNegative()
        |alert()
            .id('node \'stream0\' in task \'{{ .TaskName }}\'')
            .message('{{ .ID }} is {{ if eq .Level "OK" }}alive{{ else }}dead{{ end }}: {{ index .Fields "emitted" | printf "%0.3f" }} points/10s.')
            .crit(lambda: "emitted" <= 100.0)
    //Do normal processing of data
    data...
  • Copy
  • Fill window

The id and message alert properties can be configured globally via the ‘deadman’ configuration section.

Since the AlertNode is the last piece it can be further modified as usual. Example:

    var data = stream
        |from()...
    // Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
    data
        |deadman(100.0, 10s)
            .slack()
            .channel('#dead_tasks')
    //Do normal processing of data
    data...
  • Copy
  • Fill window

You can specify additional lambda expressions to further constrain when the deadman’s switch is triggered. Example:

    var data = stream
        |from()...
    // Trigger critical alert if the throughput drops below 100 points per 10s and checked every 10s.
    // Only trigger the alert if the time of day is between 8am-5pm.
    data
        |deadman(100.0, 10s, lambda: hour("time") >= 8 AND hour("time") <= 17)
    //Do normal processing of data
    data...
  • Copy
  • Fill window
alert|deadman(threshold float64, interval time.Duration, expr ...ast.LambdaNode)
  • Copy
  • Fill window

Returns: AlertNode

Default

Create a node that can set defaults for missing tags or fields.

alert|default()
  • Copy
  • Fill window

Returns: DefaultNode

Delete

Create a node that can delete tags or fields.

alert|delete()
  • Copy
  • Fill window

Returns: DeleteNode

Derivative

Create a new node that computes the derivative of adjacent points.

alert|derivative(field string)
  • Copy
  • Fill window

Returns: DerivativeNode

Difference

Compute the difference between points independent of elapsed time.

alert|difference(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Distinct

Produce batch of only the distinct points.

alert|distinct(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Ec2Autoscale

Create a node that can trigger autoscale events for a ec2 autoscalegroup.

alert|ec2Autoscale()
  • Copy
  • Fill window

Returns: Ec2AutoscaleNode

Elapsed

Compute the elapsed time between points.

alert|elapsed(field string, unit time.Duration)
  • Copy
  • Fill window

Returns: InfluxQLNode

Eval

Create an eval node that will evaluate the given transformation function to each data point. A list of expressions may be provided and will be evaluated in the order they are given. The results are available to later expressions.

alert|eval(expressions ...ast.LambdaNode)
  • Copy
  • Fill window

Returns: EvalNode

First

Select the first point.

alert|first(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Flatten

Flatten points with similar times into a single point.

alert|flatten()
  • Copy
  • Fill window

Returns: FlattenNode

GroupBy

Group the data by a set of tags.

Can pass literal * to group by all dimensions. Example:

    |groupBy(*)
  • Copy
  • Fill window
alert|groupBy(tag ...interface{})
  • Copy
  • Fill window

Returns: GroupByNode

HoltWinters

Compute the Holt-Winters (/influxdb/v1/query_language/functions/#holt-winters) forecast of a data set.

alert|holtWinters(field string, h int64, m int64, interval time.Duration)
  • Copy
  • Fill window

Returns: InfluxQLNode

HoltWintersWithFit

Compute the Holt-Winters (/influxdb/v1/query_language/functions/#holt-winters) forecast of a data set. This method also outputs all the points used to fit the data in addition to the forecasted data.

alert|holtWintersWithFit(field string, h int64, m int64, interval time.Duration)
  • Copy
  • Fill window

Returns: InfluxQLNode

HttpOut

Create an HTTP output node that caches the most recent data it has received. The cached data is available at the given endpoint. The endpoint is the relative path from the API endpoint of the running task. For example, if the task endpoint is at /kapacitor/v1/tasks/<task_id> and endpoint is top10, then the data can be requested from /kapacitor/v1/tasks/<task_id>/top10.

alert|httpOut(endpoint string)
  • Copy
  • Fill window

Returns: HTTPOutNode

HttpPost

Creates an HTTP Post node that POSTS received data to the provided HTTP endpoint. HttpPost expects 0 or 1 arguments. If 0 arguments are provided, you must specify an endpoint property method.

alert|httpPost(url ...string)
  • Copy
  • Fill window

Returns: HTTPPostNode

InfluxDBOut

Create an influxdb output node that will store the incoming data into InfluxDB.

alert|influxDBOut()
  • Copy
  • Fill window

Returns: InfluxDBOutNode

Join

Join this node with other nodes. The data is joined on timestamp.

alert|join(others ...Node)
  • Copy
  • Fill window

Returns: JoinNode

K8sAutoscale

Create a node that can trigger autoscale events for a kubernetes cluster.

alert|k8sAutoscale()
  • Copy
  • Fill window

Returns: K8sAutoscaleNode

KapacitorLoopback

Create an kapacitor loopback node that will send data back into Kapacitor as a stream.

alert|kapacitorLoopback()
  • Copy
  • Fill window

Returns: KapacitorLoopbackNode

Last

Select the last point.

alert|last(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Max

Select the maximum point.

alert|max(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Mean

Compute the mean of the data.

alert|mean(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Median

Compute the median of the data.

Note: This method is not a selector. If you want the median point, use .percentile(field, 50.0).

alert|median(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Min

Select the minimum point.

alert|min(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Mode

Compute the mode of the data.

alert|mode(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

MovingAverage

Compute a moving average of the last window points. No points are emitted until the window is full.

alert|movingAverage(field string, window int64)
  • Copy
  • Fill window

Returns: InfluxQLNode

Percentile

Select a point at the given percentile. This is a selector function, no interpolation between points is performed.

alert|percentile(field string, percentile float64)
  • Copy
  • Fill window

Returns: InfluxQLNode

Sample

Create a new node that samples the incoming points or batches.

One point will be emitted every count or duration specified.

alert|sample(rate interface{})
  • Copy
  • Fill window

Returns: SampleNode

Shift

Create a new node that shifts the incoming points or batches in time.

alert|shift(shift time.Duration)
  • Copy
  • Fill window

Returns: ShiftNode

Sideload

Create a node that can load data from external sources.

alert|sideload()
  • Copy
  • Fill window

Returns: SideloadNode

Spread

Compute the difference between min and max points.

alert|spread(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

StateCount

Create a node that tracks number of consecutive points in a given state.

alert|stateCount(expression ast.LambdaNode)
  • Copy
  • Fill window

Returns: StateCountNode

StateDuration

Create a node that tracks duration in a given state.

alert|stateDuration(expression ast.LambdaNode)
  • Copy
  • Fill window

Returns: StateDurationNode

Stats

Create a new stream of data that contains the internal statistics of the node. The interval represents how often to emit the statistics based on real time. This means the interval time is independent of the times of the data points the source node is receiving.

alert|stats(interval time.Duration)
  • Copy
  • Fill window

Returns: StatsNode

Stddev

Compute the standard deviation.

alert|stddev(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Sum

Compute the sum of all values.

alert|sum(field string)
  • Copy
  • Fill window

Returns: InfluxQLNode

SwarmAutoscale

Create a node that can trigger autoscale events for a Docker swarm cluster.

alert|swarmAutoscale()
  • Copy
  • Fill window

Returns: SwarmAutoscaleNode

Top

Select the top num points for field and sort by any extra tags or fields.

alert|top(num int64, field string, fieldsAndTags ...string)
  • Copy
  • Fill window

Returns: InfluxQLNode

Trickle

Create a new node that converts batch data to stream data.

alert|trickle()
  • Copy
  • Fill window

Returns: TrickleNode

Union

Perform the union of this node and all other given nodes.

alert|union(node ...Node)
  • Copy
  • Fill window

Returns: UnionNode

Where

Create a new node that filters the data stream by a given expression.

alert|where(expression ast.LambdaNode)
  • Copy
  • Fill window

Returns: WhereNode

Window

Create a new node that windows the stream by time.

NOTE: Window can only be applied to stream edges.

alert|window()
  • Copy
  • Fill window

Returns: WindowNode


Was this page helpful?

Thank you for your feedback!


The future of Flux

Flux is going into maintenance mode. You can continue using it as you currently are without any changes to your code.

Read more

InfluxDB 3 Core and Enterprise are now in Beta

InfluxDB 3 Core and Enterprise are now available for beta testing, available under MIT or Apache 2 license.

InfluxDB 3 Core is a high-speed, recent-data engine that collects and processes data in real-time, while persisting it to local disk or object storage. InfluxDB 3 Enterprise is a commercial product that builds on Core’s foundation, adding high availability, read replicas, enhanced security, and data compaction for faster queries. A free tier of InfluxDB 3 Enterprise will also be available for at-home, non-commercial use for hobbyists to get the full historical time series database set of capabilities.

For more information, check out: