PagerDuty v2 event handler

PagerDuty is an incident management platform that helps teams detect and fix infrastructure problems quickly. Kapacitor can be configured to send alert messages to PagerDuty.

This page is specific to PagerDuty’s v2 API. If still using their v1 API, view the PagerDuty v1 event handler documentation.

Configuration

Configuration as well as default option values for the PagerDuty v2 event handler are set in your kapacitor.conf. Below is an example configuration:

[pagerduty2]
  enabled = true
  routing-key = ""
  url = "https://events.pagerduty.com/v2/enqueue"
  global = false

enabled

Set to true to enable the PagerDuty v2 event handler.

routing-key

Your PagerDuty Routing Key.

url

The PagerDuty API v2 URL. This should not need to be changed.

global

If true, all alerts will be sent to PagerDuty without explicitly specifying PagerDuty in TICKscripts.

Options

The following PagerDuty v2 event handler options can be set in a handler file or when using .pagerDuty2() in a TICKscript.

Example: handler file

id: handler-id
topic: topic-name
kind: pagerduty2
options:
  routing-key: 'myroutingkey'
  links:
    - href: 'https://chronograf.example.com/sources/1/dashboards/2'
      text: 'Overview Dashboard'
    - href: 'https://chronograf.example.com/'

Example: TICKscript

|alert()
  // ...
  .pagerDuty2()
    .routingKey('myroutingkey')
    .link('https://chronograf.example.com/sources/1/dashboards/2', 'Overview Dashboard')
    .link('https://chronograf.example.com/')

PagerDuty Setup

To allow Kapacitor to send alerts to PagerDuty enable a new “Generic API” integration. Use the generated “Integration Key” as the routing-key under the [pagerduty2] section of your kapacitor.conf.

Using the PagerDuty v2 Event Handler

With the PagerDuty v2 event handler enabled in your kapacitor.conf, use the .pagerDuty2() attribute in your TICKscripts to send alerts to a PagerDuty or define a PagerDuty v2 handler that subscribes to a topic and sends published alerts to PagerDuty.

The examples below use the following PagerDuty v2 configuration defined in the kapacitor.conf:

PagerDuty v2 settings in kapacitor.conf

[pagerduty2]
  enabled = true
  routing-key = "myroutingkey"
  url = "https://events.pagerduty.com/v2/enqueue"
  global = false

Send alerts to PagerDuty from a TICKscript

The following TICKscript uses the .pagerDuty2() event handler to send the message, “Hey, check your CPU”, whenever idle CPU usage drops below 10%.

pagerduty2-cpu-alert.tick

stream
  |from()
    .measurement('cpu')
  |alert()
    .crit(lambda: "usage_idle" < 10)
    .message('Hey, check your CPU')
    .pagerDuty2()

Send alerts to PagerDuty from a defined handler

The following setup sends an alert to the cpu topic with the message, “Hey, check your CPU”. A PagerDuty v2 handler is added that subscribes to the cpu topic and publishes all alert messages to PagerDuty.

Create a TICKscript that publishes alert messages to a topic. The TICKscript below sends an alert message to the cpu topic any time CPU idle usage drops below 10% (or CPU usage is above 90%).

cpu_alert.tick

stream
  |from()
    .measurement('cpu')
  |alert()
    .crit(lambda: "usage_idle" < 10)
    .message('Hey, check your CPU')
    .topic('cpu')

Add and enable the TICKscript:

kapacitor define cpu_alert -tick cpu_alert.tick
kapacitor enable cpu_alert

Create a handler file that subscribes to the cpu topic and uses the PagerDuty v2 event handler to send alerts to PagerDuty.

pagerduty2_cpu_handler.yaml

topic: cpu
id: pagerduty2-cpu-alert
kind: pagerduty2
options:
  routing-key: 'myroutingkey'

Add the handler:

kapacitor define-topic-handler pagerduty2_cpu_handler.yaml