--- title: Threshold deadman checks plugin description: Monitor data thresholds and detect missing data with deadman checks for alerting. url: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/threshold-deadman-checks/ estimated_tokens: 4782 product: InfluxDB 3 Enterprise version: enterprise publisher: InfluxData canonical: https://docs.influxdata.com/influxdb3/enterprise/plugins/library/official/threshold-deadman-checks/ date: '2025-07-23T18:11:03-05:00' lastmod: '2025-07-23T18:11:03-05:00' --- The Threshold Deadman Checks Plugin provides comprehensive monitoring capabilities for time series data in InfluxDB 3 Enterprise, combining real-time threshold detection with deadman monitoring. Monitor field values against configurable thresholds, detect data absence patterns, and trigger multi-level alerts based on aggregated metrics. Features both scheduled batch monitoring and real-time data write monitoring with configurable trigger counts and severity levels. ## Configuration Plugin parameters may be specified as key-value pairs in the `--trigger-arguments` flag (CLI) or in the `trigger_arguments` field (API) when creating a trigger. ### Plugin metadata This plugin includes a JSON metadata schema in its docstring that defines supported trigger types and configuration parameters. This metadata enables the [InfluxDB 3 Explorer](https://docs.influxdata.com/influxdb3/explorer/) UI to display and configure the plugin. ### Required parameters | Parameter | Type |Default | Description | |-------------|------|--------|-------------------------------------------------------------------------------| |`measurement`|string|required| Measurement to monitor for deadman alerts and aggregation-based conditions | | `senders` |string|required|Dot-separated notification channels with multi-channel notification integration| | `window` |string|required| Time window for periodic data presence checking | ### Data write trigger parameters | Parameter | Type |Default | Description | |------------------|------|--------|------------------------------------------------------------------------------------------------------| | `measurement` |string|required| Measurement to monitor for real-time threshold violations in dual monitoring mode | |`field_conditions`|string|required|Real-time threshold conditions with multi-level alerting (INFO, WARN, ERROR, CRITICAL severity levels)| | `senders` |string|required| Dot-separated notification channels with multi-channel notification integration | ### Threshold check parameters | Parameter | Type |Default| Description | |--------------------------|-------|-------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------| |`field_aggregation_values`|string | none |Multi-level aggregation conditions with aggregation support for avg, min, max, count, sum, median, stddev, first\_value, last\_value, var, and approx\_median values| | `deadman_check` |boolean| false | Enable deadman detection to monitor for data absence and missing data streams | | `interval` |string |“5min” | Configurable aggregation time interval for batch processing with performance optimization | | `trigger_count` |number | 1 | Configurable triggers requiring multiple consecutive failures before alerting | ### Notification parameters | Parameter | Type |Default | Description | |-----------------------------|------|--------|-----------------------------------------------------------------------------------------| | `influxdb3_auth_token` |string|env var | InfluxDB 3 Enterprise API token with environment variable support | | `notification_deadman_text` |string|template| Customizable deadman alert template message with dynamic variables | |`notification_threshold_text`|string|template| Customizable threshold alert template message with dynamic variables | | `notification_text` |string|template|Customizable notification template message for data write triggers with dynamic variables| | `notification_path` |string|“notify”| Notification endpoint path with retry logic and exponential backoff | | `port_override` |number| 8181 | InfluxDB port override for notification delivery | ### TOML configuration | Parameter | Type |Default| Description | |------------------|------|-------|--------------------------------------------------------------------------------| |`config_file_path`|string| none |TOML config file path relative to `PLUGIN_DIR` (required for TOML configuration)| *To use a TOML configuration file, set the `PLUGIN_DIR` environment variable and specify the `config_file_path` in the trigger arguments.* This is in addition to the `--plugin-dir` flag when starting InfluxDB 3 Enterprise. Example TOML configuration files provided: * [threshold\_deadman\_config\_scheduler.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/threshold_deadman_checks/threshold_deadman_config_scheduler.toml) - for scheduled triggers * [threshold\_deadman\_config\_data\_writes.toml](https://github.com/influxdata/influxdb3_plugins/blob/master/influxdata/threshold_deadman_checks/threshold_deadman_config_data_writes.toml) - for data write triggers For more information on using TOML configuration files, see the Using TOML Configuration Files section in the [influxdb3\_plugins/README.md](https://github.com/influxdata/influxdb3_plugins/blob/master/README.md). ### Channel-specific configuration Notification channels require additional parameters based on the sender type (same as the [influxdata/notifier plugin](../notifier/README.md)). ## Schema requirement The plugin assumes that the table schema is already defined in the database, as it relies on this schema to retrieve field and tag names required for processing. ## Software requirements * **InfluxDB v3 Core/Enterprise**: with the Processing Engine enabled. * **Notification Sender Plugin for InfluxDB 3 Enterprise**: This plugin is required for sending notifications. See the [influxdata/notifier plugin](../notifier/README.md). ## Installation steps 1. **Start InfluxDB 3 Enterprise with the Processing Engine enabled** (`--plugin-dir /path/to/plugins`): ``` influxdb3 serve \ --node-id node0 \ --object-store file \ --data-dir ~/.influxdb3 \ --plugin-dir ~/.plugins ``` 2. **Install required Python packages**: ``` influxdb3 install package requests ``` 3. **Optional**: For notifications, install and configure the [influxdata/notifier plugin](../notifier/README.md) ## Trigger setup ### Scheduled trigger Create a trigger for periodic threshold and deadman checks: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:10m" \ --trigger-arguments "measurement=cpu,senders=slack,field_aggregation_values=temp:avg@>=30-ERROR,window=10m,trigger_count=3,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \ threshold_scheduler ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Data write trigger Create a trigger for real-time threshold monitoring: ```bash influxdb3 create trigger \ --database mydb \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=cpu,field_conditions=temp>30-WARN:status==ok-INFO,senders=slack,trigger_count=2,slack_webhook_url=$SLACK_WEBHOOK_URL" \ threshold_datawrite ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. ### Enable triggers ```bash influxdb3 enable trigger --database mydb threshold_scheduler influxdb3 enable trigger --database mydb threshold_datawrite ``` ## Example usage ### Example 1: Basic threshold monitoring Write test data and monitor for threshold violations: ```bash # Write test data influxdb3 write \ --database sensors \ "heartbeat,host=server1 status=1" influxdb3 write \ --database sensors \ "heartbeat,host=server1 status=0" # Create and enable the trigger influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:5m" \ --trigger-arguments "measurement=heartbeat,senders=slack,window=5m,deadman_check=true,slack_webhook_url=$SLACK_WEBHOOK_URL" \ heartbeat_monitor influxdb3 enable trigger --database sensors heartbeat_monitor # Query to verify data influxdb3 query \ --database sensors \ "SELECT * FROM heartbeat ORDER BY time DESC LIMIT 5" ``` Set `SLACK_WEBHOOK_URL` to your Slack incoming webhook URL. **Expected output** When no data is received within the window, a deadman alert is sent: “CRITICAL: No heartbeat data from heartbeat between 2025-06-01T10:00:00Z and 2025-06-01T10:05:00Z” ### Example 2: Deadman monitoring Monitor for data absence and alert when no data is received: ```bash influxdb3 create trigger \ --database sensors \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:15m" \ --trigger-arguments "measurement=heartbeat,senders=sms,window=10m,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321,notification_deadman_text=CRITICAL: No heartbeat data from \$table between \$time_from and \$time_to" \ heartbeat_monitor ``` ### Multi-level threshold monitoring Monitor aggregated values with different severity levels: ```bash influxdb3 create trigger \ --database monitoring \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:5m" \ --trigger-arguments "measurement=system_metrics,senders=slack.discord,field_aggregation_values='cpu_usage:avg@>=80-WARN cpu_usage:avg@>=95-ERROR memory_usage:max@>=90-WARN',window=5m,interval=1min,trigger_count=3,slack_webhook_url=$SLACK_WEBHOOK_URL,discord_webhook_url=$DISCORD_WEBHOOK_URL" \ system_threshold_monitor ``` Set `SLACK_WEBHOOK_URL` and `DISCORD_WEBHOOK_URL` to your webhook URLs. ### Real-time field condition monitoring Monitor data writes for immediate threshold violations: ```bash influxdb3 create trigger \ --database applications \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "all_tables" \ --trigger-arguments "measurement=response_times,field_conditions=latency>500-WARN:latency>1000-ERROR:error_rate>0.05-CRITICAL,senders=http,trigger_count=1,http_webhook_url=$HTTP_WEBHOOK_URL,notification_text=[\$level] Application alert: \$field \$op_sym \$compare_val (actual: \$actual)" \ app_performance_monitor ``` Set `HTTP_WEBHOOK_URL` to your HTTP webhook endpoint. ### Combined monitoring Monitor both aggregation thresholds and deadman conditions: ```bash influxdb3 create trigger \ --database comprehensive \ --path "gh:influxdata/threshold_deadman_checks/threshold_deadman_checks_plugin.py" \ --trigger-spec "every:10m" \ --trigger-arguments "measurement=temperature_sensors,senders=whatsapp,field_aggregation_values='temperature:avg@>=35-WARN temperature:max@>=40-ERROR',window=15m,deadman_check=true,trigger_count=2,twilio_from_number=+1234567890,twilio_to_number=+0987654321" \ comprehensive_sensor_monitor ``` ## Code overview ### Files * `threshold_deadman_checks_plugin.py`: The main plugin code containing handlers for scheduled and data write triggers * `threshold_deadman_config_scheduler.toml`: Example TOML configuration for scheduled triggers * `threshold_deadman_config_data_writes.toml`: Example TOML configuration for data write triggers ### Logging Logs are stored in the trigger’s database in the `system.processing_engine_logs` table. To view logs: ```bash influxdb3 query --database YOUR_DATABASE "SELECT * FROM system.processing_engine_logs WHERE trigger_name = 'threshold_scheduler'" ``` ### Main functions #### `process_scheduled_call(influxdb3_local, call_time, args)` Handles scheduled threshold and deadman checks. Queries data within the specified window, evaluates aggregation-based conditions, and checks for data absence. #### `process_writes(influxdb3_local, table_batches, args)` Handles real-time threshold monitoring on data writes. Evaluates incoming data against configured field conditions with multi-level severity. ## Troubleshooting ### Common issues #### Issue: No alerts triggered **Solution**: Verify threshold values are appropriate for your data ranges. Check that notification channels are properly configured. Ensure the Notifier Plugin is installed and accessible. Review plugin logs for configuration errors. #### Issue: False positive alerts **Solution**: Increase `trigger_count` to require more consecutive failures. Adjust threshold values to be less sensitive. Consider longer aggregation intervals for noisy data. #### Issue: Missing deadman alerts **Solution**: Verify `deadman_check=true` is set in configuration. Check that the measurement name matches existing data. Ensure the time window is appropriate for your data frequency. #### Issue: Authentication issues **Solution**: Set `INFLUXDB3_AUTH_TOKEN` environment variable. Verify API token has required database permissions. Check Twilio credentials for SMS/WhatsApp notifications. ### Configuration formats **Aggregation conditions (scheduled)** * Format: `field:aggregation@operator value-level` * Example: `temp:avg@>=30-ERROR` * Multiple conditions: `"temp:avg@>=30-WARN humidity:min@<40-INFO"` **Field conditions (data write)** * Format: `field operator value-level` * Example: `temp>30-WARN:status==ok-INFO` * Supported operators: `>`, `<`, `>=`, `<=`, `==`, `!=` **Supported aggregations** * `avg`: Average value * `min`: Minimum value * `max`: Maximum value * `count`: Count of records * `sum`: Sum of values * `median`: Median value * `stddev`: Standard deviation * `first_value`: First value in time interval * `last_value`: Last value in time interval * `var`: Variance of values * `approx_median`: Approximate median (faster than exact median) ### Message template variables **Deadman notifications** * `$table`: Measurement name * `$time_from`: Start of checked period * `$time_to`: End of checked period **Threshold notifications (scheduled)** * `$level`: Alert severity level * `$table`: Measurement name * `$field`: Field name * `$aggregation`: Aggregation type * `$op_sym`: Operator symbol * `$compare_val`: Threshold value * `$actual`: Actual measured value * `$row`: Unique identifier **Threshold notifications (data write)** * `$level`: Alert severity level * `$field`: Field name * `$op_sym`: Operator symbol * `$compare_val`: Threshold value * `$actual`: Actual field value ### Row identification The `row` variable uniquely identifies alert contexts using format: `measurement:level:tag1=value1:tag2=value2` This ensures trigger counts are maintained independently for each unique combination of measurement, severity level, and tag values. ## Report an issue For plugin issues, see the Plugins repository [issues page](https://github.com/influxdata/influxdb3_plugins/issues). ## Find support for InfluxDB 3 Enterprise The [InfluxDB Discord server](https://discord.gg/9zaNCW2PRT) is the best place to find support for InfluxDB 3 Core and InfluxDB 3 Enterprise. For other InfluxDB versions, see the [Support and feedback](#bug-reports-and-feedback) options. #### Related * [Threshold deadman checks plugin on GitHub](https://github.com/influxdata/influxdb3_plugins/tree/main/influxdata/threshold_deadman_checks) [plugins](/influxdb3/enterprise/tags/plugins/)[processing engine](/influxdb3/enterprise/tags/processing-engine/)[python](/influxdb3/enterprise/tags/python/)[monitoring](/influxdb3/enterprise/tags/monitoring/)[thresholds](/influxdb3/enterprise/tags/thresholds/)[deadman](/influxdb3/enterprise/tags/deadman/)[alerting](/influxdb3/enterprise/tags/alerting/) | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to monitor for deadman alerts and aggregation-based conditions | | senders | string | required | Dot-separated notification channels with multi-channel notification integration | | window | string | required | Time window for periodic data presence checking | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | measurement | string | required | Measurement to monitor for real-time threshold violations in dual monitoring mode | | field_conditions | string | required | Real-time threshold conditions with multi-level alerting (INFO, WARN, ERROR, CRITICAL severity levels) | | senders | string | required | Dot-separated notification channels with multi-channel notification integration | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | field_aggregation_values | string | none | Multi-level aggregation conditions with aggregation support for avg, min, max, count, sum, median, stddev, first_value, last_value, var, and approx_median values | | deadman_check | boolean | false | Enable deadman detection to monitor for data absence and missing data streams | | interval | string | “5min” | Configurable aggregation time interval for batch processing with performance optimization | | trigger_count | number | 1 | Configurable triggers requiring multiple consecutive failures before alerting | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | influxdb3_auth_token | string | env var | InfluxDB 3 Enterprise API token with environment variable support | | notification_deadman_text | string | template | Customizable deadman alert template message with dynamic variables | | notification_threshold_text | string | template | Customizable threshold alert template message with dynamic variables | | notification_text | string | template | Customizable notification template message for data write triggers with dynamic variables | | notification_path | string | “notify” | Notification endpoint path with retry logic and exponential backoff | | port_override | number | 8181 | InfluxDB port override for notification delivery | | Parameter | Type | Default | Description | | --- | --- | --- | --- | | Parameter | Type | Default | Description | | config_file_path | string | none | TOML config file path relative to PLUGIN_DIR (required for TOML configuration) |