StackPulse agent alert notifications

StackPulse is a software-as-a-service platform for site reliability engineering. It helps SREs, DevOps Engineers and Software Developers reduce toil and alert fatigue while improving reliability of software services by managing, analyzing and automating incident response activities.

Sending Netdata alert notifications to StackPulse allows you to create smart automated response workflows (StackPulse playbooks) that will help you drive down your MTTD and MTTR by performing any of the following:

• Enriching the incident with data from multiple sources
• Performing triage actions and analyzing their results
• Orchestrating incident management and notification flows
• Performing automatic and semi-automatic remediation actions
• Analyzing incident data and remediation patterns to improve reliability of your services

To send the notification you need:

1. Create a Netdata integration in the StackPulse Administration Portal, and copy the Endpoint URL.

2. On your node, navigate to /etc/netdata/ and run the following command:

$ ./edit-config health_alarm_notify.conf

3. Set the STACKPULSE_WEBHOOK variable to Endpoint URL you copied earlier:

SEND_STACKPULSE="YES"
STACKPULSE_WEBHOOK="https://hooks.stackpulse.io/v1/webhooks/YOUR_UNIQUE_ID"

4. Now restart Netdata using sudo systemctl restart netdata, or the appropriate method for your system. When your node creates an alert, you can see the associated notification on your StackPulse Administration Portal

React to alerts with playbooks

StackPulse allow users to create Playbooks giving additional information about events that happen in specific scenarios. For example, you could create a Playbook that responds to a "low disk space" alert by compressing and cleaning up storage partitions with dynamic data.

Create Playbooks for Netdata alerts

To create a Playbook, you need to access the StackPulse Administration Portal. After the initial setup, you need to access the TRIGGER tab to define the scenarios used to trigger the event. The following variables are available:

• Hostname: The host that generated the event.
• Chart: The name of the chart.
• OldValue : The previous value of the alert.
• Value: The current value of the alert.
• Units : The units of the value.
• OldStatus : The previous status: REMOVED, UNINITIALIZED, UNDEFINED, CLEAR, WARNING, CRITICAL.
• State: The current alert status, the acceptable values are the same of OldStatus.
• Alarm : The name of the alert, as given in Netdata's health.d entries.
• Date : The timestamp this event occurred.
• Duration : The duration in seconds of the previous alert state.
• NonClearDuration : The total duration in seconds this is/was non-clear.
• Description : A short description of the alert copied from the alert definition.
• CalcExpression : The expression that was evaluated to trigger the alert.
• CalcParamValues : The values of the parameters in the expression, at the time of the evaluation.
• TotalWarnings : Total number of alerts in WARNING state.
• TotalCritical : Total number of alerts in CRITICAL state.
• ID : The unique id of the alert that generated this event.

For more details how to create a scenario, take a look at the StackPulse documentation.