Dynatrace

Dynatrace allows you to receive notifications using their Events REST API. See the Dynatrace documentation about POSTing an event in the Events API for more details. You can send notifications to Dynatrace using Netdata's Agent alert notification feature, which supports dozens of endpoints, user roles, and more.

Setup

Prerequisites

• A Dynatrace Server. You can use the same on all your Netdata servers but make sure the server is network visible from your Netdata hosts. The Dynatrace server should be with protocol prefixed (http:// or https://), for example: https://monitor.example.com.
• An API Token. Generate a secure access API token that enables access to your Dynatrace monitoring data via the REST-based API. See Dynatrace API - Authentication for more details.
• An API Space. This is the URL part of the page you have access in order to generate the API Token. For example, the URL for a generated API token might look like: https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all In that case, the Space is 2a93fe0e-4cd5-469a-9d0d-1a064235cfce.
• A Server Tag. To generate one on your Dynatrace Server, go to Settings --> Tags --> Manually applied tags and create the Tag. The Netdata alarm is sent as a Dynatrace Event to be correlated with all those hosts tagged with this Tag you have created.
• Terminal access to the Agent you wish to configure

Configuration

File

The configuration file name for this integration is health_alarm_notify.conf.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config health_alarm_notify.conf

Options

The following options can be defined for this notification

Config Options

Name	Description	Default	Required
SEND_DYNATRACE	Set SEND_DYNATRACE to YES	YES	yes
DYNATRACE_SERVER	Set DYNATRACE_SERVER to the Dynatrace server with the protocol prefix, for example https://monitor.example.com.		yes
DYNATRACE_TOKEN	Set DYNATRACE_TOKEN to your Dynatrace API authentication token		yes
DYNATRACE_SPACE	Set DYNATRACE_SPACE to the API Space, it is the URL part of the page you have access in order to generate the API Token.		yes
DYNATRACE_TAG_VALUE	Set DYNATRACE_TAG_VALUE to your Dynatrace Server Tag.		yes
DYNATRACE_ANNOTATION_TYPE	DYNATRACE_ANNOTATION_TYPE can be left to its default value Netdata Alarm, but you can change it to better fit your needs.	Netdata Alarm	no
DYNATRACE_EVENT	Set DYNATRACE_EVENT to the Dynatrace eventType you want.	Netdata Alarm	no

DYNATRACE_SPACE

For example, the URL for a generated API token might look like: https://monitor.illumineit.com/e/2a93fe0e-4cd5-469a-9d0d-1a064235cfce/#settings/integration/apikeys;gf=all In that case, the Space is 2a93fe0e-4cd5-469a-9d0d-1a064235cfce.

DYNATRACE_EVENT

AVAILABILITY_EVENT, CUSTOM_ALERT, CUSTOM_ANNOTATION, CUSTOM_CONFIGURATION, CUSTOM_DEPLOYMENT, CUSTOM_INFO, ERROR_EVENT, MARKED_FOR_TERMINATION, PERFORMANCE_EVENT, RESOURCE_CONTENTION_EVENT. You can read more here.

Examples

Basic Configuration

#------------------------------------------------------------------------------
# Dynatrace global notification options

SEND_DYNATRACE="YES"
DYNATRACE_SERVER="https://monitor.example.com"
DYNATRACE_TOKEN="XXXXXXX"
DYNATRACE_SPACE="2a93fe0e-4cd5-469a-9d0d-1a064235cfce"
DYNATRACE_TAG_VALUE="SERVERTAG"
DYNATRACE_ANNOTATION_TYPE="Netdata Alert"
DYNATRACE_EVENT="AVAILABILITY_EVENT"

Troubleshooting

Test Notification

You can run the following command by hand, to test alerts configuration:

# become user netdata
sudo su -s /bin/bash netdata

# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1

# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test

# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"

Note that this will test all alert mechanisms for the selected role.