In the fifth step of the Netdata guide, we're introducing you to one of our core features: health monitoring.
To accurately monitor the health of your systems and applications, you need to know immediately when there's something strange going on. Netdata's alarm and notification systems are essential to keeping you informed.
Netdata comes with hundreds of pre-configured alarms that don't require configuration. They were designed by our community of system adminstrators to cover the most important parts of production systems, so, in many cases, you won't need to edit them.
Luckily, Netdata's alarm and notification system are incredibly adaptable to your infrastructure's unique needs.
We'll talk about Netdata's default configuration, and then you'll learn how to do the following:
- Tune Netdata's pre-configured alarms
- Write your first health entity
- Enable Netdata's notification systems
First, let's tune an alarm that came pre-configured with your Netdata installation.
The first chart you see on any Netdata dashboard is the
system.cpu chart, which shows the system's CPU utilization
across all cores. To figure out which file you need to edit to tune this alarm, click the Alarms button at the top
of the dashboard, click on the All tab, and find the system - cpu alarm entity.
Look at the
source row in the table. This means the
system.cpu chart sources its health alarms from
[email protected]/usr/lib/netdata/conf.d/health.d/cpu.conf. To tune these alarms, you'll need to edit the alarm file at
health.d/cpu.conf. Go to your Netdata config directory and use the
The first health entity in that file looks like this:
Let's say you want to tune this alarm to trigger warning and critical alarms at a lower CPU utilization. You can change
crit lines to the values of your choosing. For example:
You can also tune any other aspect of the default alarms. To better understand how each line in a health entity works, read our health documentation.
Many Netdata users don't need all the default alarms enabled. Instead of disabling any given alarm, or even all
alarms, you can silence individual alarms by changing one line in a given health entity. Let's look at that
health/cpu.conf file again.
To silence this alarm, change
netdatacli reload-health to reload your health configuration. You can add
to: silent to any alarm you'd rather not
bother you with notifications.
The best way to understand how health entities work is building your own and experimenting with the options. To start, let's build a health entity that triggers an alarm when system RAM usage goes above 80%.
The first line in a health entity will be
alarm:. This is how you name your entity. You can give it any name you
choose, but the only symbols allowed are
_. Let's call the alarm
You'll see some funky indentation in the lines coming up. Don't worry about it too much! Indentation is not important to how Netdata processes entities, and it will make sense when you're done.
Next, you need to specify which chart this entity listens via the
on: line. You're declaring that you want this alarm
to check metrics on the
Now comes the
lookup. This line specifies what metrics the alarm is looking for, what duration of time it's looking
at, and how to process the metrics into a more usable format.
Let's take a moment to break this line down.
average: Calculate the average of all the metrics collected.
-1m: Use metrics from 1 minute ago until now to calculate that average.
percentage: Clarify that you want to calculate a percentage of RAM usage.
of used: Specify which dimension (
used) on the
system.ramchart you want to monitor with this entity.
In other words, you're taking 1 minute's worth of metrics from the
used dimension on the
calculating their average, and returning it as a percentage.
You can move on to the
units line, which lets Netdata know that we're working with a percentage and not an absolute
every line tells Netdata how often to perform the calculation you specified in the
lookup line. For
certain alarms, you might want to use a shorter duration, which you can specify using values like
We'll put the next two lines—
crit—together. In these lines, you declare at which percentage you want to
trigger a warning or critical alarm. Notice the variable
$this, which is the value calculated by the
These lines will trigger a warning if that average RAM usage goes above 80%, and a critical alert if it's above 90%.
❗ Most default Netdata alarms come with more complicated
critlines. You may have noticed the line
warn: $this > (($status >= $WARNING) ? (75) : (85))in one of the health entity examples above, which is an example of using the conditional operator for hysteresis. Hysteresis is used to keep Netdata from triggering a ton of alerts if the metric being tracked quickly goes above and then falls below the threshold. For this very simple example, we'll skip hysteresis, but recommend implementing it in your future health entities.
Finish off with the
info line, which creates a description of the alarm that will then appear in any
notification you set up. This line is optional, but it has value—think of it as
documentation for a health entity!
Here's what the entity looks like in full. Now you can see why we indented the lines, too.
What about what it looks like on the Netdata dashboard?
If you'd like to try this alarm on your system, you can install a small program called
stress to create a synthetic load. Use the command
below, and change the
8G value to a number that's appropriate for the amount of RAM on your system.
Netdata is capable of understanding much more complicated entities. To better understand how they work, read the health documentation, look at some examples, and open the files containing the default entities on your system.
Health alarms, while great on their own, are pretty useless without some way of you knowing they've been triggered. That's why Netdata comes with a notification system that supports more than a dozen services, such as email, Slack, Discord, PagerDuty, Twilio, Amazon SNS, and much more.
To see all the supported systems, visit our notifications documentation.
We'll cover email and Slack notifications here, but with this knowledge you should be able to enable any other type of notifications instead of or in addition to these.
To use email notifications, you need
sendmail or an equivalent installed on your system. Linux systems use
or similar programs to, unsurprisingly, send emails to any inbox.
Learn more about
sendmailvia its documentation.
health_alarm_notify.conf file, which resides in your Netdata directory.
Look for the following lines:
Change the value of
DEFAULT_RECIPIENT_EMAIL to the email address at which you'd like to receive notifications.
Test email notifications system by first becoming the Netdata user and then asking Netdata to send a test alarm:
You should see output similar to this:
... and you should get three separate emails, one for each test alarm, in your inbox! (Be sure to check your spam folder.)
If you're one of the many who spend their workday getting pinged with GIFs by your colleagues, why not add Netdata notifications to the mix? It's a great way to immediately see, collaborate around, and respond to anomalies in your infrastructure.
To get Slack notifications working, you first need to add an incoming webhook to the channel of your choice. Click the green Add to Slack button, choose the channel, and click the Add Incoming WebHooks Integration button.
On the following page, you'll receive a Webhook URL. That's what you'll need to configure Netdata, so keep it handy.
Time to dive back into your
Look for the
SLACK_WEBHOOK_URL=" " line and add the incoming webhook URL you got from Slack:
A few lines down, edit the
DEFAULT_RECIPIENT_SLACK line to contain a single hash
# character. This instructs Netdata
to send a notification to the channel you configured with the incoming webhook.
Time to test the notifications again!
You should receive three notifications in your Slack channel.
Congratulations! You're set up with two awesome ways to get notified about any change in the health of your systems or applications.
To further configure your email or Slack notification setup, or to enable other notification systems, check out the following documentation:
In this step, you learned the fundamentals of Netdata's health monitoring tools: alarms and notifications. You should be able to tune default alarms, silence them, and understand some of the basics of writing health entities. And, if you so chose, you'll now have both email and Slack notifications enabled.
You're coming along quick!
Next up, we're going to cover how Netdata collects its metrics, and how you can get Netdata to collect real-time metrics from hundreds of services with almost no configuration on your part. Onward!