StatsD is a protocol and server implementation, first introduced at Etsy, to aggregate and summarize application metrics. With StatsD, applications are instrumented by developers using the libraries that already exist for the language, without caring about managing the data. The StatsD server is in charge of receiving the metrics, performing some simple processing on them, and then pushing them to the time-series database (TSDB) for long-term storage and visualization.
Netdata is a fully-functional StatsD server and TSDB implementation, so you can instantly visualize metrics by simply sending them to Netdata using the built-in StatsD server.
In this guide, we'll go through a scenario of visualizing our data in Netdata in a matter of seconds using k6, an open-source tool for automating load testing that outputs metrics to the StatsD format.
Although we'll use k6 as the use-case, the same principles can be applied to every application that supports the StatsD protocol. Simply enable the StatsD output and point it to the node that runs Netdata, which is
localhost in this case.
In general, the process for creating a StatsD collector can be summarized in 2 steps:
- Run an experiment by sending StatsD metrics to Netdata, without any prior configuration. This will create a chart per metric (called private charts) and will help you verify that everything works as expected from the application side of things.
- Make sure to reload the dashboard tab after you start sending data to Netdata.
- Create a configuration file for your app using edit-config:
sudo ./edit-config statsd.d/myapp.conf
- Each app will have it's own section in the right-hand menu.
Now, let's see the above process in detail.
- A node with the Netdata Agent installed.
- An application to instrument. For this guide, that will be k6.
The real in instrumenting an application with StatsD for you is to decide what metrics you want to visualize and how you want them grouped. In other words, you need decide which metrics will be grouped in the same charts and how the charts will be grouped on Netdata's dashboard.
Start with documentation for the particular application that you want to monitor (or the technological stack that you are using). In our case, the k6 documentation has a whole page dedicated to the metrics output by k6, along with descriptions.
If you are using StatsD to monitor an existing application, you don't have much control over these metrics. For example, k6 has a type called
trend, which is identical to timers and histograms. Thus, k6 is clearly dictating which metrics can be used as histograms and simple gauges.
On the other hand, if you are instrumenting your own code, you will need to not only decide what are the "things" that you want to measure, but also decide which StatsD metric type is the appropriate for each.
In Netdata, every metric will receive its own chart, called a
private chart. Although in the final implementation this is something that we will disable, since it can create considerable noise (imagine having 100s of metrics), it’s very handy while building the configuration file.
You can get a quick visual representation of the metrics and their type (e.g it’s a gauge, a timer, etc.).
An important thing to notice is that StatsD has different types of metrics, as illustrated in the Netdata documentation. Histograms and timers support mathematical operations to be performed on top of the baseline metric, like reporting the
average of the value.
Here are some examples of default private charts. You can see that the histogram private charts will visualize all the available operations.
Gauge private chart
Histogram private chart
Copy the following configuration into your file as a starting point.
Next, you need is to understand how to organize metrics in Netdata’s StatsD.
Netdata lets you group the metrics exposed by your instrumented application with synthetic charts.
First, create a
[dictionary] section to transform the names of the metrics into human-readable equivalents.
http_reqs are all metrics exposed by k6.
Continue this dictionary process with any other metrics you want to collect with Netdata.
Families and context are additional ways to group metrics. Families control the submenu at right-hand menu and it's a subcategory of the section. Given the metrics given by K6, we are organizing them in 2 major groups, or
k6 native metrics and
Context is a second way to group metrics, when the metrics are of the same nature but different origin. In our case, if we ran several different load testing experiments side-by-side, we could define the same app, but different context (e.g
Find more details about family and context in our documentation.
Now, having decided on how we are going to group the charts, we need to define how we are going to group metrics into different charts. This is particularly important, since we decide:
- What metrics not to show, since they are not useful for our use-case.
- What metrics to consolidate into the same charts, so as to reduce noice and increase visual correlation.
The dimension option has this syntax:
dimension = [pattern] METRIC NAME TYPE MULTIPLIER DIVIDER OPTIONS
- pattern: A keyword that tells the StatsD server the
METRICstring is actually a [simple pattern].(/libnetdata/simple_pattern/README.md). We don't simple patterns in the example, but if we wanted to visualize all the
http_reqmetrics, we could have a single dimension:
dimension = pattern 'k6.http_req*' last 1 1. Find detailed examples with patterns in our documentation.
- METRIC The id of the metric as it comes from the client. You can easily find this in the private charts above, for example:
- NAME: The name of the dimension. You can use the dictionary to expand this to something more human-readable.
- For all charts:
events: The number of events (data points) received by the StatsD server
last: The last value that the server received
- For histograms and timers:
stddev: This is helpful if you want to see different representations of the same value. You can find an example at the
[iteration_duration]above. Note that the baseline
metricis the same, but the
nameof the dimension is different, since we use the baseline, but we perform a computation on it, creating a different final metric for visualization(dimension).
- For all charts:
- MULTIPLIER DIVIDER: Handy if you want to convert Kilobytes to Megabytes or you want to give negative value. The second is handy for better visualization of send/receive. You can find an example at the packets submenu of the IPv4 Networking Section.
❕ If you define a chart, run Netdata to visualize metrics, and then add or remove a dimension from that chart, this will result in a new chart with the same name, confusing Netdata. If you change the dimensions of the chart, please make sure to also change the
nameof that chart, since it serves as the
idof that chart in Netdata's storage. (e.g http_req --> http_req_1).
It's time to assemble all the pieces together and create the synthetic charts that will consist our application dashboard in Netdata. We can do it in a few simple steps:
- Decide which metrics we want to use (we have viewed all of them as private charts). For example, we want to use
- Decide how we want organize them in different synthetic charts. For example, we want
k6.vuson their own, but
k6.http_req_connectingon the same chart.
- For each synthetic chart, we define a unique name and a human readable title.
- We decide at which
family(submenu section) we want each synthetic chart to belong to. For example, here we have defined 2 families:
- If we have multiple instances of the same metric, we can define different contexts, (Optional).
- We define a dimension according to the syntax we highlighted above.
- We define a type for each synthetic chart (line, area, stacked)
- We define the units for each synthetic chart.
Following the above steps, we append to the
k6.conf that we defined above, the following configuration:
Take note that Netdata will report the rate for metrics and counters, even if k6 or another application sends an absolute number. For example, k6 sends absolute HTTP requests with
http_reqs, but Netdat visualizes that in
To enable this StatsD configuration, restart Netdata.
At this point, you have used StatsD to gather metrics for k6, creating a whole new section in your Netdata dashboard in the process. Moreover, you can further customize the icon of the particular section, as well as the description for each chart.
To edit the section, please follow the Netdata documentation.
While the following configuration will be placed in a new file, as the documentation suggests, it is instructing to use
dashboard_info.js as a template. Open the file and see how the rest of sections and collectors have been defined.
We can then add a description for each chart. Simply find the following section in
dashboard_info.js to understand how a chart definitions are used:
Afterwards, you can open your
custom_dashboard_info.js, as suggested in the documentation linked above, and add something like the following example:
The chart is identified as
These descriptions can greatly help the Netdata user who is monitoring your application in the midst of an incident.
info field supports
html, embedding useful links and instructions in the description.
After all this hussle, not only did we illustrate how to visualize any data source in Netdata using the StatsD protocol, but we have also created a new collector in the process.
While using the same underlying collector-StatsD-every new
myapp.conf file will in essence create a new data source and dashboard section for Netdata. While Netdata will load all the configuration files by default, it will not create dashboard sections or charts, unless it start receiving data for that particular data source. This means that we can now share our collector with the rest of the Netdata community.
If you want to contribute or you need any help in developing your collector, we have a whole Forum Category dedicated to contributing to the Netdata Agent.
- Make sure you follow the contributing guide and read our Code of Conduct
- Fork the netdata/netdata repository
- Place the configuration file inside
- Add a reference in
netdata/collectors/statsd.plugin/Makefile.am. For example, if we contribute the
In this tutorial, you learned how to monitor an application using Netdata's StatsD implementation.
Netdata allows you easily visualize any StatsD metric without any configuration, since it creates a private metric per chart by default. But to make your implementation more robust, you also learned how to group metrics by family and context, and create multiple dimensions. With these tools, you can quickly instrument any application with StatsD to monitor its performance and availability with per-second metrics.