Skip to main content
Version: nightly

Prometheus endpoint monitoring with Netdata

The generic Prometheus endpoint collector gathers metrics from Prometheus endpoints that use the OpenMetrics exposition format.

  • As of v1.24, Netdata can autodetect more than 600 Prometheus endpoints, including support for Windows 10 via windows_exporter, and instantly generate new charts with the same high-granularity, per-second frequency as you expect from other collectors.

  • The full list of endpoints is available in the collector's configuration file.

  • Collecting metrics from Prometheus endpoints in Kubernetes.

Charts

Netdata will produce one or more charts for every metric collected via a Prometheus endpoint. The number of charts depends entirely on the number of exposed metrics.

For example, scraping node_exporter produces 3000+ metrics.

Configuration

Edit the go.d/prometheus.conf configuration file using edit-config from the Netdata config directory, which is typically at /etc/netdata.

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/prometheus.conf

Endpoints can be either local or remote as long as they provide their metrics at the provided URL.

Here is an example with two endpoints:

jobs:
- name: node_exporter_local
url: http://127.0.0.1:9100/metrics

- name: win10
url: http://203.0.113.0:9182/metrics

Here is an example pulling from the Prometheus demo site node exporter endpoint:

jobs:
- name: node_exporter_demo
url: https://node.demo.do.prometheus.io/metrics

For all available options, see the Prometheus collector's configuration file.

Time Series Selector (filtering)

To filter unwanted time series (metrics) use selector configuration option.

Here is an example:

jobs:
- name: node_exporter_local
url: http://127.0.0.1:9100/metrics
# (allow[0] || allow[1] || ...) && !(deny[0] || deny[1] || ...)
selector:
allow:
- <PATTERN>
- <PATTERN>
deny:
- <PATTERN>
- <PATTERN>

To find PATTERN syntax description and more examples see selectors readme.

Time Series Grouping

It has built-in grouping logic based on the type of metrics.

MetricChartDimension(s)Algorithm
Gaugefor each label setone, the metric nameabsolute
Counterfor each label setone, the metric nameincremental
Summary (quantiles)for each label set (excluding 'quantile')for each quantileabsolute
Summary (sum and count)for each label setthe metric nameincremental
Histogram (buckets)for each label set (excluding 'le')for each bucketincremental
Histogram (sum and count)for each label setthe metric nameincremental

Untyped metrics (have no '# TYPE') processing:

  • As Counter if it has suffix '_total'.
  • As Summary if it has 'quantile' label.
  • As Histogram if it has 'le' label.

The rest are ignored.

Troubleshooting

To troubleshoot issues with the prometheus collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

  • Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

    cd /usr/libexec/netdata/plugins.d/
  • Switch to the netdata user.

    sudo -u netdata -s
  • Run the go.d.plugin to debug the collector:

    ./go.d.plugin -d -m prometheus

Was this page helpful?

Contribute