Service Discovery

Stop hand-writing collector jobs for every host, container, or device. Netdata's Service Discovery (SD) finds monitorable targets in your environment and turns them into collector jobs automatically.

Each SD pipeline is a discoverer: (where to look) plus a list of services: rules (how to turn what was found into collector jobs). The discoverer emits targets; the rules create collector jobs from those targets using Go templates.

Jump To

How it works • Configuration file structure • Testing configurations • Rule evaluation semantics • Template helper reference • config_template rendering • Supported discoverers • Mixing discoverers • Troubleshooting

How it works

Each discovery pipeline runs in five stages:

1. Discover — The discoverer probes its environment for monitorable things — running containers, listening sockets, kubernetes resources, SNMP devices on a subnet, items returned by an HTTP endpoint. What gets probed and how often is controlled by discoverer: options.
2. Build a target — Each discovered thing becomes a target. The target carries discoverer-specific fields (variables) — for example, net_listeners targets have .Port / .Comm / .Cmdline; docker targets have .Image / .Name / .Labels; snmp targets have .IPAddress / .SysInfo.* / .Credential.*. Per-discoverer pages list the full variable set.
3. Match against services rules — The rule engine evaluates each services: rule against each target, top to bottom. A rule's match expression is a Go template that must render to the literal string "true" for the rule to apply.
4. Render the collector job — When a rule matches, its config_template is executed with the target as context. Each call to job submits one collector job. Existing templates that render YAML maps or sequences remain supported.
5. Hand off to the collector — The agent registers the rendered jobs with the matching collector module. From this point on, the collector runs the job — the discoverer is no longer involved.

Configuration file structure

Each discoverer has its own file under /etc/netdata/go.d/sd/. The filename determines the discoverer kind (net_listeners.conf, docker.conf, http.conf, snmp.conf, k8s.conf).

Every SD file has the same shape:

disabled: no                  # set to yes to disable this pipeline
trust_discovered_targets: no  # keep secret-reference syntax literal in generated jobs

discoverer:
  <kind>:                     # discoverer kind (must match filename)
    # discoverer-specific options — see the per-discoverer page

services:
  - id: <rule-id>
    match: <go-template>      # must render to the literal string "true"
    config_template: |        # optional — omit to make this a "skip rule"
      # job calls, evaluated with target as context

• disabled: yes keeps the file on disk but turns the pipeline off.
• trust_discovered_targets defaults to no. Setting it to yes lets generated collector jobs resolve ${env:...}, ${file:...}, ${cmd:...} and ${store:...}, including references in target-controlled values copied by service rules. Those references can make Netdata read local files, run commands, or fetch stored secrets with the secret providers' permissions; collectors may send the results back to a target.
• Enable only if you control and trust every target this pipeline can discover. Enabling accepts that risk for the whole pipeline: one untrusted target is enough to abuse it. Other pipelines remain unchanged.
• Trust the inputs used by your service rules: Docker container metadata and commands, local process command lines, Kubernetes workload labels/annotations/environment, HTTP response items, or SNMP device system information. Custom rules can expose additional target-controlled values. The option does not resolve credentials used by the discoverer itself; SNMP scanning still requires literal credentials.
• Editing a stock file requires restarting the agent. UI-managed pipelines apply live.
• Where each discoverer's stock conf ships (with the Netdata package, with the Helm chart, or not at all) is documented on its per-discoverer page.

Testing configurations

Testing a UI-managed pipeline builds a complete temporary pipeline but does not publish targets, install jobs, or make the configuration persistent. The response says explicitly when only configuration validation was possible.

Operational guarantees depend on the discoverer. Docker runs one bounded container-list query. Local-listener discovery runs and parses one helper snapshot. HTTP discovery runs one complete production fetch only when method is empty/default or exactly GET; it uses the configured authentication, headers, TLS, proxy, redirects, and timeout, requires HTTP 200, enforces the 10 MiB response limit, and parses every returned item. Redirect handling can issue at most 10 requests.

The HTTP test discards the parsed targets. It does not evaluate services: rules, render or validate collector jobs, publish targets, or install jobs. A non-empty HTTP method other than exact GET, plus Kubernetes and SNMP, is intentionally validation-only.

Rule evaluation semantics

For each discovered target, the engine walks the services: array top-to-bottom. The two rule shapes behave differently:

• Skip rule (no config_template) — When a skip rule matches, the target is dropped immediately — no jobs are produced and no further rules run for that target. Use skip rules to exclude targets the catch-all would otherwise pick up. Place them before any template rule.
• Template rule (with config_template) — When a template rule matches, its config_template is rendered into one or more collector jobs and rule evaluation continues with the next rule. A single target can therefore produce jobs from multiple matching template rules.

Order matters

Recommended ordering for a multi-rule pipeline:

1. Skip rules — drop targets you don't want monitored.
2. Specific template rules — vendor-specific, label-specific, port-specific.
3. A catch-all template rule (match: '{{ true }}') — the default fallback.

If a specific template rule already produced the right job, follow it with a skip rule keyed on the same condition to suppress the catch-all for those targets.

Template helper reference

Match expressions and config templates are Go text/template bodies with three additional helper sets: the standard Go template builtins, the sprig function library, and a small set of Netdata-specific helpers.

All templates run with missingkey=error. Referencing a field that does not exist on the target type (e.g. typo .SysInfoo.Name instead of .SysInfo.Name) makes the template execution fail; the agent logs the error and skips that rule for that target.

Go template builtins

Standard Go text/template syntax: pipelines, conditionals, loops, variable assignment, whitespace control.

Construct	Description
if/else if/else/end	Conditional rendering.
range/end	Iterate over a slice or map.
with/end	Set the dot to a value if it is non-empty.
{{- ... -}}	Whitespace-trim left/right around the action.
{{ $var := ... }}	Variable assignment, scoped to the enclosing block.
eq, ne, lt, le, gt, ge	Comparison. eq A B C ... is true if A equals any of the following arguments.
and, or, not	Boolean composition (variadic).
index	Index a slice or map: index .Labels "app".
printf	Formatted string.

Sprig functions

The full sprig library is included. The functions most commonly used in stock SD configs:

Function	Description
default DEFAULT VALUE	Return VALUE if non-empty; otherwise DEFAULT.
empty VALUE	True if the value is the zero value for its type.
hasKey MAP KEY	True if MAP contains KEY. Used heavily by the http discoverer.
kindIs KIND VALUE	True if VALUE's reflect kind matches: string, map, slice, bool, …
lower S / upper S	Lowercase / uppercase a string.
trim S / trimPrefix PREFIX S / trimSuffix SUFFIX S	Whitespace and prefix/suffix trimming.
replace OLD NEW S	String replace.
regexFind RE S	Return the first regex match, or empty.
regexMatch RE S	True if S matches RE (substring match unless anchored).
printf FMT V...	Same as Go's fmt.Sprintf.

See the sprig docs for the full list (string, math, encoding, list, dict, date helpers).

Netdata helpers

Custom helpers added to the SD template engine:

• match TYPE VALUE PATTERN [PATTERN...] — Returns the boolean true if VALUE matches any of the patterns under the named matcher type. TYPE is one of:
  • "glob" — shell glob (*, ?, [abc]).
  • "sp" — Netdata simple patterns (space-separated, * wildcard, ! for negation).
  • "re" — RE2 regular expression. Matches if the regex matches anywhere in VALUE unless explicitly anchored with ^ / $.
  • "dstar" — doublestar glob (supports ** for path-style matching).
• glob VALUE PATTERN [PATTERN...] — Shortcut for match "glob" VALUE PATTERN....
• promPort PORT — Returns the well-known Prometheus exporter module name registered for PORT, or the empty string. net_listeners-specific.
• job KEY VALUE ... or job MAP — Submit one complete collector job, capturing a detached copy of its values at the call. Returns empty text; encoding is automatic. Available only in config_template, not in match expressions.
• toYaml VALUE — Serialize a value as YAML text. Retained for existing templates; use job to create collector jobs with automatic encoding.

Notes:

• match and glob are case-sensitive.
• match "re" ... is unanchored. Use ^...$ to require a full-string match.
• There is no standalone regexp or regex helper. Use match "re" (or sprig's regexFind / regexMatch) for regex matching.
• match and glob return booleans and can be composed with if, and, and or. The rule engine compares the rendered match template with the literal text true.

config_template rendering

By default, discovered jobs treat ${env:...}, ${file:...}, ${cmd:...}, and ${store:...} as literal text, even when a reference comes from an operator-authored discovery rule or credential. A pipeline with trust_discovered_targets: yes allows these references in all its generated collector jobs. Alternatively, adopt an individual collector job through Dynamic Configuration and review its complete configuration before saving.

When a template rule matches, its config_template is executed with the target as the dot context. Create each collector job with job, for example {{ job "name" .Name "url" (printf "http://%s" .Address) }}. Encoding is automatic: no toYaml pipe or whitespace trimming is needed.

Pass alternating string keys and values, or one map such as {{ job .Item }} for HTTP pass-through. String values retain their exact contents, including YAML punctuation and trailing newlines. Keep numbers and booleans typed, for example (atoi .Port) for a numeric port. Missing values, duplicate keys, non-string keys, nil maps, and unsupported argument shapes fail the rule.

For conditional fields, start with {{ $cfg := dict "name" .Name }}, assign fields with {{ $_ := set $cfg "key" VALUE }}, and submit with {{ job $cfg }}. Assigning the result of set to $_ prevents it from printing the dictionary. Each call captures a detached snapshot, so later map changes do not alter jobs already submitted.

Existing user files and saved Dynamic Configuration rules are not rewritten during an upgrade. Templates that render raw YAML, including those using toYaml, remain supported. For rules that interpolate target values into YAML, migrate to job; metadata normalization and literal quotation marks do not safely encode those values.

• One call → one job — Each job call submits one map. Call it repeatedly or inside range to create multiple jobs in call order. If no calls execute and the template produces only whitespace, it creates no jobs.

• Multiple jobs — This local-listener rule emits Unix-socket and TCP MySQL jobs in order:

  - id: mysql
    match: '{{ or (eq .Port "3306") (eq .Comm "mysqld") }}'
    config_template: |
      {{ job "name" "local" "dsn" "netdata@unix(/var/run/mysqld/mysqld.sock)/" }}
      {{ job "name" "local" "dsn" (printf "netdata@tcp(%s)/" .Address) }}

• Legacy YAML templates — When no job call executes, rendered YAML is parsed as before: a map creates one job, and a sequence of maps creates one job per element. Do not mix submitted jobs with non-whitespace text in the same execution. That includes literal YAML, YAML comments, and accidentally printed values such as the result of set.

• Module inference from rule id — If the rendered job map has no module key, the rule's id is used as the module name. Use the collector module name, such as snmp or prometheus, as the rule ID to omit module from the dictionary; otherwise set it explicitly.

• id uniqueness — Rule IDs are not required to be unique across rules — multiple rules can share an id. The id is used for module inference and shows up in agent logs to help you identify which rule produced a job. Pick descriptive IDs (cisco, hp-printers, skip-vips).

• Failure handling — Template errors, invalid job calls, mixed text/job output, and legacy YAML parse errors are logged at warn level. All jobs from the failing rule are discarded, including earlier calls in that execution. The agent continues with the next rule for that target.

Supported discoverers

Each discoverer has its own page covering its options, target variables, evaluation specifics, and worked examples.

Discoverer	Kind	Stock conf	Discovers
Docker	docker	/etc/netdata/go.d/sd/docker.conf	Running containers on the local Docker daemon.
HTTP endpoint	http	/etc/netdata/go.d/sd/http.conf	Items returned by an HTTP/HTTPS endpoint (JSON or YAML).
Kubernetes	k8s	/etc/netdata/go.d/sd/k8s.conf	Pods and services in a Kubernetes cluster.
Local listening processes	net_listeners	/etc/netdata/go.d/sd/net_listeners.conf	Local processes that listen on TCP/UDP ports.
SNMP	snmp	/etc/netdata/go.d/sd/snmp.conf	SNMP-capable devices on configured network subnets.

Mixing discoverers

All discoverers can run simultaneously. Each /etc/netdata/go.d/sd/<kind>.conf is independent. The same target can theoretically be discovered by more than one discoverer (for example, a containerised application appears in both docker and net_listeners); each discoverer's pipeline is independent and may produce its own job.

Use disabled: yes at the top of a stock file to keep it on disk but turn the pipeline off.

UI-managed pipelines and file-based pipelines coexist. UI-managed pipelines apply live; file-based pipelines require an agent restart to reload.

Troubleshooting

Common cross-discoverer problems. For discoverer-specific issues, see the per-discoverer page.

No targets discovered

Check the agent log for discoverer=<kind> lines. Confirm disabled: no and that the discoverer's prerequisites are met (network reachability, credentials, API access).

Targets discovered but no jobs created

Check that at least one services: rule has both a matching match expression and a config_template. A rule with no config_template is a skip rule — it drops the target instead of producing a job.

match always evaluates false

Match expressions must render to the literal string "true". A bare {{ if ... }} block that does not output anything renders to the empty string, which is treated as false. Use {{ true }} for catch-all, {{ glob .X "..." }} for pattern matches, or {{ if ... }}true{{ end }} for ad-hoc conditions.

Template render error

Look for failed to execute services[N]->config_template on target in the log. The most common cause is a typo in a variable reference (e.g. .SysInfoo.Name) — missingkey=error rejects unknown fields. Use the per-discoverer page's variable table to verify spelling.

YAML parse error after rendering

failed to parse services[N] template data means a legacy template's output cannot be parsed as job YAML. Common cause: raw target text contains a colon, hash, or line break. Use {{ job "name" .Name "url" .Address }} so encoding is automatic. For cannot mix submitted jobs with rendered text, remove literal YAML and suppress printed values such as the result of set.