Skip to main content

Service discovery

Service discovery extracts all the potentially useful information from different sources, converts it to the configurations and exports them to the different destinations.

Pipeline

The service discovery pipeline has four jobs:

JobDescription
discoveryDynamically discovers monitoring targets by collecting events from kubernetes API server. It collects POD and SERVICE events.
tagDynamically add tags to discovered monitoring targets. Based on the POD and SERVICE fields and using patterns on them, one or more tags are attached to the monitoring targets.
buildDynamically creates data collection configurations for the monitored targets, using templates.
exportDynamically exports data collection configurations to allow netdata data collection plugins to use them. Data collection jobs in netdata are created and destroyed as needed.

Routing in a job and between jobs based on tags and selector.

Pipeline configuration:

name: <name>
discovery: <discovery_config>
tag: <tag_config>
build: <build_config>
export: <export_config>

Tags and selectors

Tag, build and export jobs have selector, the pipeline routes a target/config to the job only if its tags matches job selectors.

Both tags and selector are just lists of words.

A word must match the regex ^[a-zA-Z][a-zA-Z0-9=_.]*$.

Tags special cases:

  • -word: the word will be removed on tags merging.

Selectors special cases:

  • !word: should not contain the word.
  • word|word|word: should contain any word.

Discovery

Discovery job dynamically discovers targets using one of the supported service-discovery mechanisms.

Supported mechanisms:

Discovery configuration:

k8s:
- <kubernetes_discovery_config>

Kubernetes

Kubernetes discoverer retrieves targets from Kubernetes' REST API. It always stays synchronized with the cluster state.

Configuration options:

# Mandatory. Tags to add to all discovered targets.
tags: <tags>

# Mandatory. The Kubernetes role of entities that should be discovered.
role: <role>

# Optional. Discover only targets that exist on the same node as service-discovery.
# This option works only for 'pod' role and it requires MY_NODE_NAME env variable to be set.
local_mode: <boolean>

# Optional. If omitted, all namespaces are used.
namespaces:
- <namespace>

One of the following role types can be configured to discover targets:

  • pod
  • service

Pod Role

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, it generates single target. If there is no declared port it generates one target with empty Port, PortName and PortProtocol fields.

Available pod target fields:

NameTypeValue
TUIDstringNamespace_Name_ContName_PortProtocol_Port
AddressstringPodIP:Port
Namespacestringpod.metadata.namespace
Namestringpod.metadata.name
Annotationsmap[string]stringpod.metadata.annotations
Labelsmap[string]stringpod.metadata.labels
NodeNamestringpod.spec.nodeName
PodIPstringpod.status.podIP
ControllerNamestringpod.OwnerReferences.Controller.Name
ControllerKindstringpod.OwnerReferences.Controller.Kind
ContNamestringpod.spec.containers.name
Imagestringpod.spec.containers.image
Envmap[string]stringpod.spec.containers.env + pod.spec.containers.envFrom
Portstringpod.spec.containers.ports.containerPort
PortNamestringpod.spec.containers.ports.name
PortProtocolstringpod.spec.containers.ports.protocol

Service Role

The service role discovers a target for each service port for each service.

Available service target fields:

NameTypeValue
TUIDstringNamespace_Name_PortProtocol_Port
AddressstringName.Namespace.svc:Port
Namespacestringsvc.metadata.namespace
Namestringsvc.metadata.name
Annotationsmap[string]stringsvc.metadata.annotations
Labelsmap[string]stringsvc.metadata.labels
Portstringpod.spec.containers.ports.containerPort
PortNamestringpod.spec.containers.ports.name
PortProtocolstringpod.spec.containers.ports.protocol
ClusterIPstringsvc.spec.clusterIP
ExternalNamestringsvc.spec.externalName
Typestringsvc.spec.ports.type

Tag

Tag job tags targets discovered by discovery job. Its purpose is service identification.

Configuration is a list of tag rules:

- <tag_rule_config>

Tag rule configuration options:

# Mandatory. Routes targets to this tag rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to merge with the target tags if at least on of the match rules matches.
tags: <tags>

# Mandatory. Match rules, at least one should be defined.
match:
# Optional. Routes targets to this match rule with tags matching this selector.
- selector: <selector>

# Mandatory. Tags to merge with the target tags if this rule expression evaluates to true.
tags: <tags>

# Mandatory. Match expression.
expr: <expression>

Match expression evaluation result should be true or false.

Expression syntax is go-template.

Available functions

Custom functions:

  • glob reports whether arg1 matches the shell file name pattern.
  • re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Build

Build job creates configurations from targets.

Configuration is a list of build rules:

- <build_rule_config>

Build rule configuration options:

# Mandatory. Routes targets to this rule with tags matching this selector.
selector: <selector>

# Mandatory. Tags to add to all built by this rule configurations.
tags: <tags>

# Mandatory. Apply rules, at least one should be defined.
apply:
# Mandatory. Routes targets to this apply rule with tags matching this selector.
- selector: <selector>

# Optional. Tags to add to configurations built by this apply rule.
tags: <tags>

# Mandatory. Configuration template.
template: <template>

Template syntax is go-template.

Available functions

Custom functions:

  • glob reports whether arg1 matches the shell file name pattern.
  • re reports whether arg1 contains any match of the regular expression pattern.

All these functions accepts two or more arguments, returning in effect:

func(arg1, arg2) || func(arg1, arg3) || func(arg1, arg4) ...

Export

Export job exports configurations built by build job.

Supported exporters:

  • file

Export configuration:

file:
- <file_exporter_config>

File

File exporter writes configurations to a specific file.

# Mandatory. Routes configurations to this exporter with tags matching this selector.
selector: <selector>

# Mandatory. Absolute path to a file.
filename: <filename>

Troubleshooting

Service-discovery has debug mode and special stdout exporter which is enabled only when it's running from the terminal.

CLI:

Usage:
sd [OPTION]...

Application Options:
--config-file= Configuration file path
--config-map= Configuration ConfigMap (name:key)
-d, --debug Debug mode

Help Options:
-h, --help Show this help message

Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.