Kubelet collector

Kubelet is an agent that runs on each node in the cluster. It makes sure that containers are running in a pod.

This module will monitor one or more kubelet instances, depending on your configuration.

Metrics

All metrics have "k8s_kubelet." prefix.

Metric	Scope	Dimensions	Units
apiserver_audit_requests_rejected	global	rejected	requests/s
apiserver_storage_data_key_generation_failures	global	failures	events/s
apiserver_storage_data_key_generation_latencies	global	5µs, 10µs, 20µs, 40µs, 80µs, 160µs, 320µs, 640µs, 1280µs, 2560µs, 5120µs, 10240µs, 20480µs, 40960µs, +Inf	observes/s
apiserver_storage_data_key_generation_latencies_percent	global	5µs, 10µs, 20µs, 40µs, 80µs, 160µs, 320µs, 640µs, 1280µs, 2560µs, 5120µs, 10240µs, 20480µs, 40960µs, +Inf	percentage
apiserver_storage_envelope_transformation_cache_misses	global	cache misses	events/s
kubelet_containers_running	global	total	running_containers
kubelet_pods_running	global	total	running_pods
kubelet_pods_log_filesystem_used_bytes	global	a dimension per namespace and pod	B
kubelet_runtime_operations	global	a dimension per operation type	operations/s
kubelet_runtime_operations_errors	global	a dimension per operation type	errors/s
kubelet_docker_operations	global	a dimension per operation type	operations/s
kubelet_docker_operations_errors	global	a dimension per operation type	errors/s
kubelet_node_config_error	global	experiencing_error	bool
kubelet_pleg_relist_interval_microseconds	global	0.5, 0.9, 0.99	microseconds
kubelet_pleg_relist_latency_microseconds	global	0.5, 0.9, 0.99	microseconds
kubelet_token_requests	global	total, failed	token_requests/s
rest_client_requests_by_code	global	a dimension per HTTP status code	requests/s
rest_client_requests_by_method	global	a dimension per HTTP method	requests/s
volume_manager_total_volumes	volume manager	actual, desired	state

Configuration

Edit the go.d/k8s_kubelet.conf configuration file using edit-config from the Netdata config directory, which is typically at /etc/netdata.

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/k8s_kubelet.conf

Needs only url to kubelet metric-address. Here is an example for 2 instances:

jobs:
  - name: local
    url: http://127.0.0.1:10255/metrics

  - name: remote
    url: http://203.0.113.10:10255/metrics

For all available options please see module configuration file.

Troubleshooting

To troubleshoot issues with the k8s_kubelet collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/

• Switch to the netdata user.

  sudo -u netdata -s

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m k8s_kubelet