Kubernetes Cluster State

Plugin: go.d.plugin Module: k8s_state

Overview

This collector monitors Kubernetes Nodes, Pods and Containers.

This collector is supported on all platforms.

This collector only supports collecting metrics from a single instance of this integration.

Kubernetes Cluster State can be monitored further using the following other integrations:

• Kubernetes API Server
• Kubelet
• Kubeproxy
• Kubernetes Containers
• CoreDNS

Default Behavior

Auto-Detection

This integration doesn't support auto-detection.

Limits

The default configuration for this integration does not impose any limits on data collection.

Performance Impact

The default configuration for this integration is not expected to impose a significant performance impact on the system.

Setup

You can configure the k8s_state collector in two ways:

Method	Best for	How to
UI	Fast setup without editing files	Go to Nodes → Configure this node → Collectors → Jobs, search for k8s_state, then click + to add a job.
File	If you prefer configuring via file, or need to automate deployments (e.g., with Ansible)	Edit go.d/k8s_state.conf and add a job.

important

UI configuration requires paid Netdata Cloud plan.

Prerequisites

No action required.

Configuration

Options

There are no configuration options.

via UI

Configure the k8s_state collector from the Netdata web interface:

1. Go to Nodes.
2. Select the node where you want the k8s_state data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
3. The Collectors → Jobs view opens by default.
4. In the Search box, type k8s_state (or scroll the list) to locate the k8s_state collector.
5. Click the + next to the k8s_state collector to add a new job.
6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
   • Test runs the job with the provided settings and shows whether data can be collected.
   • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File

The configuration file name for this integration is go.d/k8s_state.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
  - name: some_name1
  - name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/k8s_state.conf

Examples

There are no configuration examples.

Alerts

The following alerts are available:

Alert name	On metric	Description
Alert name k8s_state_deployment_condition_available	On metrick8s_state.deployment_conditions	DescriptionDeployment ${label:k8s_deployment_name} does not have the minimum required replicas
Alert name k8s_state_cronjob_last_execution_failed	On metrick8s_state.cronjob_last_execution_status	DescriptionCronJob ${label:k8s_cronjob_name} in ${label:k8s_namespace} failing

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Per node

These metrics refer to the Node.

Labels:

Label	Description
k8s_cluster_id	Cluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_name	Cluster name. Cluster name discovery only works in GKE.
k8s_node_name	Node name.

Metrics:

Metric	Description	Dimensions	Unit
k8s_state.node_allocatable_cpu_requests_utilization	CPU requests utilization	requests	%
k8s_state.node_allocatable_cpu_requests_used	CPU requests used	requests	millicpu
k8s_state.node_allocatable_cpu_limits_utilization	CPU limits utilization	limits	%
k8s_state.node_allocatable_cpu_limits_used	CPU limits used	limits	millicpu
k8s_state.node_allocatable_mem_requests_utilization	Memory requests utilization	requests	%
k8s_state.node_allocatable_mem_requests_used	Memory requests used	requests	bytes
k8s_state.node_allocatable_mem_limits_utilization	Memory limits utilization	limits	%
k8s_state.node_allocatable_mem_limits_used	Memory limits used	limits	bytes
k8s_state.node_allocatable_pods_utilization	Pods resource utilization	allocated	%
k8s_state.node_allocatable_pods_usage	Pods resource usage	available, allocated	pods
k8s_state.node_condition	Condition status	Ready, DiskPressure, MemoryPressure, NetworkUnavailable, PIDPressure	status
k8s_state.node_schedulability	Schedulability	schedulable, unschedulable	state
k8s_state.node_pods_readiness	Pods readiness	ready	%
k8s_state.node_pods_readiness_state	Pods readiness state	ready, unready	pods
k8s_state.node_pods_condition	Pods condition	pod_ready, pod_scheduled, pod_initialized, containers_ready	pods
k8s_state.node_pods_phase	Pods phase	running, failed, succeeded, pending	pods
k8s_state.node_containers	Containers	containers, init_containers	containers
k8s_state.node_containers_state	Containers state	running, waiting, terminated	containers
k8s_state.node_init_containers_state	Init containers state	running, waiting, terminated	containers
k8s_state.node_age	Age	age	seconds

Per deployment

These metrics refer to Deployments.

Labels:

Label	Description
k8s_cluster_id	Cluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_name	Cluster name. Cluster name discovery only works in GKE.
k8s_deployment_name	Deployment name.
k8s_namespace	Namespace.

Metrics:

Metric	Description	Dimensions	Unit
k8s_state.deployment_conditions	Deployment Conditions	available, replica_failure, progressing	status
k8s_state.deployment_replicas	Deployment Replicas	desired, current, ready	replicas
k8s_state.deployment_age	Deployment Age	age	seconds

Per cronjob

These metrics refer to CronJobs.

Labels:

Label	Description
k8s_cluster_id	Cluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_name	Cluster name. Cluster name discovery only works in GKE.
k8s_cronjob_name	CronJob name.
k8s_namespace	Namespace.

Metrics:

Metric	Description	Dimensions	Unit
k8s_state.cronjob_jobs_count_by_status	CronJob Jobs Count by Status	completed, failed, running, suspended	jobs
k8s_state.cronjob_jobs_failed_by_reason	CronJob Jobs Failed by Reason	pod_failure_policy, backoff_limit_exceeded, deadline_exceeded	jobs
k8s_state.cronjob_last_execution_status	CronJob Last Execution Status	completed, failed	status
k8s_state.cronjob_last_completion_duration	CronJob Last Completion Duration	last_completion	seconds
k8s_state.cronjob_last_completed_time_ago	CronJob Last Completed Time Ago	last_completed_ago	seconds
k8s_state.cronjob_last_schedule_time_ago	CronJob Last Schedule Time Ago	last_schedule_ago	seconds
k8s_state.cronjob_suspend_status	CronJob Suspend Status	enabled, suspended	status
k8s_state.cronjob_age	CronJob Age	age	seconds

Per pod

These metrics refer to the Pod.

Labels:

Label	Description
k8s_cluster_id	Cluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_name	Cluster name. Cluster name discovery only works in GKE.
k8s_node_name	Node name.
k8s_namespace	Namespace.
k8s_controller_kind	Controller kind (ReplicaSet, DaemonSet, StatefulSet, Job, etc.).
k8s_controller_name	Controller name.
k8s_pod_name	Pod name.
k8s_qos_class	Pod QOS class (burstable, guaranteed, besteffort).

Metrics:

Metric	Description	Dimensions	Unit
k8s_state.pod_cpu_requests_used	CPU requests used	requests	millicpu
k8s_state.pod_cpu_limits_used	CPU limits used	limits	millicpu
k8s_state.pod_mem_requests_used	Memory requests used	requests	bytes
k8s_state.pod_mem_limits_used	Memory limits used	limits	bytes
k8s_state.pod_condition	Condition	pod_ready, pod_scheduled, pod_initialized, containers_ready	state
k8s_state.pod_phase	Phase	running, failed, succeeded, pending	state
k8s_state.pod_status_reason	Status reason	Evicted, NodeAffinity, NodeLost, Shutdown, UnexpectedAdmissionError, Other	status
k8s_state.pod_age	Age	age	seconds
k8s_state.pod_containers	Containers	containers, init_containers	containers
k8s_state.pod_containers_state	Containers state	running, waiting, terminated	containers
k8s_state.pod_init_containers_state	Init containers state	running, waiting, terminated	containers

Per container

These metrics refer to the Pod container.

Labels:

Label	Description
k8s_cluster_id	Cluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_name	Cluster name. Cluster name discovery only works in GKE.
k8s_node_name	Node name.
k8s_namespace	Namespace.
k8s_controller_kind	Controller kind (ReplicaSet, DaemonSet, StatefulSet, Job, etc.).
k8s_controller_name	Controller name.
k8s_pod_name	Pod name.
k8s_qos_class	Pod QOS class (burstable, guaranteed, besteffort).
k8s_container_name	Container name.

Metrics:

Metric	Description	Dimensions	Unit
k8s_state.pod_container_readiness_state	Readiness state	ready	state
k8s_state.pod_container_restarts	Restarts	restarts	restarts
k8s_state.pod_container_state	Container state	running, waiting, terminated	state
k8s_state.pod_container_waiting_state_reason	Container waiting state reason	ContainerCreating, CrashLoopBackOff, CreateContainerConfigError, CreateContainerError, ErrImagePull, ImagePullBackOff, InvalidImageName, PodInitializing, Other	state
k8s_state.pod_container_terminated_state_reason	Container terminated state reason	Completed, ContainerCannotRun, DeadlineExceeded, Error, Evicted, OOMKilled, Other	state

Troubleshooting

Diagnostics

Debug Mode

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the k8s_state collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

• Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

  cd /usr/libexec/netdata/plugins.d/

• Switch to the netdata user.

  sudo -u netdata -s

• Run the go.d.plugin to debug the collector:

  ./go.d.plugin -d -m k8s_state

  To debug a specific job:

  ./go.d.plugin -d -m k8s_state -j jobName

Getting Logs

If you're encountering problems with the k8s_state collector, follow these steps to retrieve logs and identify potential issues:

• Run the command specific to your system (systemd, non-systemd, or Docker container).
• Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.

System with systemd

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep k8s_state

System without systemd

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep k8s_state /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep k8s_state