Skip to main content

Consul

Plugin: go.d.plugin Module: consul

Overview

This collector monitors key metrics of Consul Agents: transaction timings, leadership changes, memory usage and more.

It periodically sends HTTP requests to Consul REST API.

Used endpoints:

This collector is supported on all platforms.

This collector supports collecting metrics from multiple instances of this integration, including remote instances.

Default Behavior

Auto-Detection

This collector discovers instances running on the local host, that provide metrics on port 8500.

On startup, it tries to collect metrics from:

Limits

The default configuration for this integration does not impose any limits on data collection.

Performance Impact

The default configuration for this integration is not expected to impose a significant performance impact on the system.

Metrics

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

The set of metrics depends on the Consul Agent mode.

Per Consul instance

These metrics refer to the entire monitored application.

This scope has no labels.

Metrics:

MetricDimensionsUnitLeaderFollowerClient
consul.client_rpc_requests_raterpcrequests/s
consul.client_rpc_requests_exceeded_rateexceededrequests/s
consul.client_rpc_requests_failed_ratefailedrequests/s
consul.memory_allocatedallocatedbytes
consul.memory_syssysbytes
consul.gc_pause_timegc_pauseseconds
consul.kvs_apply_timequantile_0.5, quantile_0.9, quantile_0.99ms
consul.kvs_apply_operations_ratekvs_applyops/s
consul.txn_apply_timequantile_0.5, quantile_0.9, quantile_0.99ms
consul.txn_apply_operations_ratetxn_applyops/s
consul.autopilot_health_statushealthy, unhealthystatus
consul.autopilot_failure_tolerancefailure_toleranceservers
consul.autopilot_server_health_statushealthy, unhealthystatus
consul.autopilot_server_stable_timestableseconds
consul.autopilot_server_serf_statusactive, failed, left, nonestatus
consul.autopilot_server_voter_statusvoter, not_voterstatus
consul.network_lan_rttmin, max, avgms
consul.raft_commit_timequantile_0.5, quantile_0.9, quantile_0.99ms
consul.raft_commits_ratecommitscommits/s
consul.raft_leader_last_contact_timequantile_0.5, quantile_0.9, quantile_0.99ms
consul.raft_leader_oldest_log_ageoldest_log_ageseconds
consul.raft_follower_last_contact_leader_timeleader_last_contactms
consul.raft_rpc_install_snapshot_timequantile_0.5, quantile_0.9, quantile_0.99ms
consul.raft_leader_elections_rateleaderelections/s
consul.raft_leadership_transitions_rateleadershiptransitions/s
consul.server_leadership_statusleader, not_leaderstatus
consul.raft_thread_main_saturation_percquantile_0.5, quantile_0.9, quantile_0.99percentage
consul.raft_thread_fsm_saturation_percquantile_0.5, quantile_0.9, quantile_0.99percentage
consul.raft_fsm_last_restore_durationlast_restore_durationms
consul.raft_boltdb_freelist_bytesfreelistbytes
consul.raft_boltdb_logs_per_batch_ratewrittenlogs/s
consul.raft_boltdb_store_logs_timequantile_0.5, quantile_0.9, quantile_0.99ms
consul.license_expiration_timelicense_expirationseconds

Per node check

Metrics about checks on Node level.

Labels:

LabelDescription
datacenterDatacenter Identifier
node_nameThe node's name
check_nameThe check's name

Metrics:

MetricDimensionsUnitLeaderFollowerClient
consul.node_health_check_statuspassing, maintenance, warning, criticalstatus

Per service check

Metrics about checks at a Service level.

Labels:

LabelDescription
datacenterDatacenter Identifier
node_nameThe node's name
check_nameThe check's name
service_nameThe service's name

Metrics:

MetricDimensionsUnitLeaderFollowerClient
consul.service_health_check_statuspassing, maintenance, warning, criticalstatus

Alerts

The following alerts are available:

Alert nameOn metricDescription
consul_node_health_check_status consul.node_health_check_statusnode health check ${label:check_name} has failed on server ${label:node_name} datacenter ${label:datacenter}
consul_service_health_check_status consul.service_health_check_statusservice health check ${label:check_name} for service ${label:service_name} has failed on server ${label:node_name} datacenter ${label:datacenter}
consul_client_rpc_requests_exceeded consul.client_rpc_requests_exceeded_ratenumber of rate-limited RPC requests made by server ${label:node_name} datacenter ${label:datacenter}
consul_client_rpc_requests_failed consul.client_rpc_requests_failed_ratenumber of failed RPC requests made by server ${label:node_name} datacenter ${label:datacenter}
consul_gc_pause_time consul.gc_pause_timetime spent in stop-the-world garbage collection pauses on server ${label:node_name} datacenter ${label:datacenter}
consul_autopilot_health_status consul.autopilot_health_statusdatacenter ${label:datacenter} cluster is unhealthy as reported by server ${label:node_name}
consul_autopilot_server_health_status consul.autopilot_server_health_statusserver ${label:node_name} from datacenter ${label:datacenter} is unhealthy
consul_raft_leader_last_contact_time consul.raft_leader_last_contact_timemedian time elapsed since leader server ${label:node_name} datacenter ${label:datacenter} was last able to contact the follower nodes
consul_raft_leadership_transitions consul.raft_leadership_transitions_ratethere has been a leadership change and server ${label:node_name} datacenter ${label:datacenter} has become the leader
consul_raft_thread_main_saturation consul.raft_thread_main_saturation_percaverage saturation of the main Raft goroutine on server ${label:node_name} datacenter ${label:datacenter}
consul_raft_thread_fsm_saturation consul.raft_thread_fsm_saturation_percaverage saturation of the FSM Raft goroutine on server ${label:node_name} datacenter ${label:datacenter}
consul_license_expiration_time consul.license_expiration_timeConsul Enterprise licence expiration time on node ${label:node_name} datacenter ${label:datacenter}

Setup

Prerequisites

Enable Prometheus telemetry

Enable telemetry on your Consul agent, by increasing the value of prometheus_retention_time from 0.

Add required ACLs to Token

Required only if authentication is enabled.

ACLEndpoint
operator:readautopilot health status
node:readchecks
agent:readconfiguration, metrics, and lan coordinates

Configuration

File

The configuration file name for this integration is go.d/consul.conf.

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/consul.conf

Options

The following options can be defined globally: update_every, autodetection_retry.

All options
NameDescriptionDefaultRequired
update_everyData collection frequency.1no
autodetection_retryRecheck interval in seconds. Zero means no recheck will be scheduled.0no
urlServer URL.http://localhost:8500yes
acl_tokenACL token used in every request.no
max_checksChecks processing/charting limit.no
max_filterChecks processing/charting filter. Uses simple patterns.no
usernameUsername for basic HTTP authentication.no
passwordPassword for basic HTTP authentication.no
proxy_urlProxy URL.no
proxy_usernameUsername for proxy basic HTTP authentication.no
proxy_passwordPassword for proxy basic HTTP authentication.no
timeoutHTTP request timeout.1no
methodHTTP request method.GETno
bodyHTTP request body.no
headersHTTP request headers.no
not_follow_redirectsRedirect handling policy. Controls whether the client follows redirects.nono
tls_skip_verifyServer certificate chain and hostname validation policy. Controls whether the client performs this check.nono
tls_caCertification authority that the client uses when verifying the server's certificates.no
tls_certClient tls certificate.no
tls_keyClient tls key.no

Examples

Basic

An example configuration.

jobs:
- name: local
url: http://127.0.0.1:8500
acl_token: "ec15675e-2999-d789-832e-8c4794daa8d7"

Basic HTTP auth

Local server with basic HTTP authentication.

Config
jobs:
- name: local
url: http://127.0.0.1:8500
acl_token: "ec15675e-2999-d789-832e-8c4794daa8d7"
username: foo
password: bar

Multi-instance

Note: When you define multiple jobs, their names must be unique.

Collecting metrics from local and remote instances.

Config
jobs:
- name: local
url: http://127.0.0.1:8500
acl_token: "ec15675e-2999-d789-832e-8c4794daa8d7"

- name: remote
url: http://203.0.113.10:8500
acl_token: "ada7f751-f654-8872-7f93-498e799158b6"

Troubleshooting

Debug Mode

To troubleshoot issues with the consul collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

  • Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

    cd /usr/libexec/netdata/plugins.d/
  • Switch to the netdata user.

    sudo -u netdata -s
  • Run the go.d.plugin to debug the collector:

    ./go.d.plugin -d -m consul

Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.