Skip to main content
Version: 1.37.x

Apache Pulsar monitoring with Netdata

Apache Pulsar is an open-source distributed pub-sub messaging system.

This module will monitor one or more Apache Pulsar instances, depending on your configuration.

It collects broker statistics from the prometheus endpoint.

pulsar module is tested on the following versions:

  • v2.5.0

Metrics

All metrics have "pulsar." prefix.

  • topic_* metrics are available when exposeTopicLevelMetricsInPrometheus is set to true.
  • subscription_* and namespace_subscription metrics are available when exposeTopicLevelMetricsInPrometheus si set to true.
  • replication* and namespace_replication* metrics are available when replication is configured and replicationMetricsEnabled is set to true
MetricScopeDimensionsUnits
broker_componentsglobalnamespaces, topics, subscriptions, producers, consumerscomponents
messages_rateglobalpublish, dispatchmessages/s
throughput_rateglobalpublish, dispatchKiB/s
storage_sizeglobalusedKiB
storage_operations_rateglobalread, writemessage batches/s
msg_backlogglobalbacklogmessages
storage_write_latencyglobal<=0.5ms, <=1ms, <=5ms, =10ms, <=20ms, <=50ms, <=100ms, <=200ms, <=1s, >1sentries/s
entry_sizeglobal<=128B, <=512B, <=1KB, <=2KB, <=4KB, <=16KB, <=100KB, <=1MB, >1MBentries/s
subscription_delayedglobaldelayedmessage bacthes
subscription_msg_rate_redeliverglobalredeliveredmessages/s
subscription_blocked_on_unacked_messagesglobalblockedsubscriptions
replication_rateglobalin, outmessages/s
replication_throughput_rateglobalin, outKiB/s
replication_backlogglobalbacklogmessages
namespace_broker_componentsglobaltopics, subscriptions, producers, consumerscomponents
namespace_messages_ratenamespacepublish, dispatchmessages/s
namespace_throughput_ratenamespacepublish, dispatchKiB/s
namespace_storage_sizenamespaceusedKiB
namespace_storage_operations_ratenamespaceread, writemessage batches/s
namespace_msg_backlognamespacebacklogmessages
namespace_storage_write_latencynamespace<=0.5ms, <=1ms, <=5ms, =10ms, <=20ms, <=50ms, <=100ms, <=200ms, <=1s, >1sentries/s
namespace_entry_sizenamespace<=128B, <=512B, <=1KB, <=2KB, <=4KB, <=16KB, <=100KB, <=1MB, >1MBentries/s
namespace_subscription_delayednamespacedelayedmessage bacthes
namespace_subscription_msg_rate_redelivernamespaceredeliveredmessages/s
namespace_subscription_blocked_on_unacked_messagesnamespaceblockedsubscriptions
namespace_replication_ratenamespacein, outmessages/s
namespace_replication_throughput_ratenamespacein, outKiB/s
namespace_replication_backlognamespacebacklogmessages
topic_producersnamespacea dimension per topicproducers
topic_subscriptionsnamespacea dimension per topicsubscriptions
topic_consumersnamespacea dimension per topicconsumers
topic_messages_rate_innamespacea dimension per topicpublishes/s
topic_messages_rate_outnamespacea dimension per topicdispatches/s
topic_throughput_rate_innamespacea dimension per topicKiB/s
topic_throughput_rate_outnamespacea dimension per topicKiB/s
topic_storage_sizenamespacea dimension per topicKiB
topic_storage_read_ratenamespacea dimension per topicmessage batches/s
topic_storage_write_ratenamespacea dimension per topicmessage batches/s
topic_msg_backlognamespacea dimension per topicmessages
topic_subscription_delayednamespacea dimension per topicmessage batches
topic_subscription_msg_rate_redelivernamespacea dimension per topicmessages/s
topic_subscription_blocked_on_unacked_messagesnamespacea dimension per topicblocked subscriptions
topic_replication_rate_innamespacea dimension per topicmessages/s
topic_replication_rate_outnamespacea dimension per topicmessages/s
topic_replication_throughput_rate_innamespacea dimension per topicmessages/s
topic_replication_throughput_rate_outnamespacea dimension per topicmessages/s
topic_replication_backlognamespacea dimension per topicmessages
topic_replication_backlognamespacea dimension per topicmessages

Configuration

Edit the go.d/pulsar.conf configuration file using edit-config from the Netdata config directory, which is typically at /etc/netdata.

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/pulsar.conf

Needs only url to server's /metrics endpoint. Here is an example for 2 servers:

jobs:
- name: local
url: http://127.0.0.1:8080/metrics

- name: remote
url: http://203.0.113.10:8080/metrics

For all available options please see module configuration file.

Topic filtering

By default, module collects data for all topics, but it supports topic filtering. Filtering doesn't exclude a topic stats from the summary/namespace stats, it only removes the topic from the topic charts.

To check matcher syntax see matcher documentation.

  - name: local
url: http://127.0.0.1:8080/metrics
topic_filter:
includes:
- matcher1
- matcher2
excludes:
- matcher1
- matcher2

Update every

Module default update_every is 60.

Apache Pulsar doesnt expose raw counters, it exposes rate. It counts rates every statsUpdateFrequencyInSecs. Default value is 60 seconds.

Module update_every should be equal to statsUpdateFrequencyInSecs.

Troubleshooting

To troubleshoot issues with the pulsar collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

  • Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

    cd /usr/libexec/netdata/plugins.d/
  • Switch to the netdata user.

    sudo -u netdata -s
  • Run the go.d.plugin to debug the collector:

    ./go.d.plugin -d -m pulsar

Was this page helpful?

Contribute