Skip to main content

HDFS collector

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware.

This module monitors one or more Hadoop Distributed File System nodes, depending on your configuration.

Netdata accesses HDFS metrics over Java Management Extensions (JMX) through the web interface of an HDFS daemon.

Requirements

  • hdfs node with accessible /jmx endpoint

Metrics

All metrics have "hdfs." prefix.

MetricScopeDimensionsUnits
heap_memoryglobalcommitted, usedMiB
gc_count_totalglobalgcevents/s
gc_time_totalglobalmsms
gc_thresholdglobalinfo, warnevents/s
threadsglobalnew, runnable, blocked, waiting, timed_waiting, terminatednum
logs_totalglobalinfo, error, warn, fatallogs/s
rpc_bandwidthglobalreceived, sentkilobits/s
rpc_callsglobalcallscalls/s
open_connectionsglobalopenconnections
call_queue_lengthgloballengthnum
avg_queue_timeglobaltimems
avg_processing_timeglobaltimems
capacityglobalremaining, usedKiB
used_capacityglobaldfs, non_dfsKiB
loadgloballoadload
volume_failures_totalglobalfailuresevents/s
files_totalglobalfilesnum
blocks_totalglobalblocksnum
blocksglobalcorrupt, missing, under_replicatednum
data_nodesgloballive, dead, stalenum
datanode_capacityglobalremaining, usedKiB
datanode_used_capacityglobaldfs, non_dfsKiB
datanode_failed_volumesglobalfailed volumesnum
datanode_bandwidthglobalreads, writesKiB/s

Configuration

Edit the go.d/hdfs.conf configuration file using edit-config from the Netdata config directory, which is typically at /etc/netdata.

cd /etc/netdata # Replace this path with your Netdata config directory
sudo ./edit-config go.d/hdfs.conf

Needs only url to server's /jmx endpoint. Here is an example for 2 servers:

jobs:
- name: namenode
url: http://127.0.0.1:9870/jmx

- name: datanode
url: http://127.0.0.1:9864/jmx

For all available options, please see the module configuration file.

Troubleshooting

To troubleshoot issues with the hdfs collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

  • Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

    cd /usr/libexec/netdata/plugins.d/
  • Switch to the netdata user.

    sudo -u netdata -s
  • Run the go.d.plugin to debug the collector:

    ./go.d.plugin -d -m hdfs

Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.