Hadoop is an Apache project is a framework for processing large sets of data across a distributed cluster of systems.
And while Hadoop is designed to be a highly-available and fault-tolerant service, those who operate a Hadoop cluster will want to monitor the health and performance of their Hadoop Distributed File System (HDFS) and Zookeeper implementations.
Netdata comes with built-in and pre-configured support for monitoring both HDFS and Zookeeper.
This guide assumes you have a Hadoop cluster, with HDFS and Zookeeper, running already. If you don't, please follow the official Hadoop instructions or an alternative, like the guide available from DigitalOcean.
For more specifics on the collection modules used in this guide, read the respective pages in our documentation:
As with all data sources, Netdata can auto-detect HDFS and Zookeeper nodes if you installed them using the standard installation procedure.
For Netdata to collect HDFS metrics, it needs to be able to access the node's
/jmx endpoint. You can test whether an
JMX endpoint is accessible by using
curl HDFS-IP:PORT/jmx. For a NameNode, you should see output similar to the
The JSON result for a DataNode's
/jmx endpoint is slightly different:
If Netdata can't access the
/jmx endpoint for either a NameNode or DataNode, it will not be able to auto-detect and
collect metrics from your HDFS implementation.
To configure Netdata's HDFS module, navigate to your Netdata directory (typically at
/etc/netdata/) and use
edit-config to initialize and edit your HDFS configuration file.
At the bottom of the file, you will see two example jobs, both of which are commented out:
Uncomment these lines and edit the
url value(s) according to your setup. Now's the time to add any other configuration
details, which you can find inside of the
hdfs.conf file itself. Most production implementations will require TLS
The result for a simple HDFS setup, running entirely on
localhost and without certificate authentication, might look
At this point, Netdata should be configured to collect metrics from your HDFS servers. Let's move on to Zookeeper.
edit-config again to initialize/edit your
As with the
hdfs.conf file, head to the bottom, uncomment the example jobs, and tweak the
address values according
to your setup. Again, you may need to add additional configuration options, like TLS certificates.
Finally, restart Netdata.
Upon restart, Netdata should recognize your HDFS/Zookeeper servers, enable the HDFS and Zookeeper modules, and begin showing real-time metrics for both in your Netdata dashboard. 🎉
The Netdata community helped us create sane defaults for alarms related to both HDFS and Zookeeper. You may want to investigate these to ensure they work well with your Hadoop implementation.
You can also access/edit these files directly with
For more information about editing the defaults or writing new alarm entities, see our health monitoring documentation.
If you're having issues with Netdata auto-detecting your HDFS/Zookeeper servers, or want to help improve how Netdata collects or presents metrics from these services, feel free to file an issue.
- Read up on the HDFS configuration file to understand how to configure global options or per-job options, such as username/password, TLS certificates, timeouts, and more.
- Read up on the Zookeeper configuration file to understand how to configure global options or per-job options, timeouts, TLS certificates, and more.