python.d.plugin is a Netdata external plugin. It is an orchestrator for data collection modules written in
- It runs as an independent process
ps faxshows it
- It is started and stopped automatically by Netdata
- It communicates with Netdata via a unidirectional pipe (sending data to the
- Supports any number of data collection modules
- Allows each module to have one or more data collection jobs
- Each job is collecting one or more metrics from a single data source
Every module should be compatible with python2 and python3.
All third party libraries should be installed system-wide or in
Module configurations are written in YAML and pyYAML is required.
Every configuration file must have one of two formats:
- Configuration for only one job:
- Configuration for many jobs (ex. mysql):
priority are always optional.
How to debug a python module
Depending on where Netdata was installed, execute one of the following commands to trace the execution of a python module:
[module] is the directory name under https://github.com/netdata/netdata/tree/master/collectors/python.d.plugin
How to write a new module
Writing new python module is simple. You just need to remember to include 5 major things:
- ORDER global list
- CHART global dictionary
- Service class
- _get_data method
- all code needs to be compatible with Python 2 (≥ 2.7) and 3 (≥ 3.1)
If you plan to submit the module in a PR, make sure and go through the PR checklist for new modules beforehand to make sure you have updated all the files you need to.
For a quick start, you can look at the example plugin.
ORDER list should contain the order of chart ids. Example:
CHART dictionary is a little bit trickier. It should contain the chart definition in following format:
All names are better explained in the External Plugins section.
update_every are handled by
Every module needs to implement its own
Service class. This class should inherit from one of the framework classes:
Also it needs to invoke the parent class constructor in a specific way as well as assign global variables to class variables.
This method should grab raw data from
_get_raw_data, parse it, and return a dictionary where keys are unique dimension names or
None if no data is collected.
More about framework classes
Every framework class has some user-configurable variables which are specific to this particular class. Those variables should have default values initialized in the child class constructor.
If module needs some additional user-configurable variable, it can be accessed from the
self.configuration list and assigned in constructor or custom
check method. Example:
_get_raw_data which should be used to grab raw data. This method usually returns a list of strings.
This is last resort class, if a new module cannot be written by using other framework class this one can be used.
It is the lowest-level class which implements most of module logic, like:
- handling run times
- chart formatting
- chart creation and updating
Variable from config file:
Object created from this class reads new lines from file specified in
log_path variable. It will check if file exists and is readable. Also
_get_raw_data returns list of strings where each string is one line from file specified in
Variable from config file:
This allows to execute a shell command in a secure way. It will check for invalid characters in
command variable and won't proceed if there is one of:
For additional security it uses python
shell=True option) to execute command. Command can be specified with absolute or relative name. When using relative name, it will try to find
PATH environment variable as well as in
_get_raw_data returns list of decoded lines returned by
Variables from config file:
If data is grabbed by accessing service via HTTP protocol, this class can be used. It can handle HTTP Basic Auth when specified with
Please note that the config file can use different variables according to the specification of each module.
_get_raw_data returns list of utf-8 decoded strings (lines).
Variables from config file:
Object will try execute
request using either
unix_socket or TCP/IP socket with combination of
port. This can access unix sockets with SOCK_STREAM or SOCK_DGRAM protocols and TCP/IP sockets in version 4 and 6 with SOCK_STREAM setting.
Sockets are accessed in non-blocking mode with 15 second timeout.
After every execution of
_get_raw_data socket is closed, to prevent this module needs to set
_keep_alive variable to
True and implement custom
_check_raw_data should take raw data and return
True if all data is received otherwise it should return
False. Also it should do it in fast and efficient way.
Pull Request Checklist for Python Plugins
This is a generic checklist for submitting a new Python plugin for Netdata. It is by no means comprehensive.
At minimum, to be buildable and testable, the PR needs to include:
- The module itself, following proper naming conventions:
- A README.md file for the plugin under
- The configuration file for the module:
collectors/python.d.plugin/<module_dir>/<module_name>.conf. Python config files are in YAML format, and should include comments describing what options are present. The instructions are also needed in the configuration section of the README.md
- A basic configuration for the plugin in the appropriate global config file:
collectors/python.d.plugin/python.d.conf, which is also in YAML format. Either add a line that reads
# <module_name>: yesif the module is to be enabled by default, or one that reads
<module_name>: noif it is to be disabled by default.
- A makefile for the plugin at
collectors/python.d.plugin/<module_dir>/Makefile.inc. Check an existing plugin for what this should look like.
- A line in
collectors/python.d.plugin/Makefile.amincluding the above-mentioned makefile. Place it with the other plugin includes (please keep the includes sorted alphabetically).
- Optionally, chart information in
web/gui/dashboard_info.js. This generally involves specifying a name and icon for the section, and may include descriptions for the section or individual charts.