How to write a Netdata collector in Go

Prerequisites

• Take a look at our contributing guidelines.
• Fork this repository to your personal GitHub account.
• Clone locally the forked repository (e.g git clone https://github.com/odyslam/go.d.plugin).
• Using a terminal, cd into the directory (e.g cd go.d.plugin)

Write and test a simple collector

❗ You can skip most of these steps if you first experiment directly with the existing example module, which will give you an idea of how things work.

Let's assume you want to write a collector named example2.

The steps are:

• Add the source code to modules/example2/.
  • module interface.
  • suggested module layout.
  • helper packages.
• Add the configuration to config/go.d/example2.conf.
• Add the module to config/go.d.conf.
• Import the module in modules/init.go.
• Update the available modules list.
• To build it, run make from the plugin root dir. This will create a new go.d.plugin binary that includes your newly developed collector. It will be placed into the bin directory (e.g go.d.plugin/bin)
• Run it in the debug mode bin/godplugin -d -m <MODULE_NAME>. This will output the STDOUT of the collector, the same output that is sent to the Netdata Agent and is transformed into charts. You can read more about this collector API in our documentation.
• If you want to test the collector with the actual Netdata Agent, you need to replace the go.d.plugin binary that exists in the Netdata Agent installation directory with the one you just compiled. Once you restart the Netdata Agent, it will detect and run it, creating all the charts. It is advised not to remove the default go.d.plugin binary, but simply rename it to go.d.plugin.old so that the Agent doesn't run it, but you can easily rename it back once you are done.
• Run make clean when you are done with testing.

Module Interface

Every module should implement the following interface:

type Module interface {
    Init() bool
    Check() bool
    Charts() *Charts
    Collect() map[string]int64
    Cleanup()
}

Init method

• Init does module initialization.
• If it returns false, the job will be disabled.

We propose to use the following template:

// example.go

func (e *Example) Init() bool {
    err := e.validateConfig()
    if err != nil {
        e.Errorf("config validation: %v", err)
        return false
    }

    someValue, err := e.initSomeValue()
    if err != nil {
        e.Errorf("someValue init: %v", err)
        return false
    }
    e.someValue = someValue

    // ...
    return true 
}

Move specific initialization methods into the init.go file. See suggested module layout.

Check method

• Check returns whether the job is able to collect metrics.
• Called after Init and only if Init returned true.
• If it returns false, the job will be disabled.

The simplest way to implement Check is to see if we are getting any metrics from Collect. A lot of modules use such approach.

// example.go

func (e *Example) Check() bool {
    return len(e.Collect()) > 0
}

Charts method

❗ Netdata module produces charts, not raw metrics.

Use agent/module package to create them, it contains charts and dimensions structs.

• Charts returns the charts (*module.Charts).
• Called after Check and only if Check returned true.
• If it returns nil, the job will be disabled
• ⚠️ Make sure not to share returned value between module instances (jobs).

Usually charts initialized in Init and Chart method just returns the charts instance:

// example.go

func (e *Example) Charts() *Charts {
    return e.charts
}

Collect method

• Collect collects metrics.
• Called only if Check returned true.
• Called every update_every seconds.
• map[string]int64 keys are charts dimensions ids'.

We propose to use the following template:

// example.go

func (e *Example) Collect() map[string]int64 {
    ms, err := e.collect()
    if err != nil {
        e.Error(err)
    }

    if len(ms) == 0 {
        return nil
    }
    return ms
}

Move metrics collection logic into the collect.go file. See suggested module layout.

Cleanup method

• Cleanup performs the job cleanup/teardown.
• Called if Init or Check fails, or we want to stop the job after Collect.

If you have nothing to clean up:

// example.go

func (Example) Cleanup() {}

Module Layout

The general idea is to not put everything in a single file.

We recommend using one file per logical area. This approach makes it easier to maintain the module.

Suggested minimal layout:

Filename	Contains
module_name.go	Module configuration, implementation and registration.
charts.go	Charts, charts templates and constructor functions.
init.go	Initialization methods.
collect.go	Metrics collection implementation.
module_name_test.go	Public methods/functions tests.
testdata/	Files containing sample data.

File module_name.go

❗ See the example example.go.

Don't overload this file with the implementation details.

Usually it contains only:

• module registration.
• module configuration.
• module interface implementation.

File charts.go

❗ See the example: charts.go.

Put charts, charts templates and charts constructor functions in this file.

File init.go

❗ See the example: init.go.

All the module initialization details should go in this file.

• make a function for each value that needs to be initialized.
• a function should return a value(s), not implicitly set/change any values in the main struct.

// init.go

// Prefer this approach.
func (e Example) initSomeValue() (someValue, error) {
    // ...
    return someValue, nil 
}

// This approach is ok too, but we recommend to not use it.
func (e *Example) initSomeValue() error {
    // ...
    m.someValue = someValue
    return nil
}

File collect.go

❗ See the example: collect.go.

This file is the entry point for the metrics collection.

Feel free to split it into several files if you think it makes the code more readable.

Use collect_ prefix for the filenames: collect_this.go, collect_that.go, etc.

// collect.go

func (e *Example) collect() (map[string]int64, error) {
    collected := make(map[string])int64
    // ...
    // ...
    // ...
    return collected, nil
}

File module_name_test.go

❗ See the example: example_test.go.

if you have no experience in testing we recommend starting with testing package documentation.

we use assert and require packages from github.com/stretchr/testify library, check their documentation.

Testing is mandatory.

• test only public functions and methods (New, Init, Check, Charts, Cleanup, Collect).
• do not create a test function per a case, use table driven tests . Prefer map[string]struct{ ... } over []struct{ ... }.
• use helper functions to prepare test cases to keep them clean and readable.

Directory testdata/

Put files with sample data in this directory if you need any. Its name should be testdata.

Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

Helper packages

There are some helper packages for writing a module.