The database engine uses RAM to store recent metrics. When metrics reach a certain age, and based on how much system RAM you allocate toward storing metrics in memory, they are compressed and "spilled" to disk for long-term storage.
The default settings retain about two day's worth of metrics on a system collecting 2,000 metrics every second, but the Netdata Agent is highly configurable if you want your nodes to store days, weeks, or months worth of per-second data.
The Netdata Agent uses two settings in
netdata.conf to change the behavior of the database engine:
page cache size sets the maximum amount of RAM (in MiB) the database engine uses to cache and index recent metrics.
dbengine multihost disk space sets the maximum disk space (again, in MiB) the database engine uses to store
historical, compressed metrics. When the size of stored metrics exceeds the allocated disk space, the database engine
removes the oldest metrics on a rolling basis.
You can store more or less metrics using the database engine by changing the allocated disk space. Use the calculator
below to find an appropriate value for
dbengine multihost disk space based on how many metrics your node(s) collect,
whether you are streaming metrics to a parent node, and more.
You do not need to edit the
page cache size setting to store more metrics using the database engine. However, if you
want to store more metrics specifically in memory, you can increase the cache size.
⚠️ This calculator provides an estimate of disk and RAM usage for metrics storage, along with its best recommendation for the
dbengine multihost disk spacesetting. Real-life usage may vary based on the accuracy of the values you enter below, changes in the compression ratio, and the types of metrics stored.
By default, the Agent collects metrics once per second (
1). See the
update every setting in your
netdata.conf file if yours may be different. If you have streaming nodes, use the average of their settings.
To find this value for an Agent, scroll to the bottom of the dashboard to find the number of metrics your Agent collects. If you have streaming nodes, input the average of these values.
To find this value, click on the Netdata Monitoring → dbengine items in the right-hand menu and look at the first chart under the dbengine section. You can also use your browser's search feature to look for compression savings ratio. Estimate an average value to input into the calculator.
netdata.conf file and find the
page cache size setting in the
With the above configuration, Netdata will use the following resources to store metrics:
MiBin total disk space
MiBin system memory
To enable this setup, edit your
netdata.conf file and change the
dbengine multihost disk space setting to the following:
Restart your Agent with
service netdata restart for the setting to take effect.
Your Agent now stores metrics for 1 day using a total of
MiB in disk space.
Notes on the database engine calculator
- The database engine requires a minimum disk space, which is reflected in this calculator. This required space is the maximum between your
dbengine multihost disk space/
dbengine disk spacesetting and
dimensions-being-collected * 4096 * 2.
- The system memory figure above is only for the database engine, and it may be higher in real-world situations due to memory fragmentation. The Agent will require additional memory for collection, visualization, and alerting features.
netdata.conf with recommended database engine settings#
Now that you have a recommended setting for
dbengine multihost disk space, open
edit-config and look for the
multihost disk space setting. Change it to the value recommended above. For example:
Save the file and restart the Agent with
service netdata restart to change the database engine's size.
For more information about the database engine, see our database reference doc.
Storing metrics with the database engine is completely interoperable with exporting to other time-series databases. With exporting, you can use the node's resources to surface metrics when viewing dashboards, while also archiving metrics elsewhere for further analysis, visualization, or correlation with other tools.
If you don't want to always store metrics on the node that collects them or run ephemeral nodes without dedicated storage, you can use streaming. Streaming allows you to centralize your data, run Agents as headless collectors, replicate data, and more.