Skip to main content

Netdata Logging

This document describes how Netdata generates its own logs, not how Netdata manages and queries logs databases.

Log sources

Netdata supports the following log sources:

  1. daemon, logs generated by Netdata daemon.
  2. collector, logs generated by Netdata collectors, including internal and external ones.
  3. access, API requests received by Netdata
  4. health, all alert transitions and notifications

Log outputs

For each log source, Netdata supports the following output methods:

  • off, to disable this log source
  • journal, to send the logs to systemd-journal.
  • syslog, to send the logs to syslog.
  • system, to send the output to stderr or stdout depending on the log source.
  • stdout, to write the logs to Netdata's stdout.
  • stderr, to write the logs to Netdata's stderr.
  • filename, to send the logs to a file.

For daemon and collector the default is journal when systemd-journal is available. To decide if systemd-journal is available, Netdata checks:

  1. stderr is connected to systemd-journald
  2. /run/systemd/journal/socket exists
  3. /host/run/systemd/journal/socket exists (/host is configurable in containers)

If any of the above is detected, Netdata will select journal for daemon and collector sources.

All other sources default to a file.

Log formats

journaljournald-specific log format. Automatically selected when logging to systemd-journal.
logfmtlogs data as a series of key/value pairs. The default when logging to any output other than journal.
jsonlogs data in JSON format.

Log levels

Each time Netdata logs, it assigns a priority to the log. It can be one of this (in order of importance):

emergencya fatal condition, Netdata will most likely exit immediately after.
alerta very important issue that may affect how Netdata operates.
criticala very important issue the user should know which, Netdata thinks it can survive.
erroran error condition indicating that Netdata is trying to do something, but it fails.
warningsomething unexpected has happened that may or may not affect the operation of Netdata.
noticesomething that does not affect the operation of Netdata, but the user should notice.
infothe default log level about information the user should know.
debugthese are more verbose logs that can be ignored.

Logs Configuration

In netdata.conf, there are the following settings:

# logs to trigger flood protection = 1000
# logs flood protection period = 60
# facility = daemon
# level = info
# daemon = journal
# collector = journal
# access = /var/log/netdata/access.log
# health = /var/log/netdata/health.log
  • logs to trigger flood protection and logs flood protection period enable logs flood protection for daemon and collector sources. It can also be configured per log source.
  • facility is used only when Netdata logs to syslog.
  • level defines the minimum log level of logs that will be logged. This setting is applied only to daemon and collector sources. It can also be configured per source.

Configuring log sources

Each for the sources (daemon, collector, access, health), accepts the following:

source = {FORMAT},level={LEVEL},protection={LOG}/{PERIOD}@{OUTPUT}


  • {FORMAT}, is one of the log formats,
  • {LEVEL}, is the minimum log level to be logged,
  • {LOGS} is the number of logs to trigger flood protection configured per output,
  • {PERIOD} is the equivalent of logs flood protection period configured per output,
  • {OUTPUT} is one of the `log outputs,

All parameters can be omitted, except {OUTPUT}. If {OUTPUT} is the only given parameter, @ can be omitted.

Logs rotation

Netdata comes with logrotate configuration to rotate its log files periodically.

The default is usually found in /etc/logrotate.d/netdata.

Sending a SIGHUP to Netdata, will instruct it to re-open all its log files.

Log Fields

All fields exposed by Netdata
_SOURCE_REALTIME_TIMESTAMPtimetimethe timestamp of the event
SYSLOG_IDENTIFIERcommcommthe program logging the event
ND_LOG_SOURCEsourcesourceone of the log sources
one of the log levels
ERRNOerrnoerrnothe numeric value of errno
INVOCATION_ID--a unique UUID of the Netdata session, reset on every Netdata restart, inherited by systemd when available
CODE_LINE--the line number of of the source code logging this event
CODE_FILE--the filename of the source code logging this event
CODE_FUNCTION--the function name of the source code logging this event
TIDtidtidthe thread id of the thread logging this event
THREAD_TAGthreadthreadthe name of the thread logging this event
MESSAGE_IDmsg_idmsg_idsee message IDs
ND_MODULEmodulemodulethe Netdata module logging this event
ND_NIDL_NODEnodenodethe hostname of the node the event is related to
ND_NIDL_INSTANCEinstanceinstancethe instance of the node the event is related to
ND_NIDL_CONTEXTcontextcontextthe context the event is related to (this is usually the chart name, as shown on netdata dashboards
ND_NIDL_DIMENSIONdimensiondimensionthe dimension the event is related to
ND_SRC_TRANSPORTsrc_transportsrc_transportwhen the event happened during a request, this is the request transport
ND_SRC_IPsrc_ipsrc_ipwhen the event happened during an inbound request, this is the IP the request came from
ND_SRC_PORTsrc_portsrc_portwhen the event happened during an inbound request, this is the port the request came from
ND_SRC_FORWARDED_HOSTsrc_forwarded_hostsrc_forwarded_hostthe contents of the HTTP header X-Forwarded-Host
ND_SRC_FORWARDED_FORsrc_forwarded_forsrc_forwarded_forthe contents of the HTTP header X-Forwarded-For
ND_SRC_CAPABILITIESsrc_capabilitiessrc_capabilitieswhen the request came from a child, this is the communication capabilities of the child
ND_DST_TRANSPORTdst_transportdst_transportwhen the event happened during an outbound request, this is the outbound request transport
ND_DST_IPdst_ipdst_ipwhen the event happened during an outbound request, this is the IP the request destination
ND_DST_PORTdst_portdst_portwhen the event happened during an outbound request, this is the port the request destination
ND_DST_CAPABILITIESdst_capabilitiesdst_capabilitieswhen the request goes to a parent, this is the communication capabilities of the parent
ND_REQUEST_METHODreq_methodreq_methodwhen the event happened during an inbound request, this is the method the request was received
ND_RESPONSE_CODEcodecodewhen responding to a request, this this the response code
ND_CONNECTION_IDconnconnwhen there is a connection id for an inbound connection, this is the connection id
ND_TRANSACTION_IDtransactiontransactionthe transaction id (UUID) of all API requests
ND_RESPONSE_SENT_BYTESsent_bytessent_bytesthe bytes we sent to API responses
ND_RESPONSE_SIZE_BYTESsize_bytessize_bytesthe uncompressed bytes of the API responses
ND_RESPONSE_PREP_TIME_USECprep_utprep_utthe time needed to prepare a response
ND_RESPONSE_SENT_TIME_USECsent_utsent_utthe time needed to send a response
ND_RESPONSE_TOTAL_TIME_USECtotal_uttotal_utthe total time needed to complete a response
ND_ALERT_IDalert_idalert_idthe alert id this event is related to
ND_ALERT_EVENT_IDalert_event_idalert_event_ida sequential number of the alert transition (per host)
ND_ALERT_UNIQUE_IDalert_unique_idalert_unique_ida sequential number of the alert transition (per alert)
ND_ALERT_TRANSITION_IDalert_transition_idalert_transition_idthe unique UUID of this alert transition
ND_ALERT_CONFIGalert_configalert_configthe alert configuration hash (UUID)
ND_ALERT_NAMEalertalertthe alert name
ND_ALERT_CLASSalert_classalert_classthe alert classification
ND_ALERT_COMPONENTalert_componentalert_componentthe alert component
ND_ALERT_TYPEalert_typealert_typethe alert type
ND_ALERT_EXECalert_execalert_execthe alert notification program
ND_ALERT_RECIPIENTalert_recipientalert_recipientthe alert recipient(s)
ND_ALERT_VALUEalert_valuealert_valuethe current alert value
ND_ALERT_VALUE_OLDalert_value_oldalert_value_oldthe previous alert value
ND_ALERT_STATUSalert_statusalert_statusthe current alert status
ND_ALERT_STATUS_OLDalert_value_oldalert_value_oldthe previous alert value
ND_ALERT_UNITSalert_unitsalert_unitsthe units of the alert
ND_ALERT_SUMMARYalert_summaryalert_summarythe summary text of the alert
ND_ALERT_INFOalert_infoalert_infothe info text of the alert
ND_ALERT_DURATIONalert_durationalert_durationthe duration the alert was in its previous state
ND_ALERT_NOTIFICATION_TIMESTAMP_USECalert_notification_timestampalert_notification_timestampthe timestamp the notification delivery is scheduled
ND_REQUESTrequestrequestthe full request during which the event happened
MESSAGEmsgmsgthe event message

Message IDs

Netdata assigns specific message IDs to certain events:

  • ed4cdb8f1beb4ad3b57cb3cae2d162fa when a Netdata child connects to this Netdata
  • 6e2e3839067648968b646045dbf28d66 when this Netdata connects to a Netdata parent
  • 9ce0cb58ab8b44df82c4bf1ad9ee22de when alerts change state
  • 6db0018e83e34320ae2a659d78019fb7 when notifications are sent

You can view these events using the Netdata systemd-journal.plugin at the MESSAGE_ID filter, or using journalctl like this:

# query children connection
journalctl MESSAGE_ID=ed4cdb8f1beb4ad3b57cb3cae2d162fa

# query parent connection
journalctl MESSAGE_ID=6e2e3839067648968b646045dbf28d66

# query alert transitions
journalctl MESSAGE_ID=9ce0cb58ab8b44df82c4bf1ad9ee22de

# query alert notifications
journalctl MESSAGE_ID=6db0018e83e34320ae2a659d78019fb7

Using journalctl to query Netdata logs

The Netdata service's processes execute within the netdata journal namespace. To view the Netdata logs, you should specify the --namespace=netdata option.

# Netdata logs since the last time the service was started
journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata

# All netdata logs, the oldest entries are displayed first
journalctl -u netdata --namespace=netdata

# All netdata logs, the newest entries are displayed first
journalctl -u netdata --namespace=netdata -r

Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.