systemd journal plugin

KEY FEATURES | PREREQUISITES | JOURNAL SOURCES | JOURNAL FIELDS | VISUALIZATION | PLAY MODE | FULL TEXT SEARCH | QUERY PERFORMANCE | PERFORMANCE AT SCALE | BEST PRACTICES | CONFIGURATION | FAQ | HOW TO TROUBLESHOOT COMMON ISSUES | HOW TO VERIFY SETUP

You view, explore, and analyze systemd journal logs from the Netdata dashboard: filter on journal fields, follow new entries as they arrive, and break down log frequency per field value over time.

Key features

• Unified view of logs from multiple sources (system, user, namespace, remote)
• Real-time streaming with PLAY mode for continuous monitoring
• Filtering on a curated set of journal fields, with counters showing matching entries
• Full-text search with wildcard and pattern matching
• Visual analysis with interactive histograms showing log frequency
• Enriched field display for improved readability (priorities, UIDs, timestamps)
• Sampling that keeps queries responsive on large datasets
• No configuration needed on supported systems
• Multi-node support for centralized log analysis
• UI-based exploration without needing to learn complex journalctl syntax
• Integrated with Netdata's dashboard for correlation with metrics

Prerequisites

Requirement	Details
Netdata Agent v1.44+	This plugin requires Netdata version 1.44 or newer
Netdata Cloud account	Required to use Netdata Functions, including this plugin
Not supported in:	Non-Linux builds (macOS, Windows, FreeBSD), the ARMv6l static build (Raspberry Pi 1 / Zero), or a source build compiled with --disable-plugin-systemd-journal

tip

This plugin is a Netdata Function Plugin. A free Netdata Cloud account is required. See the Netdata Functions discussion.

The plugin is designed for native package installations, source installations, and Docker installations (Debian-based). If using Docker, make sure you're using the Debian-based containers.

Offline Function test mode

Fixture harnesses can execute the Function query path directly against an existing journal directory:

systemd-journal.plugin --test systemd-journal --dir <journal-dir> [--timeout <seconds>] < payload.json

Requirements:

• <journal-dir> is scanned recursively for systemd journal files.
• stdin is the JSON Function request body (non-empty, maximum 16 MiB).
• --request is not supported and fails with usage output.
• --timeout <seconds> controls the offline Function execution timeout. It defaults to 60; use --timeout 0 to map to a very large finite timeout for long-running fixture comparisons.
• stdout contains only the raw JSON Function response.
• errors are written to stderr and return non-zero.

For example, an info request can use this payload:

{"info":true}

Function output includes volatile fields such as versions and timing-derived values. Test harnesses should normalize those fields before comparing fixture outputs.

Journal sources

The plugin automatically detects available journal sources based on the journal files in /var/log/journal (persistent logs) and /run/log/journal (volatile logs).

By default, all sources merge into a unified view of log messages.

tip

Select a specific source before analyzing logs in depth to improve query performance.

System journals

Default journals on all systemd-based systems. Includes:

• Kernel log messages (kmsg)
• Audit records
• Syslog messages via systemd-journald
• Output from service units
• Native journal API messages

User journals

• Shows journal files for all users, not just the current user
• Each regular user (UID > 999) typically has their own user journal
• Merged into remote journals on centralization servers

Namespace journals

• Isolate log streams per project or service
• Set with LogNamespace= in systemd unit files
• Requires special setup to propagate to central servers

Remote journals

• Created by systemd-journal-remote
• Typically named by sender IP, then resolved to hostname

Journal fields

systemd journals support dynamic fields per log entry. The journal's own on-disk index covers every field of every entry.

Fields are enriched for readability:

Field	Enrichment applied
_BOOT_ID	Timestamp of first message for boot
PRIORITY	Human-readable priority name
SYSLOG_FACILITY	Named facility
ERRNO	Error name
UID and GID fields	Resolved to user and group names
_CAP_EFFECTIVE	Human-readable capabilities
_SOURCE_REALTIME_TIMESTAMP	UTC timestamp
MESSAGE_ID	Well-known event name if known

tip

Enrichments are visual only and not searchable. UID/GID values are based on the system where the plugin runs.

Fields in the table

Use the ⚙️ icon above the table to select fields to display as columns.

The table view lets you analyze logs with:

• Scrollable list of log entries with customizable columns
• Color-coded PRIORITY levels for quick identification of issues
• Clickable entries for detailed viewing
• Time-ordered display (newest first by default)
• Pagination controls to navigate large datasets

Fields in the sidebar

Click a log entry to open the right-hand info panel showing all fields for that entry.

The sidebar shows:

• Every field present in the selected journal entry
• Raw and enriched field values
• Copyable text for sharing or further analysis
• Quick filtering options for any field value

Fields as filters

The plugin offers a curated set of journal fields as filters by default, each with counters, and you can enable any other journal field as a filter from the dashboard — the query then evaluates it the same way. Message text and timestamps are covered by full-text search and the time selection rather than filters. Full-text search covers the full text of every field of every entry.

tip

"Full data queries" mode enables negative/empty matches but may slow performance.

Key filter features:

• Real-time counters showing matching entry counts
• Multi-select capability for each field
• Toggleable inclusion/exclusion mode
• Persistent filter selections across page reloads

Fields as histogram sources

Histograms visualize log frequency per field value over time. Supports:

• Zoom
• Pan
• Click-to-navigate

Visualization capabilities

The plugin offers several visualization features to help you understand and navigate your logs effectively.

Timeline view

The timeline at the top of the interface shows:

• Log frequency distribution over time
• Interactive zoom and pan controls
• Time selection capabilities
• Anomaly highlighting

Histograms

Field-specific histograms provide:

• Visual breakdown of log entries by field value
• Color-coded frequency indicators
• Click-to-filter capability
• Time correlation with the main timeline

Color coding

The plugin uses color to enhance readability:

• Priority levels (emergency, alert, critical, etc.) have distinct colors
• Selected filters are highlighted
• Active elements use consistent color indicators
• Error states and warnings have clear visual differentiation

UI navigation

The interface offers several ways to navigate logs:

• Scroll through paginated results
• Jump to specific timeframes
• Click on histogram bars to focus on specific values
• Use filter panels to narrow down results
• Toggle between data views

PLAY mode

The plugin supports PLAY mode for real-time log streaming. Click the ▶️ button at the top of the dashboard to activate it.

• Continuously updates the screen with newly received logs
• Works for both single nodes and centralized log servers

tip

PLAY mode offers a similar experience to journalctl -f, but with visual enhancements.

Full text search

The plugin supports full-text search using flexible pattern matching:

Feature	Description
Contains match	Default pattern style (e.g., error matches error, error_count)
Wildcards	* matches any characters (e.g., a*b matches acb, a_long_b)
Multiple patterns	Separate with | for OR logic (e.g., error|warning matches lines containing either "error" OR "warning")
Negation	Prefix with ! to exclude (e.g., !systemd|* excludes lines with systemd)

tip

Full-text search applies across all fields. Combine with filters for precise results.

Query performance

The plugin reads journal files directly: through libsystemd on native packages (DEB, RPM), or through Netdata's own journal reader on static builds and Docker images, which do not use libsystemd for reading journals. Both read concurrently with the journal's single writer.

Two mechanisms shape how a query touches the files:

• Each journal file is opened with its own journal handle and queried on its own, one file at a time, instead of merging all files into a single interleaved handle.
• In the libsystemd build, fstat64() calls are cached inside the plugin process through an interposed function, so repeatedly checking the same journal files during a query does not repeat the system calls.

A Netdata query costs no more than the equivalent journalctl query on the same journal files.

Query performance depends on several factors:

Factor	Impact on Performance
Number of journal files queried	Fewer files lead to faster queries
Disk speed	Faster disks improve query times
Available memory	More memory allows better caching
Filters applied	Using fewer filters speeds up the query

For best performance:

• Keep the visible timeframe short
• Limit the number of rows displayed
• Apply filters to reduce the dataset
• Use specific sources instead of querying across all journals

Performance at scale

The plugin handles large datasets efficiently using a sampling algorithm, ensuring responsive queries even on busy log servers.

How sampling works

Step	Description
1	Fully evaluates the latest 500,000 log entries
2	Distributes evaluation across journal files for up to 1 million entries
3	Counts skipped entries in an [unsampled] histogram bucket beyond the evaluation budget
4	Stops scanning a file when sampling no longer pays off and adds the estimated remaining entries to an [estimated] histogram bucket
5	Uses sequence numbers (if available) for precise estimation
6	Continues responsive histogram generation while managing performance

Sampling prioritizes newer logs: the newest entries are fully evaluated first, and only after the budget is exhausted does the plugin stop evaluating every entry.

tip

Sampling never fabricates log entries. The rows shown in the table are always real journal entries.

What sampling changes in the results

Once a query goes beyond its evaluation budget, three things happen:

Result	Behavior beyond the budget
Rows in the table	Always real journal entries — nothing is synthesized
Histogram	Skipped entries are added to dedicated [unsampled] and [estimated] buckets, so the bars still account for them
Field filter counters	Stop counting: they reflect the evaluated entries only and are never extrapolated. Only the field selected for the histogram gains [unsampled] and [estimated] entries, mirroring its buckets

So a filter counter is a lower bound once you see [unsampled] or [estimated] in the histogram. Narrow the timeframe, select fewer sources, or apply more filters to bring the query back inside the budget and get exact counters.

Best practices for better performance

systemd-journal is designed for reliability first and performance second. It uses deduplication, field linking, and compression to minimize disk footprint, but the structure of journal files can still result in higher disk I/O during queries.

Filesystem and storage recommendations

Recommendation	Benefit
Use compressed filesystems (ext4, btrfs, zfs)	Reduces disk I/O by minimizing file size
Use SSD or NVMe storage	Speeds up journal file reads
Avoid small fragmented journal files	Prevents query slowdowns on busy centralization servers

Memory and caching

Recommendation	Benefit
Allocate more RAM for the system	Improves OS caching of journal files
Query the same timeframe repeatedly	Benefits from cached journal data
Limit query timeframes on large datasets	Reduces memory overhead and improves speed

tip

Journal data is cached by the operating system. The more RAM available for caching, the faster your queries will be.

Query strategies

Strategy	Why it helps
Narrow the timeframe of your queries	Minimizes data scanned per request
Use specific filters and source selections	Limits the scope of journal files being queried
Limit the number of rows returned in the UI	Keeps response times fast and manageable
Enable PLAY mode only when necessary	Reduces continuous query load on the system

Retention and disk usage

How long journals are kept, and how much disk they use, is controlled by systemd-journald itself, not by Netdata. See Log Storage and Retention for the journald.conf disk-usage and retention settings.

Configuration and maintenance

The systemd journal plugin needs no configuration on supported systems.

Requirements

Requirement	Description
Netdata Agent	Installed on the node or centralization server
Journal files	Located in /var/log/journal (persistent) or /run/log/journal (volatile)
Netdata Cloud account	Required to access Netdata Functions, including this plugin

tip

No additional configuration is required for this plugin to operate on supported systems.

Maintenance considerations

Task	Purpose
Keep Netdata up to date	Ensures plugin compatibility and performance optimizations
Monitor disk usage of journal files	Prevents performance issues caused by excessive log volume
Verify journal file locations	Confirms the plugin can access the intended sources
Review source selections periodically	Adjusts scope as infrastructure changes

Changing the journal directories at runtime

By default, the plugin scans /var/log/journal and /run/log/journal recursively. You can change this list at runtime — for example to include journal files mounted from other systems, backups, or container hosts — without restarting the Agent. The list you set replaces the defaults, so keep /var/log/journal and /run/log/journal in it if you still want them scanned.

The plugin exposes a Dynamic Configuration entry with the id systemd-journal:monitored-directories, at the path /logs/systemd-journal. Edit it from the Dynamic Configuration Manager on the Netdata Cloud dashboard: open the node's configuration, go to /logs/systemd-journal, and set the list of directories.

Access rules for this entry:

• Listing and reading it requires a signed-in Netdata Cloud user of the same Space.
• Editing it additionally requires the permission to edit Agent configuration, and a Space on a paid plan.

The underlying endpoint is /api/v3/config, which enforces the same rules — an unauthenticated curl against the Agent cannot read or change this setting.

Rules for the value:

• Up to 100 directories. Paths must be absolute; /, /dev, /proc, /sys, /etc, /lib, /lib32 and /lib64 are rejected, as are paths containing /./ or /../.
• Each update replaces the whole list — the configured directories replace the defaults — and the journal watcher restarts immediately, so no Agent restart is needed.
• Directories that do not exist are accepted, with a warning in the response. The directory watcher is inotify-based, so new journal files appearing in existing watched directories are detected automatically.

The configured directories are scanned recursively, the same way as the default locations, and the journals found in them are offered as sources.

FAQ

Can I use this plugin on journal centralization servers?

Yes. Where journals are already aggregated with systemd-journal-remote, install Netdata on the aggregation point and it manages the aggregated journals there, in their native format, combining the logs of all senders in one view.

tip

For details on configuring a journal centralization point, see Logs Centralization Points.

To centralize logs into Netdata's own log store instead, use OpenTelemetry — see Centralizing Logs with OpenTelemetry.

Can I use this plugin from a parent Netdata node?

Yes — if your nodes are connected to a Netdata parent, all their functions are accessible via the parent's UI.
This includes access to the systemd journal plugin for each child node.

Does this plugin expose any data to Netdata Cloud?

No — when accessing the Agent directly, no data is exposed to Netdata Cloud.
The Cloud account is only used for authentication. Data flows directly from your Netdata Agent to your web browser.

tip

When using https://app.netdata.cloud, communication is encrypted but data is not stored in Netdata Cloud.
See this discussion for more details.

What are volatile and persistent journals?

• Persistent journals are stored on disk in /var/log/journal
• Volatile journals are kept in memory in /run/log/journal and cleared on reboot

tip

For more, check man systemd-journald.

Which journal fields can I work with?

The plugin reads the journal files directly, so every field each entry carries is available, including dynamic, high-cardinality fields. No schema or field selection has to be declared in advance: full-text search covers the full text of every field, a curated set of fields is offered as filters by default, and any other field can be enabled as a filter from the dashboard.

Should I build a journal centralization point for Netdata?

Not for Netdata's sake. Netdata reads the journals of each node where they already are, and fully supports aggregated journals where a centralization point already exists.

To centralize logs into Netdata's own log store, use OpenTelemetry — see Centralizing Logs with OpenTelemetry.

How do I configure a journal centralization point?

Two main strategies:

Strategy	Description
Active sources	Central server fetches logs from each node
Passive sources	Nodes push logs to the central server

tip

See the passive journal centralization without encryption guide
or the encrypted setup guide.

What are the limitations when using centralization?

Limitation	Notes
Namespaces not supported by Docker	Related issue
systemd-journal-upload does not handle namespaces automatically	Requires manual configuration per namespace

How can I report bugs or request features?

If you encounter issues or have ideas for improvements:

1. Check the existing GitHub issues
2. Submit a new issue with detailed reproduction steps
3. For feature requests, describe your use case clearly

The plugin is actively maintained, and feedback helps improve it for everyone.

Can I customize the plugin's appearance or behavior?

Currently, customization options are limited to:

• Column selection in the table view
• Filter configurations
• Time range selection
• Source selection

Additional customization features may be added in future releases based on user feedback.

How to troubleshoot common issues

Use the following solutions to resolve common issues with the systemd journal plugin.

How to resolve plugin availability problems

Possible Cause	Solution
Running an older Netdata version (pre-1.44)	Update Netdata to version 1.44 or later
Using Alpine-based Docker container	Use the Debian-based Netdata container (from 1.44+)
Using the ARMv6l static build (Raspberry Pi 1 / Zero)	Switch to a package-based installation, or a source build with --enable-plugin-systemd-journal
Missing libsystemd or required dependencies	Make sure libsystemd is installed on the host

How to fix slow or timing out queries

Possible Cause	Solution
Querying too many journal files at once	Select specific sources before running your query
Long timeframes selected	Narrow the timeframe to improve performance
Low disk speed or insufficient RAM	Use faster disks and increase memory for better caching
Too many active filters	Reduce the number of filters applied

tip

Sampling ensures responsiveness at scale, but selecting sources and filters remains the best way to optimize performance.

How to solve missing journal sources

Possible Cause	Solution
Journals stored outside default paths	Create a symlink to /var/log/journal or /run/log/journal
Journals not persistent across reboots	Configure systemd-journald to enable persistent logs with Storage=persistent in /etc/systemd/journald.conf

How to address missing or incomplete logs

Possible Cause	Solution
Journals rotated or deleted	Ensure persistent storage is enabled
Misconfigured journal centralization	Check systemd-journal-remote and systemd-journal-upload settings
Namespace logs not forwarded	Manually configure forwarding for each namespace

How to fix UI rendering issues

Possible Cause	Solution
Outdated browser	Update to the latest version of Chrome, Firefox, Safari, or Edge
Zoom level affecting layout	Reset browser zoom to 100%
Ad blockers or script blockers	Temporarily disable to test if they're interfering
Network connectivity issues	Check network connections to Netdata server

How to understand error messages

Error Message	Meaning	Solution
"Plugin not available"	The systemd-journal plugin isn't loaded	See How to check if the plugin is running
"Unable to open journal"	Permission issues accessing journal files	Ensure Netdata has proper permissions for journal directories
"Timeout while querying"	Query is taking too long to complete	Reduce the query scope with filters or shorter timeframes
"No sources detected"	Cannot find valid journal files	Check journal file locations and setup
"Source selection failed"	Selected source cannot be accessed	Verify the source exists and permissions are correct

How to verify setup

How to check if the plugin is running

Confirm the plugin binary is present and executable:

# native/package install
ls -l /usr/libexec/netdata/plugins.d/systemd-journal.plugin

# static install
ls -l /opt/netdata/usr/libexec/netdata/plugins.d/systemd-journal.plugin

Confirm the Agent was built with the journal plugin. A line for systemd-journal (monitor journal logs) showing YES means the plugin is compiled into this build. NO means this build doesn't include it — for example, a non-Linux build (macOS, Windows, FreeBSD), a source build compiled with --disable-plugin-systemd-journal, or the ARMv6 static build (Raspberry Pi 1 / Zero):

sudo netdata -W buildinfo

Confirm the plugin process is running while the Agent is active:

ps aux | grep '[s]ystemd-journal.plugin'

If the plugin is present and running but you still see no persistent journal data, see How to confirm journal sources are detected below.

How to confirm journal sources are detected

1. Open the Logs tab in the Netdata UI
2. Use the Sources filter on the right sidebar
3. Ensure you can see your expected sources (e.g., system, user, remote, or specific namespaces)

tip

If sources are missing, check the journal file locations and verify symlinks if needed.

How to test basic queries

• Apply a single filter (like PRIORITY=3) and confirm logs are returned
• Use full-text search (e.g., search for error) and verify results populate correctly
• Toggle PLAY mode to confirm live logs are streaming

How to validate plugin logs

Check the Netdata Agent logs for plugin startup messages:

sudo journalctl -u netdata | grep journal

Look for lines confirming the journal plugin started successfully and detected sources.