systemd journal plugin
KEY FEATURES | PREREQUISITES | JOURNAL SOURCES | JOURNAL FIELDS | VISUALIZATION | PLAY MODE | FULL TEXT SEARCH | QUERY PERFORMANCE | PERFORMANCE AT SCALE | BEST PRACTICES | CONFIGURATION | FAQ | HOW TO TROUBLESHOOT COMMON ISSUES | HOW TO VERIFY SETUP
The systemd journal plugin provides an efficient way to view, explore, and analyze systemd journal logs directly from the Netdata dashboard. It combines powerful filtering, real-time updates, and visual analysis tools to help you troubleshoot system issues effectively.
Key features
- Unified view of logs from multiple sources (system, user, namespace, remote)
- Real-time streaming with PLAY mode for continuous monitoring
- Powerful filtering on all journal fields with counters showing matching entries
- Full-text search with wildcard and pattern matching
- Visual analysis with interactive histograms showing log frequency
- Enriched field display for improved readability (priorities, UIDs, timestamps)
- High performance with intelligent sampling for large datasets
- Zero configuration for immediate use on supported systems
- Multi-node support for centralized log analysis
- UI-based exploration without needing to learn complex journalctl syntax
- Integrated with Netdata's dashboard for correlation with metrics
Prerequisites
| Requirement | Details |
|---|---|
| Netdata Agent v1.44+ | This plugin requires Netdata version 1.44 or newer |
| Netdata Cloud account | Required to use Netdata Functions, including this plugin |
| Not supported in: | Static builds (use Debian-based containers instead) |
This plugin is a Netdata Function Plugin. A free Netdata Cloud account is required. See the Netdata Functions discussion.
The plugin is designed for native package installations, source installations, and Docker installations (Debian-based). If using Docker, make sure you're using the Debian-based containers.
Journal sources
The plugin automatically detects available journal sources based on the journal files in /var/log/journal (persistent logs) and /run/log/journal (volatile logs).
By default, all sources merge into a unified view of log messages.
Select a specific source before analyzing logs in depth to improve query performance.
System journals
Default journals on all systemd-based systems. Includes:
- Kernel log messages (
kmsg) - Audit records
- Syslog messages via
systemd-journald - Output from service units
- Native journal API messages
User journals
- Shows journal files for all users, not just the current user
- Each regular user (UID > 999) typically has their own user journal
- Merged into
remotejournals on centralization servers
Namespace journals
- Isolate log streams per project or service
- Set with
LogNamespace=in systemd unit files - Requires special setup to propagate to central servers
Remote journals
- Created by
systemd-journal-remote - Typically named by sender IP, then resolved to hostname
Journal fields
systemd journals support dynamic fields per log entry. All fields and values are indexed for fast querying.
Fields are enriched for readability:
| Field | Enrichment applied |
|---|---|
_BOOT_ID | Timestamp of first message for boot |
PRIORITY | Human-readable priority name |
SYSLOG_FACILITY | Named facility |
ERRNO | Error name |
| UID and GID fields | Resolved to user and group names |
_CAP_EFFECTIVE | Human-readable capabilities |
_SOURCE_REALTIME_TIMESTAMP | UTC timestamp |
MESSAGE_ID | Well-known event name if known |
Enrichments are visual only and not searchable. UID/GID values are based on the system where the plugin runs.
Fields in the table
Use the ⚙️ icon above the table to select fields to display as columns.
The table view provides a powerful way to analyze logs with:
- Scrollable list of log entries with customizable columns
- Color-coded PRIORITY levels for quick identification of issues
- Clickable entries for detailed viewing
- Time-ordered display (newest first by default)
- Pagination controls to navigate large datasets
Fields in the sidebar
Click a log entry to open the right-hand info panel showing all fields for that entry.
The sidebar shows:
- Every field present in the selected journal entry
- Raw and enriched field values
- Copyable text for sharing or further analysis
- Quick filtering options for any field value
Fields as filters
The plugin offers select fields as filters with counters. Field allowlists and blocklists protect performance.
"Full data queries" mode enables negative/empty matches but may slow performance.
Key filter features:
- Real-time counters showing matching entry counts
- Multi-select capability for each field
- Toggleable inclusion/exclusion mode
- Persistent filter selections across page reloads
Fields as histogram sources
Histograms visualize log frequency per field value over time. Supports:
- Zoom
- Pan
- Click-to-navigate
Visualization capabilities
The plugin offers several visualization features to help you understand and navigate your logs effectively.
Timeline view
The timeline at the top of the interface shows:
- Log frequency distribution over time
- Interactive zoom and pan controls
- Time selection capabilities
- Anomaly highlighting
Histograms
Field-specific histograms provide:
- Visual breakdown of log entries by field value
- Color-coded frequency indicators
- Click-to-filter capability
- Time correlation with the main timeline
Color coding
The plugin uses color to enhance readability:
- Priority levels (emergency, alert, critical, etc.) have distinct colors
- Selected filters are highlighted
- Active elements use consistent color indicators
- Error states and warnings have clear visual differentiation
UI navigation
The interface offers several ways to navigate logs:
- Scroll through paginated results
- Jump to specific timeframes
- Click on histogram bars to focus on specific values
- Use filter panels to narrow down results
- Toggle between data views
PLAY mode
The plugin supports PLAY mode for real-time log streaming. Click the ▶️ button at the top of the dashboard to activate it.
- Continuously updates the screen with newly received logs
- Works for both single nodes and centralized log servers
PLAY mode offers a similar experience to journalctl -f, but with visual enhancements.
Full text search
The plugin supports full-text search using flexible pattern matching:
| Feature | Description |
|---|---|
| Contains match | Default pattern style (e.g., error matches error, error_count) |
| Wildcards | * matches any characters (e.g., a*b matches acb, a_long_b) |
| Multiple patterns | Separate with | for OR logic (e.g., error|warning matches lines containing either "error" OR "warning") |
| Negation | Prefix with ! to exclude (e.g., !systemd|* excludes lines with systemd) |
Full-text search applies across all fields. Combine with filters for precise results.
Query performance
The plugin reads journal files directly using libsystemd, supporting concurrent readers and one writer.
Query performance depends on several factors:
| Factor | Impact on Performance |
|---|---|
| Number of journal files queried | Fewer files lead to faster queries |
| Disk speed | Faster disks improve query times |
| Available memory | More memory allows better caching |
| Filters applied | Using fewer filters speeds up the query |
For best performance:
- Keep the visible timeframe short
- Limit the number of rows displayed
- Apply filters to reduce the dataset
- Use specific sources instead of querying across all journals
Performance at scale
The plugin handles large datasets efficiently using a sampling algorithm, ensuring responsive queries even on busy log servers.
How sampling works
| Step | Description |
|---|---|
| 1 | Fully evaluates the latest 500,000 log entries |
| 2 | Distributes evaluation across journal files for up to 1 million entries |
| 3 | Marks additional entries as [unsampled] beyond the evaluation budget |
| 4 | Estimates counts as [estimated] once unsampled limits are hit |
| 5 | Uses sequence numbers (if available) for precise estimation |
| 6 | Continues responsive histogram generation while managing performance |
The plugin uses a sophisticated algorithm that prioritizes newer logs while maintaining reasonable estimates for historical data. This approach ensures that even with terabytes of journal data, the interface remains responsive and usable.
At scale, this plugin achieves up to 25–30x faster query performance compared to journalctl, especially on multi-journal queries.
The sampling algorithm is designed to be resilient to large datasets. Even if you see [unsampled] or [estimated] indicators, the results remain statistically representative of the full dataset.
Best practices for better performance
systemd-journal is designed for reliability first and performance second. It uses deduplication, field linking, and compression to minimize disk footprint, but the structure of journal files can still result in higher disk I/O during queries.
Filesystem and storage recommendations
| Recommendation | Benefit |
|---|---|
Use compressed filesystems (ext4, btrfs, zfs) | Reduces disk I/O by minimizing file size |
| Use SSD or NVMe storage | Speeds up journal file reads |
| Avoid small fragmented journal files | Prevents query slowdowns on busy centralization servers |
Memory and caching
| Recommendation | Benefit |
|---|---|
| Allocate more RAM for the system | Improves OS caching of journal files |
| Query the same timeframe repeatedly | Benefits from cached journal data |
| Limit query timeframes on large datasets | Reduces memory overhead and improves speed |
Journal data is cached by the operating system. The more RAM available for caching, the faster your queries will be.
Query strategies
| Strategy | Why it helps |
|---|---|
| Narrow the timeframe of your queries | Minimizes data scanned per request |
| Use specific filters and source selections | Limits the scope of journal files being queried |
| Limit the number of rows returned in the UI | Keeps response times fast and manageable |
| Enable PLAY mode only when necessary | Reduces continuous query load on the system |
Configuration and maintenance
The Netdata systemd journal plugin is designed to work out of the box with minimal configuration.
Requirements
| Requirement | Description |
|---|---|
| Netdata Agent | Installed on the node or centralization server |
| Journal files | Located in /var/log/journal (persistent) or /run/log/journal (volatile) |
| Netdata Cloud account | Required to access Netdata Functions, including this plugin |
No additional configuration is required for this plugin to operate on supported systems.
Maintenance considerations
| Task | Purpose |
|---|---|
| Keep Netdata up to date | Ensures plugin compatibility and performance optimizations |
| Monitor disk usage of journal files | Prevents performance issues caused by excessive log volume |
| Verify journal file locations | Confirms the plugin can access the intended sources |
| Review source selections periodically | Adjusts scope as infrastructure changes |
FAQ
Can I use this plugin on journal centralization servers?
Yes — you can centralize your logs using systemd-journal-remote and install Netdata on the centralization server to explore logs from your entire infrastructure.
The plugin provides multi-node views and allows you to combine logs from multiple servers.
For details on configuring a journal centralization server, see the journal centralization setup guide.
Can I use this plugin from a parent Netdata node?
Yes — if your nodes are connected to a Netdata parent, all their functions are accessible via the parent's UI.
This includes access to the systemd journal plugin for each child node.
Does this plugin expose any data to Netdata Cloud?
No — when accessing the Agent directly, no data is exposed to Netdata Cloud.
The Cloud account is only used for authentication. Data flows directly from your Netdata Agent to your web browser.
When using https://app.netdata.cloud, communication is encrypted but data is not stored in Netdata Cloud.
See this discussion for more details.
What are volatile and persistent journals?
- Persistent journals are stored on disk in
/var/log/journal - Volatile journals are kept in memory in
/run/log/journaland cleared on reboot
For more, check man systemd-journald.
I centralize my logs with Loki. Why use Netdata for journals?
systemd journals support dynamic, high-cardinality labels with all fields indexed by default.
When sending logs to Loki, you must predefine which fields to include, reducing flexibility.
Netdata reads journals directly, providing:
| Feature | Netdata journal plugin | Loki |
|---|---|---|
| Indexed on all fields | ✔️ Yes | ❌ Only selected labels |
| Supports dynamic fields | ✔️ Yes | ❌ Assumes fixed schema |
| Requires configuration | ❌ No | ✔️ Yes (relabel rules, label selection) |
Loki and systemd journals serve different use cases — they can complement, not replace, each other.
Is it worth setting up a systemd logs centralization server?
Yes — the tools required are included in modern Linux systems, and setup is straightforward.
Centralized logs provide high visibility with minimal overhead.
How do I configure a journal centralization server?
Two main strategies:
| Strategy | Description |
|---|---|
| Active sources | Central server fetches logs from each node |
| Passive sources | Nodes push logs to the central server |
Limitations when using centralization
| Limitation | Notes |
|---|---|
| Namespaces not supported by Docker | Related issue |
systemd-journal-upload does not handle namespaces automatically | Requires manual configuration per namespace |
How can I report bugs or request features?
If you encounter issues or have ideas for improvements:
- Check the existing GitHub issues
- Submit a new issue with detailed reproduction steps
- For feature requests, describe your use case clearly
The plugin is actively maintained, and feedback helps improve it for everyone.
Can I customize the plugin's appearance or behavior?
Currently, customization options are limited to:
- Column selection in the table view
- Filter configurations
- Time range selection
- Source selection
Additional customization features may be added in future releases based on user feedback.
How to troubleshoot common issues
Use the following solutions to resolve common issues with the systemd journal plugin.
How to resolve plugin availability problems
| Possible Cause | Solution |
|---|---|
| Running an older Netdata version (pre-1.44) | Update Netdata to version 1.44 or later |
| Using Alpine-based Docker container | Use the Debian-based Netdata container (from 1.44+) |
Using a static build of Netdata (/opt/netdata) | Switch to a package-based installation or source build |
Missing libsystemd or required dependencies | Make sure libsystemd is installed on the host |
How to fix slow or timing out queries
| Possible Cause | Solution |
|---|---|
| Querying too many journal files at once | Select specific sources before running your query |
| Long timeframes selected | Narrow the timeframe to improve performance |
| Low disk speed or insufficient RAM | Use faster disks and increase memory for better caching |
| Too many active filters | Reduce the number of filters applied |
Sampling ensures responsiveness at scale, but selecting sources and filters remains the best way to optimize performance.
How to solve missing journal sources
| Possible Cause | Solution |
|---|---|
| Journals stored outside default paths | Create a symlink to /var/log/journal or /run/log/journal |
| Journals not persistent across reboots | Configure systemd-journald to enable persistent logs with Storage=persistent in /etc/systemd/journald.conf |
How to address missing or incomplete logs
| Possible Cause | Solution |
|---|---|
| Journals rotated or deleted | Ensure persistent storage is enabled |
| Misconfigured journal centralization | Check systemd-journal-remote and systemd-journal-upload settings |
| Namespace logs not forwarded | Manually configure forwarding for each namespace |
How to fix UI rendering issues
| Possible Cause | Solution |
|---|---|
| Outdated browser | Update to the latest version of Chrome, Firefox, Safari, or Edge |
| Zoom level affecting layout | Reset browser zoom to 100% |
| Ad blockers or script blockers | Temporarily disable to test if they're interfering |
| Network connectivity issues | Check network connections to Netdata server |
How to understand error messages
| Error Message | Meaning | Solution |
|---|---|---|
| "Plugin not available" | The systemd-journal plugin isn't loaded | Check your Netdata installation type (must not be Alpine or static) |
| "Unable to open journal" | Permission issues accessing journal files | Ensure Netdata has proper permissions for journal directories |
| "Timeout while querying" | Query is taking too long to complete | Reduce the query scope with filters or shorter timeframes |
| "No sources detected" | Cannot find valid journal files | Check journal file locations and setup |
| "Source selection failed" | Selected source cannot be accessed | Verify the source exists and permissions are correct |
How to verify setup
How to check if the plugin is running
sudo netdata -W plugins
Check that the systemd-journal plugin is listed as active.
How to confirm journal sources are detected
- Open the Logs tab in the Netdata UI
- Use the Sources filter on the right sidebar
- Ensure you can see your expected sources (e.g.,
system,user,remote, or specific namespaces)
If sources are missing, check the journal file locations and verify symlinks if needed.
How to test basic queries
- Apply a simple filter (like
PRIORITY=3) and confirm logs are returned - Use full-text search (e.g., search for
error) and verify results populate correctly - Toggle PLAY mode to confirm live logs are streaming
How to validate plugin logs
Check the Netdata Agent logs for plugin startup messages:
sudo journalctl -u netdata | grep journal
Look for lines confirming the journal plugin started successfully and detected sources.
Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.