diff --git a/DC-SLES-journalctl b/DC-SLES-journalctl new file mode 100644 index 000000000..0aaa4826a --- /dev/null +++ b/DC-SLES-journalctl @@ -0,0 +1,18 @@ +# This file originates from the project https://github.com/openSUSE/doc-kit +# This file can be edited downstream. + +## Basics +MAIN="journalctl.asm.xml" +SRC_DIR="articles" +IMG_SRC_DIR="images" + +## Profiling +PROFOS="sles" +PROFCONDITION="16.0" +#PROFARCH="x86_64;zseries;power;aarch64" + +DOCBOOK5_RNG_URI="urn:x-suse:rng:v2:geekodoc-flat" + +## stylesheet location +STYLEROOT="/usr/share/xml/docbook/stylesheet/suse2022-ns" +FALLBACK_STYLEROOT="/usr/share/xml/docbook/stylesheet/suse2021-ns" \ No newline at end of file diff --git a/articles/journalctl.asm.xml b/articles/journalctl.asm.xml new file mode 100644 index 000000000..0e4165cb8 --- /dev/null +++ b/articles/journalctl.asm.xml @@ -0,0 +1,157 @@ + + + +%entities; +]> + + + + + + + + + + + + + + + + + + + + +Legal Notice + + +GNU Free Documentation License + + + + +Mastering System Logs: From <command>systemd-journald</command> to <command>journalctl</command> + + +2025-11-26 + + +Initial version + + + + + + + + +&x86-64; +&power; +&zseries; +&aarch64; + + + +&sles; +&sles4sap; +&sleha; + + +Mastering System Logs: From systemd-journald to journalctl + +Learn how to view and manage &productnameshort; logs using journalctl + +View and filter logs with journalctl + + +Configuration +Storage +Maintenance +Administration + + +Systems Management + + +Products & Solutions + + + +https://bugzilla.suse.com/enter_bug.cgi +Documentation +SUSE Linux Enterprise Server 16.0 +SUSE Linux Enterprise Server 16.0 +SUSE Linux Enterprise Server 16.0 +shalaka.harne@suse.com + +yes + + + + +WHAT? + + +View and analyze logs from systemd-journald using the journalctl command-line tool. + + + + +WHY? + + +This article provides a complete overview of tasks that can be performed using the journalctl command-line tool. + + + + +EFFORT + + +The average reading time of this article is approximately 30 minutes. + + + + +GOAL + + +You will be able to view your system logs using journalctl. + + + + +REQUIREMENTS + + +You must have &sudo; privileges. + + + + + + + + + + + + + + + + + + + + + + diff --git a/concepts/journactl-about.xml b/concepts/journactl-about.xml new file mode 100644 index 000000000..84ed3672a --- /dev/null +++ b/concepts/journactl-about.xml @@ -0,0 +1,38 @@ + + + + + %entities; +]> + + + + + + + The <command>journalctl</command> command + + + + +journalctl is a command-line utility used to query and display logs collected by +the systemd-journald. + + + + + Because the logging system stores data in a binary format (for performance and security) rather + than plain text, you cannot use standard tools like cat or + less to open the files directly. So, the journalctl + command decodes the data and displays it according to the provided parameters. + + + diff --git a/concepts/journal-about-journald.xml b/concepts/journal-about-journald.xml new file mode 100644 index 000000000..c014e17c2 --- /dev/null +++ b/concepts/journal-about-journald.xml @@ -0,0 +1,34 @@ + + + %entities; +]> + + + + + + + The <command>systemd-journald</command> service + + + + journald is a &systemd; service responsible for collecting, indexing, and storing log data. The collected events include kernel messages, initrd (initial RAM disk) messages, + service startup/shutdown, application events, and authentication and session data. + + + + + On &productname;, the systemd-journald service is enabled and started by + default. The service logs the data into the journal described in . + + + + diff --git a/concepts/journal-structure.xml b/concepts/journal-structure.xml new file mode 100644 index 000000000..536f7d58d --- /dev/null +++ b/concepts/journal-structure.xml @@ -0,0 +1,151 @@ + + + + + %entities; +]> + + + + + + + About the journal + + + + + The journal is a centralized, unified storage system for log data from all sources. It stores events in a structured manner, which helps with error identification and crash recovery. + + + +
+ Where can I find the journal? + + By default, journald stores the collected data in the volatile memory in + /run/log/journal, so after reboot the data is lost. + + + If systemd-journald is configured to store the logs in a persistent + manner, you can find the journal in /var/log/journal. See if the + directory is there: + + + + + Check if the directory exist: + + &prompt.sudo;ls /var/log +... +journal +... + + + + + + If the /var/log/journal directory does not exist, create + it: + &prompt.sudo;mkdir /var/log/journal + + + + + Set the correct ownership and access permissions: + + &prompt.sudo; systemd-tmpfiles --create --prefix=/var/log/journal + + + + To flush the data from RAM to the directory: + + &prompt.sudo; journalctl --flush + + + For details about + the service configuration, refer to . + +
+
+ The journal log structure + + The logs are stored in binary format across several files. To view the structure of a log + entry, proceed as follows: + + &prompt.sudo; journalctl -n 1 -o verbose + + Fri 2025-12-05 10:52:53.284321 CET [s=5b7ae2e926794210bf832d014f5f560a;i=d49752> + _UID=1000 + _AUDIT_SESSION=3 + _AUDIT_LOGINUID=1000 + _SYSTEMD_OWNER_UID=1000 + _SYSTEMD_UNIT=user@1000.service + _SYSTEMD_SLICE=user-1000.slice + _MACHINE_ID=d3489468c8534c5d81a2860cf9a2a20e + _RUNTIME_SCOPE=system + _SELINUX_CONTEXT=unconfined + PRIORITY=6 + _TRANSPORT=syslog + _BOOT_ID=d095069bd59b4bc4bf67c8c71999243b + SYSLOG_IDENTIFIER=sudo + SYSLOG_TIMESTAMP=Dec 5 10:52:53 + _PID=25806 + _COMM=sudo + _EXE=/usr/bin/sudo + _CMDLINE=sudo journalctl -n 1 -o verbose + + ... + + The log entry is a data structure containing the log message and metadata fields: + + + Timestamp + + +The exact time the event occurred + + + + + Source fields + + + + _PID: process ID of the sender + _UID/_GID: User/Group ID of the sender + _EXE: path to the executable + _COMM: name of the executable + + + + + System fields + + + _SYSTEMD_UNIT: The systemd unit (service) that generated the + log (for example, sshd.service) + _BOOT_ID: A unique identifier for the specific system boot session + + + + + Kernel fields + _TRANSPORT: How the message was logged (e.g., kernel, syslog, stdout) + + + Priority level + + A numeric value indicating the severity of the message (0=emerg, 7=debug) + + + +
+
diff --git a/references/journalctl-usage.xml b/references/journalctl-usage.xml new file mode 100644 index 000000000..13638a9f3 --- /dev/null +++ b/references/journalctl-usage.xml @@ -0,0 +1,102 @@ + + + %entities; +]> + + + + + + + + + Using <command>journalctl</command> + + +This section describes the generic usage of the journalctl command. + + + + + + + Running journalctl without any options displays all logged messages, usually starting from the oldest, and pipes the output through a pager (like less) for easy navigation. + + + The general syntax of the journalctl command is as follows: + + &prompt.sudo;journalctl [OPTIONS...] [MATCHES...] + + Messages related to a specific executable + + To show all journal messages related to a specific executable, specify the + full path to the executable: + +&prompt.sudo;journalctl /usr/lib/systemd/systemd + + + + +Listed below are the common useful options to enhance the default journalctl behavior: + + + -f + + + Shows only the most recent journal messages, and prints new log entries + as they are added to the journal. + + + + + + + + Prints the messages and jumps to the end of the journal, so that the + latest entries are visible within the pager. + + + + + -r + + + Prints the messages of the journal in reverse order, so that the latest + entries are listed first. + + + + + -k + + + Shows only kernel messages. This is equivalent to the field match + _TRANSPORT=kernel. + + + + + -u + + + Shows only messages for the specified &systemd; unit. This is equivalent + to the field match + _SYSTEMD_UNIT=UNIT. + +&prompt.sudo;journalctl -u apache2 + + + + + For a complete list of options refer to man page, man 1 journalctl. + + + diff --git a/tasks/journald-configure.xml b/tasks/journald-configure.xml new file mode 100644 index 000000000..83f34bb0d --- /dev/null +++ b/tasks/journald-configure.xml @@ -0,0 +1,105 @@ + + + %entities; +]> + + + + + + + Configuring <command>systemd-journald</command> + + + + + You can configure the basic behavior of systemd-journald service by modifying /etc/systemd/journald.conf. + + + + To configure systemd-journald, proceed as follows: + + + Configuring <command>systemd-journald</command> + + Open the systemd-journald configuration file: +&prompt.sudo; vi /etc/systemd/journald.conf +[Journal] + +# Store journals on disk so logs survive reboot +Storage=persistent + +# Disk space limits +SystemMaxUse=2G +SystemKeepFree=10% + +# Per-file max size +SystemMaxFileSize=200M + +# RAM (runtime) journal limits +RuntimeMaxUse=500M +RuntimeKeepFree=10% + +# Log retention +MaxRetentionSec=4weeks + +# Compression for old logs +Compress=yes + +# Optional: also forward to rsyslog/syslog +ForwardToSyslog=yes + + + Modify the configurations: + + + + To modify the storage type, modify Storage=auto. + The available options are: + + volatile: RAM only (clears on reboot). + persistent: stored in /var/log/journal. + auto: persistent if directory exists, otherwise volatile. + none: no logs written to disk or memory. + + If the journal log data is saved to a persistent location, it uses up to 10% of the file system the /var/log/journal resides on. For example, if /var/log/journal is located on a 30 GB /var partition, the journal may use up to 3 GB of the disk space. To change this limit, change (and uncomment) the SystemMaxUse option: + SystemMaxUse=50M +&prompt.sudo; systemctl restart systemd-journald + + + To limit the log rate to prevent flooding logs: + RateLimitIntervalSec=30s +RateLimitBurst=1000 + + + + To send the journal to a terminal device to inform you about system messages on a preferred terminal screen, for example /dev/tty12: + + ForwardToConsole=yes +TTYPath=/dev/tty12 + + + To forward logs to syslog, modify ForwardToSyslog=yes. journald is backward compatible with traditional syslog implementations such as rsyslog. + + Install rsyslog:rpm -q rsyslog + Enable rsyslog:systemctl is-enabled rsyslog + Enable forwarding to rsyslog in /etc/systemd/journald.conf:ForwardToSyslog=yes + + For more information on file descriptions, see man 5 journald.conf. + + + + Save and restart systemd-journald. + &prompt.sudo; systemctl restart systemd-journald + + + diff --git a/tasks/journald-filter-journals.xml b/tasks/journald-filter-journals.xml new file mode 100644 index 000000000..a6743dd7f --- /dev/null +++ b/tasks/journald-filter-journals.xml @@ -0,0 +1,89 @@ + + + %entities; +]> + + + + + + + Filtering the journal output + + + + + This section describes how to refine searches in logs, for example, by boot number, + by a specific time interval, or to view specific data fields. By default, the journal lists + the oldest entries first. The output can be filtered using specific switches and fields. + + + + You can filter journals based on boot number, time interval, and fields. For details, refer to the following sections. + +
+ Filter logs based on a specific system boot + + To list logs for all the available boots, run the command as follows: + + &prompt.sudo;journalctl --list-boots + The first column lists the boot offset: 0 for the current boot, -1 for the previous one, + -2 for the one before that, etc. The second column contains the boot ID followed by the limiting time stamps of the specific + boot. + + To show all messages from the current boot: + + &prompt.sudo;journalctl -b + To view journal messages from the previous boot, add an offset + parameter. The following example command shows the previous boot messages: + &prompt.sudo;journalctl -b -1 + To view boot messages based on the boot ID, 156019a44a774a0bb0148a92df4af81b: + &prompt.sudo;journalctl _BOOT_ID=156019a44a774a0bb0148a92df4af81b +
+
+ Filtering logs based on time interval + You can filter the output of journalctl by specifying the starting and/or ending date. The date specification should be of the format YYYY-MM-DD H:MM:SS. If the time part is omitted, midnight is assumed. If seconds are omitted, :00 is assumed. If the date part is omitted, the current day is assumed. Instead of a numeric expression, you can specify the keywords yesterday, + today or tomorrow. They refer to midnight of the day + before the current day, of the current day, or of the day after the current day. If you specify + now, it refers to the current time. You can also specify relative times + prefixed with - or +, referring to times before or after + the current time. + + To view only new messages since now, and update the output continuously: + &prompt.sudo;journalctl --since "now" -f + To view all messages since last midnight until 3:20am: + &prompt.sudo;journalctl --since "today" --until "3:20" +
+
+ Filtering logs based on fields + + You can filter the output of the journal by specific fields. The syntax of a field to be matched is FIELD_NAME=MATCHED_VALUE, such + as _SYSTEMD_UNIT=httpd.service. You can specify multiple matches in a single query to filter the output messages even more. See + man 7 systemd.journal-fields for a list of default fields. + + To view messages produced by a specific process ID: PID_1039: +&prompt.sudo;journalctl _PID=1039To view messages belonging to a specific user ID: UID_1000: + &prompt.sudo; journalctl _UID=1000 + To view messages from the kernel ring buffer (the same as dmesg displays): + &prompt.sudo;journalctl _TRANSPORT=kernel + To view messages from the service's standard or error output: + &prompt.sudo;journalctl _TRANSPORT=stdout + To view messages produced by a specified service only: + &prompt.sudo;journalctl _SYSTEMD_UNIT=avahi-daemon.service + If two different fields are specified, only entries that match both expressions at the same time are shown: + &prompt.sudo;journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=1488 + If two matches refer to the same field, all entries matching either expression are shown: + &prompt.sudo;journalctl _SYSTEMD_UNIT=avahi-daemon.service _SYSTEMD_UNIT=dbus.service + You can use the + separator to combine two expressions in a logical OR. The following example shows all messages from the Avahi service process with the process ID 1480 together with all messages from the D-Bus service: + &prompt.sudo;journalctl _SYSTEMD_UNIT=avahi-daemon.service _PID=1480 + _SYSTEMD_UNIT=dbus.service +
+
diff --git a/tasks/systemd-journald-troubleshooting.xml b/tasks/systemd-journald-troubleshooting.xml new file mode 100644 index 000000000..2b69d8d89 --- /dev/null +++ b/tasks/systemd-journald-troubleshooting.xml @@ -0,0 +1,110 @@ + + + %entities; +]> + + + + + + + Troubleshooting &systemd; errors + + + + + This section introduces ways to interpret and fix the log messages retrieved through journalctl. + + + +
View &systemd; errors + + View the list of failed &systemd; units: + systemctl --failed +The list of all failed services appears. + Identify the severity and content of the error. + journalctl -p 0..7 -b + The list of errors with priority levels 0 to 7 appears. The errors are marked with priority level: + + + 0-2: emerg, alert, crit: Critical issues, system collapse imminent. + 3: err (Error): A service or application failed to complete a requested operation. + 4: warning : Something undesirable happened but is not an immediate failure. + 5-7: notice, info, debug: Normal operational information and developer diagnostics. + + +View the error log for the failing service using the following command: +journalctl -u <failing_service_name> +The lines preceding the termination message include information about the error. + + Exit Status: If a service stops, look for a message like Process xxx exited with status 1/FAILURE. A status of 0 indicates success. Any nonzero status indicates an error. + Configuration Errors: Messages containing phrases like No such file or directory, Permission denied or Address already in use usually point to a problem in the service's configuration file. + Out-of-Memory (OOM): The service was terminated because + it exceeded memory limits. + + +Use detailed views of logs to get detailed information about the error. +journalctl -xe +For example, to view all messages from the Apache service during the current boot with detailed explanations, run the following command: +journalctl -u httpd.service -b -xe + +
+
+ Troubleshoot &systemd; error + + + This section introduces an example to illustrate how to find and fix + the error reported by &systemd; during apache2 start-up. + + Start the apache2 service: +systemctl start apache2 +Job for apache2.service failed. See 'systemctl status apache2' and 'journalctl -xn' for details. + + View the service status: +&prompt.sudo;systemctl status apache2 +apache2.service - The Apache Webserver + Loaded: loaded (/usr/lib/systemd/system/apache2.service; disabled) + Active: failed (Result: exit-code) since Tue 2025-09-03 11:08:13 EDT; 7min ago + Process: 11026 ExecStop=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND \ + -k graceful-stop (code=exited, status=1/FAILURE) + The ID of the process causing the failure is 11026. + + + View the verbose version of messages related to process ID 11026: +&prompt.sudo;journalctl -o verbose _PID=11026 +[...] +MESSAGE=AH00526: Syntax error on line 6 of /etc/apache2/default-server.conf: +[...] +MESSAGE=Invalid command 'DocumenttRoot', perhaps misspelled or defined by a module +[...] + + Fix the typo inside /etc/apache2/default-server.conf, start the apache2 service, and print its status: +&prompt.sudo;systemctl start apache2 && systemctl status apache2 +apache2.service - The Apache Webserver + Loaded: loaded (/usr/lib/systemd/system/apache2.service; disabled) + Active: active (running) since Tue 2025-09-03 11:26:24 EDT; 4ms ago + Process: 11026 ExecStop=/usr/sbin/start_apache2 -D SYSTEMD -DFOREGROUND + -k graceful-stop (code=exited, status=1/FAILURE) + Main PID: 11263 (httpd2-prefork) + Status: "Processing requests..." + CGroup: /system.slice/apache2.service + ├─11263 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...] + ├─11280 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...] + ├─11281 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...] + ├─11282 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...] + ├─11283 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...] + └─11285 /usr/sbin/httpd2-prefork -f /etc/apache2/httpd.conf -D [...] + + + +
+