Update logging for containerization

guides/common/assembly_managing-logs-for-project.adoc

@@ -0,0 +1,20 @@
+include::modules/con_managing-logs-for-project.adoc[]
+
+include::modules/con_logging-architecture.adoc[leveloffset=+1]
+
+include::modules/proc_viewing-logs-with-journalctl.adoc[leveloffset=+1]
+
+include::modules/proc_configuring-log-levels-at-installation-time.adoc[leveloffset=+1]
+
+include::modules/proc_configuring-service-log-levels-at-runtime.adoc[leveloffset=+1]
+
+include::modules/ref_foremanctl-log-level-parameters.adoc[leveloffset=+2]
+
+include::modules/ref_logging-exceptions-limitations.adoc[leveloffset=+2]
+
+include::modules/proc_collecting-sosreport-data.adoc[leveloffset=+1]
+
+include::modules/proc_configuring-journald-retention.adoc[leveloffset=+1]
+
+include::modules/ref_journald-retention-parameters.adoc[leveloffset=+2]
+

guides/common/modules/con_logging-architecture.adoc

@@ -0,0 +1,21 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="logging-architecture-overview"]
+= Logging architecture overview
+
+[role="_abstract"]
+{ProjectServer} routes all service output through `systemd` and `journald`.
+You can use standard `journalctl` commands to view, filter, and manage logs for all {Project} services.
+
+{Project} uses `stdout` logging by default and logs are accessible through a single interface: the `journalctl` command.
+
+ifdef::satellite,katello,orcharhino[]
+Not all services support per-service log level configuration through `{foremanctl}`.
+Services such as Candlepin, Pulp, PostgreSQL, and `httpd` do not have dedicated log level parameters in `{foremanctl} deploy`.
+To adjust the log level for these services, use their native configuration mechanisms.
+
+Valkey uses a different log level vocabulary than other {Project} services.
+Instead of the standard `debug`, `info`, `warn`, `error`, and `fatal` levels, Valkey uses `debug`, `verbose`, `notice`, `warning`, and `nothing`.
+endif::[]
+
+NOTE: In disconnected environments, the core journald-based logging architecture works the same way as in connected environments because all logging is local to the host.

guides/common/modules/con_managing-logs-for-project.adoc

@@ -0,0 +1,8 @@
+:_mod-docs-content-type: CONCEPT
+
+[id="managing-logs-for-project"]
+= Managing logs for {Project}
+
+[role="_abstract"]
+Centralized logging gives you a single interface to monitor {Project} service health, diagnose errors, and control log retention and disk usage.
+You can also collect diagnostic data and attach it to support cases.

guides/common/modules/proc_collecting-sosreport-data.adoc

@@ -0,0 +1,60 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="collecting-diagnostic-data-with-sos-report"]
+= Collecting diagnostic data with sos report
+
+[role="_abstract"]
+You can collect diagnostic data from {ProjectServer} by using the `sos report` command and attach the resulting archive to a {Team} support case.
+The `sos` plugins automatically collect `journald` logs, container state, service configuration, and {SmartProxy} data.
+
+.Prerequisites
+ifdef::satellite[]
+* {RHEL} 9 with `fapolicyd` version 1.6-2 or later is installed.
+endif::[]
+* You have root access to the {ProjectServer} host.
+
+.Procedure
+. Run the `sos report` command on the {ProjectServer} host:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# sos report
+----
++
+The command collects configuration and diagnostic information, including `journald` logs for all {Project} services, container state, and service configuration files.
+. Wait for the collection to complete.
+The command displays the path to the generated archive file when finished.
+. Note the path to the archive file.
+The output includes a line similar to the following:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+Your sosreport has been generated and saved in:
+	/var/tmp/sosreport-_hostname_-_date_.tar.xz
+----
+. Attach the archive file to your {Team} support case.
++
+[IMPORTANT]
+====
+The `sos report` command removes security information such as passwords, tokens, and keys while collecting information.
+However, the archive can still contain sensitive data about the {ProjectServer}.
+Send the archive directly to the intended recipient and not to a public target.
+====
+
+.Verification
+. Verify that the archive file exists at the reported path:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# ls -lh /var/tmp/sosreport-*.tar.xz
+----
+. Verify that the file size is reasonable for your environment.
+A typical `sos report` archive ranges from tens to hundreds of megabytes depending on the log volume and configuration complexity.
+
+.Additional resources
+ifdef::satellite[]
+* https://access.redhat.com/solutions/3592[What is a sosreport and how can I create one?]
+endif::[]
+ifndef::satellite[]
+* https://github.com/sosreport/sos/wiki#for-users[_SOS user documentation_]
+endif::[]

guides/common/modules/proc_configuring-journald-retention.adoc

@@ -0,0 +1,96 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="configuring-journald-log-retention-and-disk-space-limits"]
+= Configuring journald log retention and disk space limits
+
+[role="_abstract"]
+Configure `journald` retention settings to prevent unbounded log growth on the `/var` partition.
+These settings replace the `logrotate` configuration used in traditional deployments.
+
+.Prerequisites
+* You have root access to the {ProjectServer} host.
+* You understand that containerized {ProjectServer} routes all logs through `journald`.
+For more information, see xref:logging-architecture-overview[].
+
+.Procedure
+. Create the drop-in configuration directory if it does not exist:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# mkdir -p /etc/systemd/journald.conf.d
+----
+. Create a drop-in configuration file with your retention settings:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# cat > /etc/systemd/journald.conf.d/10-{project-context}-retention.conf << EOF
+[Journal]
+Storage=persistent
+SystemMaxUse=10G
+SystemKeepFree=5G
+MaxRetentionSec=4week
+MaxFileSec=1week
+Compress=yes
+EOF
+----
++
+[NOTE]
+====
+Do not edit `/etc/systemd/journald.conf` directly.
+Use a drop-in file in `/etc/systemd/journald.conf.d/` so that your settings are preserved during system updates.
+====
+. Restart the `systemd-journald` service to apply the new settings:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# systemctl restart systemd-journald
+----
+. Verify the current disk usage of journal files:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# journalctl --disk-usage
+----
+. Optional: To reduce journal size immediately, rotate the current journal files and vacuum old entries.
++
+To remove journal files older than a specific time period:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# journalctl --rotate
+# journalctl --vacuum-time=2weeks
+----
++
+To reduce journal files to a specific size:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# journalctl --rotate
+# journalctl --vacuum-size=5G
+----
+
+.Verification
+* Confirm that the retention limits are applied by running:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# journalctl --disk-usage
+----
++
+The reported usage must not exceed the `SystemMaxUse` value after `journald` has completed its next rotation cycle.
+
+.Troubleshooting
+* If `SystemMaxUse` does not appear to take effect,
+ifdef::satellite[]
+see https://access.redhat.com/solutions/7026910[Known issue with SystemMaxUse] in the _{Team} Knowledgebase_.
+endif::[]
+ifndef::satellite[]
+check that the `Storage=persistent` setting is enabled and that the `/var/log/journal/` directory exists.
+endif::[]
+Ensure that the drop-in file name begins with a number so that it is loaded in the correct order.
+
+.Additional resources
+* xref:journald-retention-and-disk-space-parameters[]
+ifdef::satellite[]
+* {RHELDocsBaseURL}9/html-single/configuring_basic_system_settings/index#configuring-a-persistent-journal_assembly_troubleshooting-problems-using-log-files[Configuring a persistent journal in _{RHEL}{nbsp}9 Configuring basic system settings_]
+endif::[]

guides/common/modules/proc_configuring-log-levels-at-installation-time.adoc

@@ -0,0 +1,66 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="configuring-service-log-levels-at-installation-time"]
+= Configuring service log levels at installation time
+
+[role="_abstract"]
+At the initial deployment, when the containers are first created, you can configure the log level for {Project} services by using `{foremanctl} deploy` parameters.
+
+Increasing the log level to `debug` provides detailed output for troubleshooting a specific service, while resetting it to the default level reduces log volume in production.
+
+The `--log-level` parameter sets a global baseline log level for all services that support per-service log level configuration.
+You can override the global level for individual services by using service-specific parameters such as `--foreman-log-level` and `--foreman-proxy-log-level`.
+
+.Procedure
+* To set the global log level for all supported services, run `{foremanctl} deploy` with the `--log-level` option:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} deploy --log-level debug
+----
++
+Valid values are `debug`, `info`, `warn`, `error`, and `fatal`.
+The default value is `info`.
+
+* To override the log level for a specific service, use the service-specific parameter.
++
+For example, to set only Foreman to `debug` while keeping other services at the global level:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} deploy --foreman-log-level debug
+----
++
+To set only {SmartProxy} to `debug`:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} deploy --foreman-proxy-log-level debug
+----
++
+After you complete debugging, reset the log level to the default:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} deploy --log-level info
+----
+
+.Verification
+* Verify that the log level change took effect by checking the `journalctl` output for the affected service:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# journalctl -u foreman --since "1 minute ago"
+----
++
+When the log level is set to `debug`, the output contains detailed diagnostic messages.
+
+[NOTE]
+====
+Not all services support per-service log level parameters in `{foremanctl} deploy`.
+ifdef::satellite,katello,orcharhino[]
+Candlepin, Pulp, PostgreSQL, and `httpd` do not have dedicated log level parameters.
+endif::[]
+For a list of supported parameters, see xref:foremanctl-deploy-log-level-parameters[].
+For exceptions and limitations, see xref:logging-exceptions-limitations[].
+====

guides/common/modules/proc_configuring-service-log-levels-at-runtime.adoc

@@ -0,0 +1,21 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="configuring-service-log-levels-at-runtime"]
+= Configuring service log levels at runtime
+
+[role="_abstract"]
+You can temporarily elevate the log level to `debug` to capture failure details, resolve the issue, and then drop it back down to `info` to avoid filling up host disk space.
+
+.Prerequisites
+* {ProjectServer} is installed and running.
+* You have root access to the {ProjectServer} host.
+
+.Procedure
+* To change the runtime log level of running containers, run:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} logs set-level debug
+----
+
+.Verification

guides/common/modules/proc_viewing-logs-with-journalctl.adoc

@@ -0,0 +1,55 @@
+:_mod-docs-content-type: PROCEDURE
+
+[id="viewing-service-logs-with-journalctl"]
+= Viewing service logs with journalctl
+
+[role="_abstract"]
+You can view and filter logs for any containerized {Project} service by using `journalctl` with the appropriate unit or syslog identifier filter.
+Monitoring service output helps you verify service health, troubleshoot issues, and follow log output in real time.
+
+.Prerequisites
+* {ProjectServer} is installed and running.
+* You have root access to the {ProjectServer} host.
+
+.Procedure
+* View logs for a specific service.
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} logs show _service_name_
+----
++
+For example, to view 100 most recent logs of Foreman:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} logs show foreman --tail 100
+----
+
+* Filter logs by time range.
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} logs show _service_name_ --since _YYYY-MM-DD HH:MM:SS_
+----
++
+For example, to view Pulp logs since 8:00 AM on July 15, 2026:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# {foremanctl} logs show pulp --since "2026-07-15 08:00:00"
+----
+
+.Verification
+* Confirm that log output appears for the targeted service.
+If no output appears, verify that the service is running:
++
+[options="nowrap", subs="+quotes,verbatim,attributes"]
+----
+# systemctl status _service_name_
+----
+
+.Additional resources
+ifdef::satellite[]
+* {RHELDocsBaseURL}9/html-single/configuring_basic_system_settings/index#viewing-logs-using-the-command-line_assembly_troubleshooting-problems-using-log-files[Viewing logs using the command line in _{RHEL}{nbsp}9 Configuring basic system settings_]
+endif::[]