Journald input is not compatible with Journald < `v242`

[belimawr]

#47324 started using the argument --boot all when calling journalctl to solve #46933, however the support for the keyword all was introduced in Systemd v424 (see docs)

Some Linux distributions that we still support use an older version of journalctl, making the latest versions of Filebeat incapable of running the Journald input.

SUSE Linux Enterprise Server 12 SP5 is one of them (based on our tests). However it has reached end of general support on 31 Oct 2024, long term service pack support runs until 31 Oct 2027 with the core support extending even further, until October 2030.

Filebeat will fail to start journalctl with log messages like those:

{
  "@timestamp": "2025-12-10T11:12:13.138+0000",
  "ecs.version": "1.6.0",
  "input_source": "LOCAL_SYSTEM_JOURNAL",
  "log.level": "error",
  "log.logger": "input.journald.reader.journalctl-runner",
  "log.origin": {
    "file.line": 154,
    "file.name": "journalctl/journalctl.go",
    "function": "github.com/elastic/beats/v7/filebeat/input/journald/pkg/journalctl.Factory.func3"
  },
  "message": "journalctl exited with an error, exit code 1",
  "path": "LOCAL_SYSTEM_JOURNAL",
  "service.name": "filebeat"
}

The current workaround is to downgrade Filebeat to a version that does not contain #47324, like v8.19.6, v9.1.6 or v9.2.0.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[cmacknz]

As suggested in #48160 (comment), we should make at least the --all a configuration option like supports_all or similar, or we could auto-detect this.

[orestisfl]

Needs to be verified but jourlanctl <v242 does not need --boot all, -f implying -b was added in systemd/systemd@2dd9285 / v250

So we could dynamically do

func maybeAddBootAll(args []string) []string {
    if journalctl --version >= 242 {
        args = append(args, "--boot", "all")
    }
	return args
}

[belimawr]

I'm running V258 and I still need --boot all:

% journalctl --version 
systemd 258 (258.3-1-arch)
+PAM +AUDIT -SELINUX +APPARMOR -IMA +IPE +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +BTF +XKBCOMMON +UTMP -SYSVINIT +LIBARCHIVE

% journalctl --utc --no-pager --all --follow --no-tail | head -n 1
Dec 15 13:44:37 millennium-falcon kernel: Linux version 6.17.9-arch1-1 (linux@archlinux) (gcc (GCC) 15.2.1 20251112, GNU ld (GNU Binutils) 2.45.1) #1 SMP PREEMPT_DYNAMIC Mon, 24 Nov 2025 15:21:09 +0000
tiago@millennium-falcon ~ 

% journalctl --utc --no-pager --all --follow --no-tail --boot | head -n 1
Dec 15 13:44:37 millennium-falcon kernel: Linux version 6.17.9-arch1-1 (linux@archlinux) (gcc (GCC) 15.2.1 20251112, GNU ld (GNU Binutils) 2.45.1) #1 SMP PREEMPT_DYNAMIC Mon, 24 Nov 2025 15:21:09 +0000
tiago@millennium-falcon ~ 

% journalctl --utc --no-pager --all --follow --no-tail --boot all| head -n 1
Jun 17 13:00:08 millennium-falcon kernel: microcode: updated early: 0x4e -> 0x50, date = 2023-09-13

I decided to test Debian 12 and the results are even worse 😢, it just does not show previous boots with the --follow flag:

root@bookworm:/home/vagrant# journalctl --version
systemd 252 (252.39-1~deb12u1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified


root@bookworm:/home/vagrant# journalctl --boot=all --no-pager |head -n 1
Dec 16 01:36:16 bookworm kernel: Linux version 6.1.0-18-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01)

root@bookworm:/home/vagrant# journalctl --follow --boot=all --no-pager |head -n 1
Dec 18 14:43:52 bookworm systemd[1226]: Reached target default.target - Main User Target.

root@bookworm:/home/vagrant# journalctl --utc --no-pager --all --follow --no-tail --boot | head -n 1
Dec 18 14:43:09 bookworm kernel: Linux version 6.1.0-18-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01)

root@bookworm:/home/vagrant# journalctl --utc --no-pager --all --follow --no-tail --boot all| head -n 1
Dec 18 14:43:09 bookworm kernel: Linux version 6.1.0-18-amd64 (debian-kernel@lists.debian.org) (gcc-12 (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01)

Auto-detecting the version is an interesting option. I believe getting the major version is easy, at least in the 3 different ones I checked:

vagrant@ubuntu-jammy:~$ journalctl  --version
systemd 249 (249.11-0ubuntu3.17)

% journalctl --version
systemd 258 (258.3-1-arch)

vagrant@bookworm:~$ journalctl --version
systemd 252 (252.39-1~deb12u1)

I'm starting to like the idea of making the command arguments configurable as @cmacknz suggested on #48160 (comment). Relying on an external binary is proving to be unreliable when we add the variety of Linux distributions we support.

[orestisfl]

I'm running V258 and I still need --boot all:
...
I decided to test Debian 12 and the results are even worse 😢, it just does not show previous boots with the --follow flag:

Yes, anything V250 and above needs it.

I'm starting to like the idea of making the command arguments configurable as @cmacknz suggested on #48160 (comment). Relying on an external binary is proving to be unreliable when we add the variety of Linux distributions we support.

The decision logic is quite simple here though, we need the flag for all >=250 (or 242) versions. So far, we don't have any evidence of distributions customizing this further.

[cmacknz]

The decision logic is quite simple here though, we need the flag for all >=250 (or 242) versions. So far, we don't have any evidence of distributions customizing this further.

This makes it a good candidate for us to look at the output of journalctl --version and handle this automatically.

I still think customizing the command line arguments is a good escape hatch, but we shouldn't require that customers change them manually unless something is broken that we can't fix for them.