Rootless regression on Ubuntu 26.04: same source-built Mosquitto image works on 24.04 and rootful 26.04, but gets EACCES opening /mosquitto/config/mosquitto.conf on 26.04 rootless

[johanrydell]

Issue Description

I found a rootless regression across an Ubuntu upgrade.

The same source-built Mosquitto 2.1.1 image behaves like this:

• Ubuntu 24.04.4 LTS, rootless: works
• Ubuntu 26.04 LTS, rootful: works
• Ubuntu 26.04 LTS, rootless: fails

The failing case is opening /mosquitto/config/mosquitto.conf inside the container.

This is reproducible with a self-built image, not only the official eclipse-mosquitto image. The official image also shows the same rootless symptom on Ubuntu 26.04.

I found Podman issue #6989, which looks related in family because it also involves Mosquitto and Podman non-root behavior, but my case appears different: I can reproduce it with a source-built image, and I have a clear 24.04 vs 26.04 regression split. :contentReference[oaicite:0]{index=0}

The failure is reproducible with a self-built image as well as the official image, which suggests this is not only an official-image packaging issue.

### Steps to reproduce the issue

1. Build a source-based Mosquitto 2.1.1 image that:
   - creates user/group `mosquitto:mosquitto` with uid/gid 1883
   - copies config to `/mosquitto/config/mosquitto.conf`
   - installs the built broker as `/usr/sbin/mosquitto`

2. Run the same image with the same command in each environment:

```bash
podman run --rm -it \
  --entrypoint /usr/sbin/mosquitto \
  mosq-src-baseline \
  -c /mosquitto/config/mosquitto.conf -v


### Describe the results you received


```markdown
Results:

- Ubuntu 24.04.4 LTS rootless: works
- Ubuntu 26.04 LTS rootful: works
- Ubuntu 26.04 LTS rootless: fails

On Ubuntu 26.04 rootless, Mosquitto exits with:

```text
1776976898: Error: Unable to open config file '/mosquitto/config/mosquitto.conf'.
1776976898: mosquitto version 2.1.1 terminating

A rootless strace from the failing Ubuntu 26.04 case shows:
open("/mosquitto/config/mosquitto.conf", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOTDIR (Not a directory)
open("/mosquitto/config/mosquitto.conf", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied)

### Describe the results you expected

```markdown
I expected the same source-built image to behave consistently across the upgrade path.

Specifically, I expected Ubuntu 26.04 rootless to work the same way as:
- Ubuntu 24.04.4 rootless
- Ubuntu 26.04 rootful

Mosquitto should load `/mosquitto/config/mosquitto.conf` and start normally.

### podman info output

```yaml
host:
  arch: amd64
  buildahVersion: 1.42.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.13+ds1-2_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: unknown'
  cpus: 4
  databaseBackend: sqlite
  distribution:
    codename: resolute
    distribution: ubuntu
    version: "26.04"
  eventLogger: journald
  hostname: ubuntu-26lts
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 7.0.0-14-generic
  logDriver: journald
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.21-1ubuntu3_amd64
    path: /usr/bin/crun
  security:
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
    apparmorEnabled: false
  rootlessNetworkCmd: pasta
store:
  graphDriverName: overlay
  graphRoot: /home/mqttsrv/.local/share/containers/storage
  runRoot: /tmp/storage-run-1001/containers
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
version:
  Version: 5.7.0
  BuildOrigin: Ubuntu
  GoVersion: go1.25.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

- VMware virtual machine
- Ubuntu 26.04 LTS failing host: `ubuntu-26lts`
- Ubuntu 24.04.4 LTS working host: `t-work`
- Rootless uses:
  - `graphRoot: /home/mqttsrv/.local/share/containers/storage`
  - `runRoot: /tmp/storage-run-1001/containers`
- OCI runtime on failing host: `crun`
- Network backend: `netavark`

### Additional information

Cross-check summary:

- Same source-built image
- Same command
- Same config path: `/mosquitto/config/mosquitto.conf`

Results:
- Ubuntu 24.04.4 rootless: works
- Ubuntu 26.04 rootful: works
- Ubuntu 26.04 rootless: fails

In the failing Ubuntu 26.04 rootless case, `strace` shows:

```text
open("/mosquitto/config/mosquitto.conf", O_RDONLY|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOTDIR
open("/mosquitto/config/mosquitto.conf", O_RDONLY|O_LARGEFILE) = -1 EACCES (Permission denied)

[Luap99]

Please fix your output formatting and provide a proper reproducer with the actual full podman commands being used.

In general we only support the latest version upstream so it would be good if you can test podman 5.8.2. If you think this is a regression it would be helpful if you can actually narrow it down to in which podman version this started or do a full proper git bisect ideally.

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[johanrydell]

I retested with a simpler reproducer using only the official image and default Ubuntu packages.

Setup:

• Two freshly installed, fully patched systems

• Installed from Ubuntu 24.04 LTS and Ubuntu 26.04 LTS ISO images

• Only Podman installed via apt install podman

• Rootless tests use the default user created during Ubuntu install

• Image: docker.io/library/eclipse-mosquitto:2.1-alpine

Summary

Host | Mode | Podman | Result -- | -- | -- | -- Ubuntu 24.04 LTS | rootless | 4.9.3 | GOOD Ubuntu 26.04 LTS | rootless | 5.7.0 | FAIL Ubuntu 26.04 LTS | rootful | 5.7.0 | GOOD

Reproducer

podman run --rm -it docker.io/library/eclipse-mosquitto:2.1-alpine

Ubuntu 24.04 LTS rootless: GOOD

1777057177: Info: running mosquitto as user: mosquitto.
1777057177: mosquitto version 2.1.2 starting
1777057177: Config loaded from /mosquitto/config/mosquitto.conf.
...
1777057177: mosquitto version 2.1.2 running

Podman version:

Version: 4.9.3
BuildahVersion: 1.33.7
crun version 1.14.1
kernel: 6.8.0-110-generic
distribution: Ubuntu 24.04 / noble
rootless: true
graphDriverName: overlay
Native Overlay Diff: "true"
Supports shifting: "false"
runRoot: /run/user/1000/containers
graphRoot: /home/johanr/.local/share/containers/storage

Ubuntu 26.04 LTS rootless: FAIL

1777057182: Error: Unable to open config file '/mosquitto/config/mosquitto.conf'.
1777057182: mosquitto version 2.1.2 terminating

Podman version:

Version: 5.7.0
BuildahVersion: 1.42.1
crun version 1.21
kernel: 7.0.0-14-generic
distribution: Ubuntu 26.04 / resolute
rootless: true
graphDriverName: overlay
Native Overlay Diff: "true"
Supports shifting: "false"
runRoot: /run/user/1000/containers
graphRoot: /home/johanr/.local/share/containers/storage

Ubuntu 26.04 LTS rootful: GOOD

1777057731: Info: running mosquitto as user: mosquitto.
1777057731: mosquitto version 2.1.2 starting
1777057731: Config loaded from /mosquitto/config/mosquitto.conf.
...
1777057731: mosquitto version 2.1.2 running

Podman version:

Version: 5.7.0
BuildahVersion: 1.42.1
crun version 1.21
kernel: 7.0.0-14-generic
distribution: Ubuntu 26.04 / resolute
rootless: false
graphDriverName: overlay
Supports shifting: "true"
runRoot: /run/containers/storage
graphRoot: /var/lib/containers/storage

So the failure is now reproducible with the official eclipse-mosquitto:2.1-alpine image and a single command, not just my source-built image.

The key split appears to be:

• Same official image works on Ubuntu 24.04 rootless

• Same official image fails on Ubuntu 26.04 rootless

• Same official image works on Ubuntu 26.04 rootful

This suggests the issue is specific to the Ubuntu 26.04 rootless stack, rather than Mosquitto itself or only my custom image.

ubuntu24_rootless_mosquitto_good.txt
ubuntu26_rootful_mosquitto_good.txt
ubuntu26_rootless_mosquitto_failed.txt

[johanrydell]

Update: I built and tested current upstream Podman main.

After fixing rootless storage config, rootless Podman now uses the expected paths:

rootless: true
graphRoot: /home/johanr/.local/share/containers/storage
runRoot: /run/user/1000/containers

However, the original Mosquitto failure still reproduces:

podman run --rm -it docker.io/library/eclipse-mosquitto:2.1-alpine

Result:

1777059782: Error: Unable to open config file '/mosquitto/config/mosquitto.conf'.
1777059782: mosquitto version 2.1.2 terminating

Tested with upstream Podman:

Version: 6.0.0-dev
APIVersion: 6.0.0-dev
GitCommit: a8c36318565ddc35e5b3b980fc29264949041390
BuiltTime: Fri Apr 24 19:28:17 2026
GoVersion: go1.26.0

Environment:

Ubuntu 26.04 / resolute
kernel: 7.0.0-14-generic
rootless: true
OCI runtime: crun 1.21
storage: overlay
Native Overlay Diff: true
Supports shifting: false

So this is not fixed by current upstream main. The earlier /run/containers/storage issue was a separate Ubuntu packaging/config issue, but after correcting that, the Mosquitto rootless failure remains.

ubuntu26_rootless_mosquitto_failed_Podman_6_0_0-dev.txt

[Luap99]

podman run --rm -it docker.io/library/eclipse-mosquitto:2.1-alpine

This runs just fine on podman 5.8.2 on fedora 43 so this seems to suggest the problem is with ubuntu not the upstream code. As such please report problems through your distro bug tracker. ubuntu uses apparmor so maybe check that.

[YShow]

Unfortunatelly im facing the same problem in Opensuse Tumbleweed @johanrydell , have u had any luck?

[johanrydell]

Sorry, my solution is to stay on Ubuntu 24.04 LTS. I opened a question at Ubuntu, but no solution found there. It might be a 'podman' issue after all as you now found it in Opensuse.

[mdc-webdb]

I have see the same problem. It looks like SELinux is broken under the Ubuntu 26.04 container.
Works:
als none root:
podman run --rm --entrypoint '["tail", "-f", "/dev/null"]' docker.io/library/ubuntu:24.04
Fails:
podman run --rm --entrypoint '["tail", "-f", "/dev/null"]' docker.io/library/ubuntu:26.04
Workaround:
podman run --rm --security-opt label=disable --entrypoint '["tail", "-f", "/dev/null"]' docker.io/library/ubuntu:26.04