podman auto-update(.service) fails for containers in user directory.

[bjoli]

Issue Description

podman audo-update and podman-auto-update.service fails to restart services that have to be started using the systemctl --user flag.

The update works, but services placed in .config/containers/systemd are error with

* restarting unit beebop.service during rollback: Unit beebop.service not found.

Containers placed in the system-wide directory work as expected.

Steps to reproduce the issue

Steps to reproduce the issue

1. Put a .container file containing AutoUpdate=registry in the user directory under .config (also works for root). Make sure it is running properly
2. run systemctl [--user] start podman-auto-update.service
3. If there is an update, it will download but it will fail to restart.

Describe the results you received

I received the error mentioned above, and the container failed to restart since it didn't find the container, presumably by not telling systemctl to start the service as a --user.

Describe the results you expected

The same result as a container run without --user: that the image would update and the service would be restarted.

podman info output

host:                                                                                arch: amd64                                                                        buildahVersion: 1.37.3                                                             cgroupControllers:                                                                 - cpuset                                                                           - cpu                                                                              - io                                                                               - memory                                                                           - hugetlb                                                                          - pids                                                                             - rdma                                                                             - misc                                                                             cgroupManager: systemd
  cgroupVersion: v2                                                                  conmon:                                                                              package: conmon-2.1.12-1.1.x86_64                                                  path: /usr/bin/conmon                                                              version: 'conmon version 2.1.12, commit: unknown'                                cpuUtilization:                                                                      idlePercent: 99.76                                                                 systemPercent: 0.08                                                                userPercent: 0.16                                                                cpus: 48                                                                           databaseBackend: sqlite                                                            distribution:                                                                        distribution: opensuse-microos                                                     version: "20240927"                                                              eventLogger: journald                                                              freeLocks: 2026                                                                    hostname: burken.local                                                             idMappings:                                                                          gidmap: null                                                                       uidmap: null                                                                     kernel: 6.11.0-1-default                                                           linkmode: dynamic                                                                  logDriver: journald
  memFree: 125957226496
  memTotal: 134919950336
networkBackend: netavark
  networkBackendInfo:                                                                  backend: netavark                                                                  dns:                                                                                 package: aardvark-dns-1.12.2-1.1.x86_64                                            path: /usr/libexec/podman/aardvark-dns                                             version: aardvark-dns 1.12.2                                                     package: netavark-1.12.2-1.1.x86_64                                                path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.1.x86_64
    path: /usr/bin/crun
    version: |-                                                                          crun version 1.17                                                                  commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-20240906.6b38f07-2.1.x86_64
    version: |
      pasta 20240906.6b38f07-2.1
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true                                                                       path: /run/podman/podman.sock                                                    rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true                                                               seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 0h 48m 35.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.opensuse.org
  - registry.suse.com
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 13
    paused: 0
    running: 9
    stopped: 4
graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 938184347648
  graphRootUsed: 12894605312
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 5.2.3
  Built: 1727197645
  BuiltTime: Tue Sep 24 19:07:25 2024
  GitCommit: ""
  GoVersion: go1.23.1
  Os: linux
  OsArch: linux/am

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

The only reason I noticed is that I have friends with accounts on this server. One of them brought it up. It is probably not a common setup with several users running their own containers.

Additional information

No response

[vrothberg]

Thanks for the report. Can you try without --user but by logging into the user (e.g., via ssh)? I wonder whether there's some implication by --user.

Cc: @ygalblum @alexlarsson

[bjoli]

Trying without --user tries to start a system service and won't find the service file. That is what I suspect the podman-auto-update.service is doing (I have only tried this as root, but have the same error for files under /root/.config/containers/systemd )

It won't work at all for non-root-users.

[vrothberg]

Yes, I am not sure --user will work for this purpose. That's why I'd like to try to login as the non-root user and run it from there with proper login session.

[bjoli]

Yes, I am not sure --user will for this purpose. That's why I'd like to try to login as the non-root user and run it from there with proper login session.

The same thing as it does for the root account, except it asks for auth first.

==== AUTHENTICATION COMPLETE ==== Failed to restart beebop.service: Unit beebop.service not found.

Restarting it with systemctl --user restart beebop.service works fine.

[vrothberg]

Apologies, I was in a haste last week and didn't think clearly. For sure, --user is supposed to work.

Does a manual systemctl --user restart beebop.service work? Same for systemctl stop and start? It's odd that the service isn't found. What happens when you run podman auto-update manually?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[bjoli]

So, a manual restart of a user service works fine. Everything works as expected. The only thing is that there is no auto update of user images for services defined in the users systems container dir. The auto updater works fine for every podman container that can be started using systemctl start beebop.container - i.e. a container in the system directory.

The podman auto updater only works for those systemd units. If the units are placed in a home dir it will not work because it it will try to restart those units as if they were a system unit.

In the example in my initial post: systemctl --user restart beebop.service works. sysytemctl restart beebop.service does not since the service is not in the system wide directory. This is expected. What is not expected is the auto updater not recognizing this.

[vrothberg]

@ygalblum could this be an ordering/dependency issue where the auto-update.service runs before the Quadlets are generated?

[bjoli]

@ygalblum could this be an ordering/dependency issue where the auto-update.service runs before the Quadlets are generated?

I don't think so, because running podman auto-update fails as well. (Tried with root just now).

Does it try to restart user services when logged in as root or does it assume they are in the system directory?

[bjoli]

That seems like the case, because now podman auto-update works as a non-root user.

Perhaps It should trust the other users of my server and verify their claims better next time...

The issue for root still stands though.

[vrothberg]

Good news. So the rootless/user auto-update.service works but the rootful/system service fails?

[bjoli]

Not only that, podman auto-update fails if the containers are started from the .config/containers/systemd dir.