LP#2125549 "ssh agent no longer work a questing upgrade"

[github-actions]

Reported first on Launchpad at https://bugs.launchpad.net/ubuntu/+source/xubuntu-meta/+bug/2125549

[github-actions]

LP#0: Andreas Schultz (aschultz) wrote on 2025-09-24:

ssh agent worked fine in XUbuntu 25.04. After upgrading to 25.10 (beta) it no longer works.

• ssh-add reports "Could not open a connection to your authentication agent."
• ssh keeps asking for key password
• there is no graphical dialog asking for passwords when adding ssh-keys anymore
• no SSH variable in terminal environment

ProblemType: Bug
DistroRelease: Ubuntu 25.10
Package: openssh-client 1:10.0p1-5ubuntu4
ProcVersionSignature: Ubuntu 6.17.0-4.4-generic 6.17.0-rc5
Uname: Linux 6.17.0-4-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.33.1-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: XFCE
Date: Wed Sep 24 07:58:24 2025
InstallationDate: Installed on 2017-09-26 (2920 days ago)
InstallationMedia: Ubuntu-Server 17.10 "Artful Aardvark" - Alpha amd64 (20170924)
RelatedPackageVersions:
ssh-askpass N/A
libpam-ssh N/A
keychain N/A
ssh-askpass-gnome N/A
SSHClientVersion: OpenSSH_10.0p2 Ubuntu-5ubuntu4, OpenSSL 3.5.2 5 Aug 2025
SourcePackage: openssh
UpgradeStatus: Upgraded to questing on 2025-09-22 (2 days ago)

[github-actions]

LP#1: Andreas Schultz (aschultz) wrote on 2025-09-24:

Attachments:

• Dependencies.txt
• ProcCpuinfoMinimal.txt
• ProcEnviron.txt

[github-actions]

LP#2: Nick Rosbrook (enr0n) wrote on 2025-09-24:

This appears to be a breakage related to ssh-agent.socket.

On my questing system, ssh-socket.service is listening on $XDG_RUNTIME_DIR/openssh_agent, but SSH_AUTH_SOCK is not set in my environment.

ubuntu@q-vm:~$ systemctl --user status ssh-agent.socket
● ssh-agent.socket - OpenSSH Agent socket
Loaded: loaded (/usr/lib/systemd/user/ssh-agent.socket; enabled; preset: enabled)
Active: active (listening) since Wed 2025-09-24 13:49:25 UTC; 6min ago
Invocation: 23fe2f1f95f04f8386666cc05887bdd3
Triggers: ● ssh-agent.service
Docs: man:ssh-agent(1)
Listen: /run/user/1000/openssh_agent (Stream)
Process: 25258 ExecStartPost=/usr/bin/systemctl --user set-environment SSH_AUTH_SOCK=/run/user/1000/openssh_agent (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 986)
Memory: 4K (peak: 1.7M)
CPU: 4ms
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/ssh-agent.socket

Sep 24 13:49:25 q-vm systemd[25248]: Starting ssh-agent.socket - OpenSSH Agent socket...
Sep 24 13:49:25 q-vm systemd[25248]: Listening on ssh-agent.socket - OpenSSH Agent socket.
ubuntu@q-vm:~$ env | grep SSH_

It's unclear to me if the systemctl --user set-environment call in ssh-agent.socket is intended to hook things up in the user session, but that's not what it does.

I will keep digging.

See also:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961311
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1039919
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1103037

[github-actions]

LP#3: Nick Rosbrook (enr0n) wrote on 2025-09-24:

Okay, I confirmed that it's wrong to assume that:

ExecStartPost=/usr/bin/systemctl --user set-environment SSH_AUTH_SOCK=/run/user/1000/openssh_agent

will do the right thing in the general case. This does correctly set the environment in the user manager, and if you launch an application like e.g.:

$ systemd-run --user --wait --pipe -q -- ssh-add

then the environment is inherited, and $SSH_AUTH_SOCK is set. But, if one has a session spawned by login(1), this is not going to happen.

Maybe as a quick fix, we could patch openssh to try $XDR_RUNTIME_DIR/openssh_agent (the default) if $SSH_AUTH_SOCK is unset.

[github-actions]

LP#4: Andreas Schultz (aschultz) wrote on 2025-09-25:

The behavior of the environment is also explained in this systemd GH issue systemd/systemd#29414

Gnome uses a special terminal server service that inherits systemd user environment to launch terminals.

[github-actions]

LP#5: Alessandro Astone (aleasto) wrote on 2025-09-29:

I'm not sure ssh-agent.socket is at all related here.
On a desktop system like Xubuntu you were probably using the gnome-keyring ssh agent (at $XDG_RUNTIME_DIR/keyring/ssh). In questing that was replaced by the gcr4 ssh agent (at $XDG_RUNTIME_DIR/gcr4/ssh), managed by /usr/lib/systemd/user/gcr4-ssh-agent.socket. Both gnome-keyring and gcr4-ssh-agent use systemctl --user set-environment to expose the socket path.

systemctl --user set-environment affects the environment of all units started after that call. If the systemd startup sequence of your desktop is correct, the desktop shell will be started after the gcr4-ssh-agent.socket so that the shell will get SSH_AUTH_SOCK set and it will correctly pass it to its children apps.

[github-actions]

LP#6: Nick Rosbrook (enr0n) wrote on 2025-09-30:

I thought I tested in a full questing GNOME desktop environment, but apparently not, because in those cases the SSH_AUTH_SOCK is set accordingly by gcr-ssh-agent.socket (which I didn't know existed).

It's still true that systemctl --user set-envrionment only works when the process is launched as a child of the systemd user manager, e.g. with StartTransientUnit or systemd-run. For example, an SSH session will not have SSH_AUTH_SOCK set.

So, I guess it depends on whether xfce is actually launching apps as systemd units? I know nothing about Xubuntu specifically.

Andreas - can you please provide the output of:

$ systemctl --user status gcr-ssh-agent.socket gcr-ssh-agent.service
$ systemctl --user status gnome-keyring-daemon.socket gnome-keyring-daemon.service

Maybe there are other relevant units for Xubuntu?

[github-actions]

LP#7: Alessandro Astone (aleasto) wrote on 2025-09-30:

Andreas - can you please provide the output of:

And also just echo $SSH_AUTH_SOCK. We don't know yet what it is set to in their case.

For example, an SSH session will not have SSH_AUTH_SOCK set.

Mh, yeah ssh-agent.socket should probably be fixed to expose its SSH_AUTH_SOCK to the remote shell.

[github-actions]

LP#8: Launchpad Janitor (janitor) wrote on 2025-10-01:

This bug was fixed in the package openssh - 1:10.0p1-5ubuntu5

---

openssh (1:10.0p1-5ubuntu5) questing; urgency=medium

• test: workaround test failure caused by uutils dd (LP: #2125943)
• authfd: fallback to default if $SSH_AUTH_SOCK is unset (LP: #2125549)

-- Nick Rosbrook Mon, 29 Sep 2025 14:43:07 -0400

[github-actions]

LP#9: Nick Rosbrook (enr0n) wrote on 2025-10-01:

Okay, I have had some assistance in debugging on Xubuntu 25.04 vs 25.10, and this is what I have at the moment.

(1) openssh is indeed not a factor for the desktop use case, though ssh-agent.socket uses the same method for setting the environment, and has the same drawback

(2) It does not appear that xfce launches applications as transient systemd user units. To see this, try running the following from a terminal in a graphical session:

$ systemctl --user status $$

On GNOME, you will see something like:

ubuntu@q-vm:~$ systemctl --user status $$
● ptyxis-spawn-00f46578-4dcb-472e-8bda-8ea66aa37249.scope - [systemd-run] /usr/bin/bash
Loaded: loaded (/run/user/1000/systemd/transient/ptyxis-spawn-00f46578-4dcb-472e-8bda-8ea66aa37249.scope; transient)
Transient: yes
Active: active (running) since Wed 2025-10-01 22:18:25 UTC; 8min ago
Invocation: 0e1008e4a3964294b70663380994d239
Tasks: 3 (limit: 3936)
Memory: 5.6M (peak: 22M)
CPU: 514ms
CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/ptyxis-spawn-00f46578-4dcb-472e-8bda-8ea66aa37249.scope
├─3590 /usr/bin/bash
├─3900 systemctl --user status 3590
└─3901 pager

Oct 01 22:18:25 q-vm systemd[2034]: Started ptyxis-spawn-00f46578-4dcb-472e-8bda-8ea66aa37249.scope - [systemd-run] /usr/bin/bash

On xfce (and e.g. an SSH session), you will see:

ubuntu@q-vm:~$ systemctl --user status $$
PID 3860 does not belong to any loaded unit.

(3) So, given that we moved from gnome-keyring to gcr4, my guess is that gnome-keyring had another method of setting the SSH_AUTH_SOCK variable in addition to the systemctl --user set-environment call in the socket unit, and that gcr4 has no such mechanism. I have not verified this, but setting gcr4 as an affected package based on that idea.

[github-actions]

LP#10: Sean Davis (bluesabre) wrote on 2025-10-01:

Confirming the bug and documented behavior so far:

bluesabre@xubuntu-questing:$ ssh-add
Could not open a connection to your authentication agent.
bluesabre@xubuntu-questing:$ ssh -T
Enter passphrase for key '/home/bluesabre/.ssh/id_rsa':
Hi bluesabre! You've successfully authenticated, but GitHub does not provide shell access.
bluesabre@xubuntu-questing:$ ssh -T
Enter passphrase for key '/home/bluesabre/.ssh/id_rsa':
Hi bluesabre! You've successfully authenticated, but GitHub does not provide shell access.
bluesabre@xubuntu-questing:$ echo $SSH_AUTH_SOCK

bluesabre@xubuntu-questing:~$ systemctl --user status $$
PID 2706 does not belong to any loaded unit.

[github-actions]

LP#11: Sean Davis (bluesabre) wrote on 2025-10-02:

Plucky does not exhibit the same behavior.

sean-davis@sean-davis-Standard-PC-Q35-ICH9-2009:$ ssh-add
sean-davis@sean-davis-Standard-PC-Q35-ICH9-2009:$ echo $SSH_AUTH_SOCK
/run/user/1000/keyring/ssh

[github-actions]

LP#12: Alessandro Astone (aleasto) wrote on 2025-10-02:

(2) It does not appear that xfce launches applications as transient systemd user units. To see this, try > running the following from a terminal in a graphical session:

That's not necessary, provided that the graphical shell process has the environment set and exports it when forking to the app process.

(3) So, given that we moved from gnome-keyring to gcr4, my guess is that gnome-keyring had another method of setting the SSH_AUTH_SOCK variable in addition to the systemctl --user set-environment call in the socket unit, and that gcr4 has no such mechanism.

Ok, xfce4-session has special integration with gnome-keyring-daemon.
It runs gnome-keyring-daemon which prints out some key-value-pairs to stdout and xfce4-session uses those to set its own environment:

$ gnome-keyring-daemon --start 2>/dev/null
GNOME_KEYRING_CONTROL=/run/user/1000/keyring
SSH_AUTH_SOCK=/run/user/1000/keyring/ssh

See https://salsa.debian.org/xfce-team/desktop/xfce4-session/-/blob/972de48270971c2dc872bb5608ddac4a9ad8e0a1/xfce4-session/xfsm-compat-gnome.c#L158

Now that gnome-keyring-daemon no longer provides an ssh agent, xfce4-session doesn't know where to find one anymore.

I suggest a small patch to xfce4-session to try and find other known sockets, in order:
$XDG_RUNTIME_DIR/gcr/ssh
$XDG_RUNTIME_DIR/openssh_agent

Long-term, xfce4-session should invest in moving its session startup to be systemd-based. But that's definitely out of scope for 25.10