Broken pipe after 60 seconds with start-session and SSO

[avylove]

I originally reported this in aws/amazon-ssm-agent#626, but, after reaing through other issues, it seems like it might be more of a client issue.

When using start-session, the connection will fail with a broken pipe if there is no input or output for 60 seconds.

Cannot perform start session: write tcp 1.2.3.4:45678->100.99.98.97:443: write: broken

As long as there is input or output, the connection will hold.

The shell is still active on the instance, it just seems the socket connection get's dropped.

Unable to find any indicators in the logs or strace.

The weird thing I found is this only seems to happen when using an SSO profile to connect and does not seem to occur when using access keys.

I'm wondering if somehow it's a side effect of the changes identified in #83 (comment)

Client OS: Fedora 41
AWS CLI: 2.17.18
SSM plugin: 1.2.707.0

Instance OS: Rocky 9 (Official AMI with LVM),
SSM Agent: 3.3.1957.0
Same behavior on an Ubuntu instance

[avylove]

I was able to further isolate the failure condition, It only seems to fail under these conditions:

• SSO (Access keys work fine)
• Over VPN (direct works fine)
• No I/O for 60 seconds (inbound or outbound)

SSO over VPN (Fails)
Looking at packet traces for the ssmmessages stream, it seems normal with a keep-alive every 15 seconds, but 1 to 2 seconds before the 4th keepalive, the server sends a FIN, ACK using the same sequence numbers as the keepalives. The client responds with an ACK. Then, if you enter something on the terminal, the client send application data to the server. In response, the server send back an RST to the client.

However, just after receiving the FIN,ACK, the client establishes a connection with ssm.us-west-2.amazon.us. I'm not seeing an ssm stream before this. Is that taken care of by SSO? It does not establish a new connection with ssmmessages. Perhaps this is the issue? It seems to respond to the ssmmessages stream closing by establishing a connection to ssm, but tries to use the existing, now failed, ssmmessages stream. Is it supposed to be establishing a connection to ssmmessages instead?

Access keys over VPN (Successful)
This behaves the same, except after recieving the FIN,ACK it establishes a new connection to ssm.us-west-2.amazon.us and then ssmmessages.us-west-2.amazon.us.

SSO direct (Successful)
No FIN, ACK is sent. Keep-alives continue

[avylove]

I was able to get debug output and am seeing the failure

ERROR Resume Session failed: refresh cached SSO token failed, missing required field ExpiresIn

That's generated here:

session-manager-plugin/src/sessionmanagerplugin/session/sessionhandler.go

Line 113 in b2b0bcd

log.Errorf("Resume Session failed: %v", err)

This seems to call ResumeSessionRequest in vendor/src/github.com/aws/aws-sdk-go/service/ssm/api.go which calls https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_ResumeSession.html

I'm not sure from here, but looking at the SSO token, it uses expiresAt, not ExpiresIn

[avylove]

More digging. It looks like the error is from here.
https://github.com/aws/session-manager-plugin/blob/b2b0bcd769d1c0693f77047360748ed45b09a72b/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/token_provider.go#L114C1-L115C1

It seems to get here because it the existing token does not evaluate as valid and then it tries to renew it. My guess is it's looking at the wrong token. I wonder if it's because sdkutil.SetRegionAndProfile() isn't called before sdkutil.GetnewSessionWithEndpoints() like it is when establishing sessions.

Port forwarding session also seem to fail with SSO. It establishes the connection, then gets an error for use of a closed port. Has any of this been tested? SSO is the AWS-recommended and more secure authentication mechanism, yet basic tools seem to be failing.

[Yangtao-Hua]

Hello @avylove!
Thanks for reaching out to SSM Session Manager team, and appreciate for the dive deep!
We are working on the issue reproduce and researching the root cause. Will update if there is any findings.

Thanks,
Yangtao

[avylove]

@Yangtao-Hua Were you able to find the root cause?

[Yangtao-Hua]

Hi @avylove,

I wasn’t able to reproduce the issue. Below are the steps I followed:

1. Created an SSO user with admin access, then ran the following command to refresh the token:
   aws sso login --profile AdministratorAccess-12345

2. Started a standard stream session to the target instance:
   aws ssm start-session --target i-XXXXXX --profile AdministratorAccess-12345

3. Kept the session open for more than 3 minute after verifying the connection was successful.

4. Started another port forwarding session:
   aws ssm start-session --target i-XXXXXX --document-name AWS-StartPortForwardingSession --parameters "localPortNumber=9999" --profile AdministratorAccess-12345

5. Opened http://localhost:9999 in the browser, confirmed it connected to the server on the target instance, then closed the webpage.

6. Waited for 5 minutes, then opened http://localhost:9999 again.

I was using a VPN during this process.
My session-manager-plugin version is 1.2.707.0 and amazon-ssm-agent version is 3.3.1957.0.

Would you mind sharing more details on how to reproduce this issue on your side?
Also, I noticed you mentioned a ResumeSession—just curious, what triggered the disconnection that caused it? Was it automatic or possibly due to a network issue?

Thanks!

[avylove]

I'm not sure what causes the server to close the stream, but it happens every 60 seconds with no I/O over VPN. I'm using a popular commercial VPN. Given the regular frequency it seems more like a feature than a bug, but I don't know at what level.

As the issue is specific to ResumeSession, if it's not being triggered (verifiable in the debug log), perhaps you can inject it artificially?