Sessoin manager cannot resume session on Web identity file based credentintials #117
Description
The ssm manager cannot resume when I have configured session with web-identity token files.
2025-07-30 07:35:02 DEBUG Start File Watcher On: /usr/local/sessionmanagerplugin/seelog.xml
2025-07-30 07:35:02 DEBUG Start Watcher on directory: /usr/local/sessionmanagerplugin
2025-07-30 07:35:02 DEBUG Calling Initialize Datachannel for role: publish_subscribe
2025-07-30 07:35:02 INFO Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44?role=publish_subscribe&cell-number=AAEAAXBUDbfdq/bnUaOAF85owbx9NCvF3A/sRJz8zSyeJBguAAAAAGiJyybJdMD/Cw71L4DBicVnUlfFIioc9tPjYpSrznr9JVJ8YjbrwNRo)
2025-07-30 07:35:02 DEBUG Event on file /usr/local/sessionmanagerplugin/logs : "/usr/local/sessionmanagerplugin/logs": CREATE
2025-07-30 07:35:04 INFO Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44?role=publish_subscribe&cell-number=AAEAAXBUDbfdq/bnUaOAF85owbx9NCvF3A/sRJz8zSyeJBguAAAAAGiJyybJdMD/Cw71L4DBicVnUlfFIioc9tPjYpSrznr9JVJ8YjbrwNRo)
2025-07-30 07:35:04 INFO Sending token through data channel wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44?role=publish_subscribe&cell-number=AAEAAXBUDbfdq/bnUaOAF85owbx9NCvF3A/sRJz8zSyeJBguAAAAAGiJyybJdMD/Cw71L4DBicVnUlfFIioc9tPjYpSrznr9JVJ8YjbrwNRo to acknowledge connection
2025-07-30 07:35:04 DEBUG WebsocketChannel: Send ping. Message.
2025-07-30 07:35:04 DEBUG Processing HandshakeRequest message {%!s(uint32=116) output_stream_data %!s(uint32=1) %!s(uint64=1753860902863) %!s(int64=0) %!s(uint64=1) f9cc3460-18c5-49ac-a27e-f60b97df8f7e
Ÿ³!l;0Î�CIó�Aj�ìíÎ$â¥0@ù ô$¥�8 %!s(uint32=5) %!s(uint32=160) {"AgentVersion":"3.3.2471.0","RequestedClientActions":[{"ActionType":"SessionType","ActionParameters":{"SessionType":"Port","Properties":{"portNumber":"22"}}}]}}
2025-07-30 07:35:04 DEBUG Unexpected sequence message received. Received Sequence Number: 0. Expected Sequence Number: 1
2025-07-30 07:35:04 DEBUG Resend stream data message 0 for the 0 attempt.
2025-07-30 07:35:05 DEBUG Handshake Complete. Handshake time to complete is: %!s(float64=2.024818465) seconds
2025-07-30 07:35:05 INFO Connected to instance[] on port: 22
2025-07-30 07:35:05 DEBUG Unexpected sequence message received. Received Sequence Number: 1. Expected Sequence Number: 3
2025-07-30 07:35:05 DEBUG Unexpected sequence message received. Received Sequence Number: 2. Expected Sequence Number: 4
2025-07-30 07:35:05 DEBUG Unexpected sequence message received. Received Sequence Number: 3. Expected Sequence Number: 5
2025-07-30 07:35:29 DEBUG An error happened when receiving the message. Retried times: 1, Error: read tcp 172.18.0.2:52086->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:35:29 DEBUG An error happened when receiving the message. Retried times: 2, Error: read tcp 172.18.0.2:52086->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:35:29 DEBUG An error happened when receiving the message. Retried times: 3, Error: read tcp 172.18.0.2:52086->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:35:29 DEBUG An error happened when receiving the message. Retried times: 4, Error: read tcp 172.18.0.2:52086->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:35:29 ERROR Reach the retry limit 5 for receive messages.
2025-07-30 07:35:29 ERROR Trying to reconnect the session: wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44?role=publish_subscribe&cell-number=AAEAAXBUDbfdq/bnUaOAF85owbx9NCvF3A/sRJz8zSyeJBguAAAAAGiJyybJdMD/Cw71L4DBicVnUlfFIioc9tPjYpSrznr9JVJ8YjbrwNRo with seq num: 12
2025-07-30 07:35:29 DEBUG Resume Session input parameters: {
SessionId: "botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44"
}
2025-07-30 07:35:31 ERROR Resume Session failed: WebIdentityErr: failed to retrieve credentials
caused by: InvalidAction: Could not find operation AssumeRoleWithWebIdentity for version 2011-06-15
status code: 400, request id: 086e3b52-3c99-4c49-9023-13c5130daf6c
2025-07-30 07:35:31 ERROR Failed to get token: WebIdentityErr: failed to retrieve credentials
caused by: InvalidAction: Could not find operation AssumeRoleWithWebIdentity for version 2011-06-15
status code: 400, request id: 086e3b52-3c99-4c49-9023-13c5130daf6c
2025-07-30 07:35:31 DEBUG Resume Session input parameters: {
SessionId: "botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44"
}
2025-07-30 07:35:31 ERROR Resume Session failed: WebIdentityErr: failed to retrieve credentials
caused by: InvalidAction: Could not find operation AssumeRoleWithWebIdentity for version 2011-06-15
status code: 400, request id: 10e7db86-0cd9-487e-b9f2-6329c7b1a5f9
2025-07-30 07:35:31 ERROR Failed to get token: WebIdentityErr: failed to retrieve credentials
caused by: InvalidAction: Could not find operation AssumeRoleWithWebIdentity for version 2011-06-15
status code: 400, request id: 10e7db86-0cd9-487e-b9f2-6329c7b1a5f9
2025-07-30 07:35:32 DEBUG Resume Session input parameters: {
SessionId: "botocore-session-1753860460-xpqrkpenvzxfs62j8xkoi3he44"
While When I let the pipeline be configured with AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN It works as expected.
2025-07-30 07:41:40 DEBUG Start File Watcher On: /usr/local/sessionmanagerplugin/seelog.xml
2025-07-30 07:41:40 DEBUG Start Watcher on directory: /usr/local/sessionmanagerplugin
2025-07-30 07:41:40 DEBUG Calling Initialize Datachannel for role: publish_subscribe
2025-07-30 07:41:40 INFO Opening websocket connection to: %!(EXTRA string=wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/session-name-jbsvx7hu63l2yn3ef4gapt7ug8?role=publish_subscribe&cell-number=AAEAATf2ktZ+bscwVmuWltVr3h+Zlv+uCV/gmakDyjjgOjKYAAAAAGiJzLO/MJzoAlxMGwH2ngXGo1DkjjdVyDagnC/CEq3Mg9oGhV3Df0su)
2025-07-30 07:41:40 DEBUG Event on file /usr/local/sessionmanagerplugin/logs : "/usr/local/sessionmanagerplugin/logs": CREATE
2025-07-30 07:41:42 INFO Successfully opened websocket connection to: %!(EXTRA string=wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/session-name-jbsvx7hu63l2yn3ef4gapt7ug8?role=publish_subscribe&cell-number=AAEAATf2ktZ+bscwVmuWltVr3h+Zlv+uCV/gmakDyjjgOjKYAAAAAGiJzLO/MJzoAlxMGwH2ngXGo1DkjjdVyDagnC/CEq3Mg9oGhV3Df0su)
2025-07-30 07:41:42 INFO Sending token through data channel wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/session-name-jbsvx7hu63l2yn3ef4gapt7ug8?role=publish_subscribe&cell-number=AAEAATf2ktZ+bscwVmuWltVr3h+Zlv+uCV/gmakDyjjgOjKYAAAAAGiJzLO/MJzoAlxMGwH2ngXGo1DkjjdVyDagnC/CEq3Mg9oGhV3Df0su to acknowledge connection
2025-07-30 07:41:42 DEBUG WebsocketChannel: Send ping. Message.
2025-07-30 07:41:42 DEBUG Processing HandshakeRequest message {%!s(uint32=116) output_stream_data %!s(uint32=1) %!s(uint64=1753861300111) %!s(int64=0) %!s(uint64=1) ccf305de-73b2-4ddf-8b8a-b5585ff174fe
Ÿ³!l;0Î�CIó�Aj�ìíÎ$â¥0@ù ô$¥�8 %!s(uint32=5) %!s(uint32=160) {"AgentVersion":"3.3.2471.0","RequestedClientActions":[{"ActionType":"SessionType","ActionParameters":{"SessionType":"Port","Properties":{"portNumber":"22"}}}]}}
2025-07-30 07:41:42 DEBUG Unexpected sequence message received. Received Sequence Number: 0. Expected Sequence Number: 1
2025-07-30 07:41:42 DEBUG Resend stream data message 0 for the 0 attempt.
2025-07-30 07:41:42 DEBUG Handshake Complete. Handshake time to complete is: %!s(float64=2.677490403) seconds
2025-07-30 07:41:42 WARN Stream data message with sequence number 2 is not processed as session handler is not ready.
2025-07-30 07:41:42 INFO Connected to instance[] on port: 22
2025-07-30 07:41:43 DEBUG Unexpected sequence message received. Received Sequence Number: 1. Expected Sequence Number: 2
2025-07-30 07:41:43 DEBUG Unexpected sequence message received. Received Sequence Number: 3. Expected Sequence Number: 2
2025-07-30 07:41:43 DEBUG Received Sequence Number 3 is higher than Expected Sequence Number 2, adding to IncomingMessageBuffer
2025-07-30 07:41:43 DEBUG Unexpected sequence message received. Received Sequence Number: 4. Expected Sequence Number: 2
2025-07-30 07:41:43 DEBUG Received Sequence Number 4 is higher than Expected Sequence Number 2, adding to IncomingMessageBuffer
2025-07-30 07:41:43 DEBUG Process stream data message from IncomingMessageBuffer. Sequence Number: 3
2025-07-30 07:41:43 DEBUG Process stream data message from IncomingMessageBuffer. Sequence Number: 4
2025-07-30 07:41:43 DEBUG Unexpected sequence message received. Received Sequence Number: 2. Expected Sequence Number: 5
2025-07-30 07:42:07 DEBUG An error happened when receiving the message. Retried times: 1, Error: read tcp 172.18.0.2:44532->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:42:07 DEBUG An error happened when receiving the message. Retried times: 2, Error: read tcp 172.18.0.2:44532->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:42:07 DEBUG An error happened when receiving the message. Retried times: 3, Error: read tcp 172.18.0.2:44532->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:42:07 DEBUG An error happened when receiving the message. Retried times: 4, Error: read tcp 172.18.0.2:44532->176.32.116.92:443: read: connection reset by peer, Messagetype: -1
2025-07-30 07:42:07 ERROR Reach the retry limit 5 for receive messages.
2025-07-30 07:42:07 ERROR Trying to reconnect the session: wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/session-name-jbsvx7hu63l2yn3ef4gapt7ug8?role=publish_subscribe&cell-number=AAEAATf2ktZ+bscwVmuWltVr3h+Zlv+uCV/gmakDyjjgOjKYAAAAAGiJzLO/MJzoAlxMGwH2ngXGo1DkjjdVyDagnC/CEq3Mg9oGhV3Df0su with seq num: 11
2025-07-30 07:42:07 DEBUG Resume Session input parameters: {
SessionId: "session-name-jbsvx7hu63l2yn3ef4gapt7ug8"
}
2025-07-30 07:42:08 INFO Closing datachannel with url wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/session-name-jbsvx7hu63l2yn3ef4gapt7ug8?role=publish_subscribe&cell-number=AAEAATf2ktZ+bscwVmuWltVr3h+Zlv+uCV/gmakDyjjgOjKYAAAAAGiJzLO/MJzoAlxMGwH2ngXGo1DkjjdVyDagnC/CEq3Mg9oGhV3Df0su
2025-07-30 07:42:08 INFO Closing websocket channel connection to: wss://ssmmessages.us-west-1.amazonaws.com/v1/data-channel/session-name-jbsvx7hu63l2yn3ef4gapt7ug8?role=publish_subscribe&cell-number=AAEAATf2ktZ+bscwVmuWltVr3h+Zlv+uCV/gmakDyjjgOjKYAAAAAGiJzLO/MJzoAlxMGwH2ngXGo1DkjjdVyDagnC/CEq3Mg9oGhV3Df0su