Random "PAM authentication failed" error when connecting to PostgreSQL RDS instance

[deenns]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

From time to time there is a "PAM authentication failed for user ..." error thrown when the server tries to establish a new connection to a PostgreSQL RDS instance.

According to the docs, the RDS token is valid for 15 minutes (https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.Connecting.html). We update it every 10 minutes, but it still unexpectedly expires. Usually it happens 1-3 times a day.

This bug appeared about half a year ago. It takes place only on an EC2, it cannot be reproduced locally.

The reason (almost for sure) is that the RDS auth token is issued with an EC2 token. Although, the RDS token is kind of valid for 15 minutes, it's not always so - its life time is limited by the life time of the EC2 token. That is, if the EC2 token expires in 2 minutes, and at that point of time a new RDS token is issued, the RDS token is valid only for 2 minutes rather than 15 minutes.

This bug seems to be exactly the same as the one reported here: aws/aws-sdk-net#3050

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/rds-signer@3.1052.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v24.14.0

Reproduction Steps

Although, I don't know how to reproduce it locally, I managed to check the assumption that the reason in the EC2 token. It's possible to get an EC2 token expiration time using a private method of an RDS signer instance.

import { Signer } from '@aws-sdk/rds-signer';

const signer = new Signer({
    port: port,
    hostname: host,
    username: user,
    region: REGION,
});

const serverToken = await (signer as any).credentials(); // it's a private method
const expDate = new Date(serverToken.expiration);
console.info(expDate);

Using the method above, I logged when a new RDS token is issued and when the PAM auth error occurs. These are my logs:

May 27 07:10:14: the server token will expire shortly: in 385023 ms or at Wed May 27 2026 07:16:40 GMT+0000
May 27 07:10:14: new auth token.
May 27 07:16:55: PAM auth error. Invalidate the token and try again. The error: error: PAM authentication failed for user ...

According to my logs, the PAM auth error always happens when a new RDS token is issued with an almost expired EC2 token.

I managed to create a workaround - in the case of the error the auth token is reissued and SQL query is resent. There is no error on the second attempt.

Observed Behavior

The RDS auth token expires unexpectedly, and there is a "PAM authentication failed" error.

Expected Behavior

The RDS auth token must be always valid for 15 minutes as it's said in the docs.

Possible Solution

The rds-signer package should check the expiration time of the EC2 token, and if it's less than 15 minutes, the EC2 token should be reissued first, and then an RDS auth token should be issued with a fresh EC2 token.

Additional Information/Context

No response

[kuhe]

A derived credential cannot exceed the authorization of the original, including the expiration time. This is a global authentication rule, which the documentation should've mentioned.

That said, it would be convenient for the RDS signer package to request a credential refresh automatically. This is what I've added in #8054.

---

RDS Signer's interface accepts any credential provider function. The default is the AWS credential provider chain in Node.js. The change in the PR simply calls the provider with forceRefresh: true. This is understood by the default chain, but note that if the function is custom, there may be no way to get new credentials.

The new version of the signer will, if noticing credentials are expiring in under 15 minutes, make one attempt per invocation to refresh credentials. If this does not obtain new credentials, for whatever reason, it proceeds with the old credentials as before.

[kuhe]

released in https://github.com/aws/aws-sdk-js-v3/releases/tag/v3.1058.0, could you give it a try and let us know?

[deenns]

Thank you for the fix!
I updated @aws-sdk/rds-signer to the version 3.1058.0 a week ago, and since then there have been no "PAM authentication failed" errors.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.