Upgrading sdk packages causes "28000: PAM authentication failed for user" error

[mrizzini]

Describe the bug

Hi there,

My team has been investigating an error occurring when connecting to our our postgres DB instance, from our EKS container, and have not been able to get to the root cause. The error we are seeing is "28000: PAM authentication failed for user {ourUser}"

We have recently started upgrading our AWSSDK packages in our EKS container, and that is when we started getting these errors. Before the errors began, we were stable, and we were on the following packages:

<PackageReference Include="AWSSDK.Core" Version="4.0.1.3" />
<PackageReference Include="AWSSDK.EventBridge" Version="4.0.5.2" />
<PackageReference Include="AWSSDK.RDS" Version="4.0.10.7" />
<PackageReference Include="AWSSDK.S3" Version="4.0.9.1" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.0.2" />
<PackageReference Include="AWSSDK.SecurityToken" Version="4.0.3" />
<PackageReference Include="AWSSDK.SQS" Version="4.0.2" />
<PackageReference Include="AWSSDK.SSO" Version="4.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.0.3.1" />
<PackageReference Include="AWSSDK.StepFunctions" Version="4.0.1" />
<PackageReference Include="AWSSDK.Transfer" Version="4.0.4.3" />

Sometime in April or May, we upgraded to the following:

<PackageReference Include="AWSSDK.Core" Version="4.0.3.31" />
<PackageReference Include="AWSSDK.EventBridge" Version="4.0.5.2" />
<PackageReference Include="AWSSDK.RDS" Version="4.0.10.7" />
<PackageReference Include="AWSSDK.S3" Version="4.0.9.1" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.0.2" />
<PackageReference Include="AWSSDK.SecurityToken" Version="4.0.5.20" />
<PackageReference Include="AWSSDK.SQS" Version="4.0.2" />
<PackageReference Include="AWSSDK.SSO" Version="4.0.2" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.0.3.1" />
<PackageReference Include="AWSSDK.StepFunctions" Version="4.0.1" />
<PackageReference Include="AWSSDK.Transfer" Version="4.0.4.3" />

Notably in the above, we upgraded to "AWSSDK.Core" Version="4.0.3.31" and "AWSSDK.SecurityToken" Version="4.0.5.20".

When we made that switch, the errors began at a frequent rate, happening multiple times an hour. It is important to note too, that there were many successful connections made to the DB during this stretch too, so not all were failing. It seems to happen randomly.

When triaging the issue, we downgraded back to the original versions we were on, and the errors stopped. Then we tried to upgrade to even newer packages, and went on the following:

<PackageReference Include="AWSSDK.Core" Version="4.0.6.1" />
<PackageReference Include="AWSSDK.EventBridge" Version="4.0.5.29" />
<PackageReference Include="AWSSDK.RDS" Version="4.0.20.3" />
<PackageReference Include="AWSSDK.S3" Version="4.0.23" />
<PackageReference Include="AWSSDK.SecretsManager" Version="4.0.4.20" />
<PackageReference Include="AWSSDK.SecurityToken" Version="4.0.6.3" />
<PackageReference Include="AWSSDK.SQS" Version="4.0.2.28" />
<PackageReference Include="AWSSDK.SSO" Version="4.0.2.27" />
<PackageReference Include="AWSSDK.SSOOIDC" Version="4.0.3.28" />
<PackageReference Include="AWSSDK.StepFunctions" Version="4.0.2.23" />
<PackageReference Include="AWSSDK.Transfer" Version="4.0.9.1" />

Notably in the above, we upgraded to "AWSSDK.Core" Version="4.0.6.1" and "AWSSDK.SecurityToken" Version="4.0.6.3".

The errors came back this time, happening again multiple times per hour, and again with many successful connections as well. Finally, we went back to the original versions again and the errors stopped.

We believe the SDK versions upgrades are causing these PAM errors, but have been unable to diagnose them ourselves. We did some investigation of the error on your github page and found some information:

This person started getting PAM errors when they upgraded from SDK 3.7.4 to 4.0.1 : #3873

Then someone from the aws-sdk team said it was because of a change they introduced: #3873 (comment)

So aws reverted it in Core 4.0.0.32 : https://github.com/aws/aws-sdk-net/blob/main/changelogs/SDK.CHANGELOG.2025.md#401030-2025-10-01-183

And presumably this fixed the problem by reverting back to the old code

And finally they reintroduced it in Core 4.0.3.0 : https://github.com/aws/aws-sdk-net/blob/main/changelogs/SDK.CHANGELOG.2025.md#401290-2025-11-07-194…

We had been on "AWSSDK.Core" Version="4.0.1.3" for awhile, and we were good, and that was the version that had the rolled back code,

When we upgraded to "AWSSDK.Core" Version="4.0.3.31", we see the errors. that was the version that had the re-released code, that presumably worked for other people but we see errors with.

Then we tried to upgrade to 4.0.6.1 and still had the problem. Currently there is one newer version, Core 4.0.6.2, which we have not tried yet.

Could you take a look into this for us?

Thank you!

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

No PAM authentication failed errors
OR
A document explaining what changed and what specific values/settings are needed

Current Behavior

Npgsql.PostgresException (0x80004005): 28000: PAM authentication failed for user "{ourUser}"

Reproduction Steps

The line of code we are executing where this error occurs is
RDSAuthTokenGenerator.GenerateAuthToken(region, hostName, portNumber, dbUserName);

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

<PackageReference Include="AWSSDK.Core" Version="4.0.6.1" /> <PackageReference Include="AWSSDK.EventBridge" Version="4.0.5.29" /> <PackageReference Include="AWSSDK.RDS" Version="4.0.20.3" /> <PackageReference Include="AWSSDK.S3" Version="4.0.23" /> <PackageReference Include="AWSSDK.SecretsManager" Version="4.0.4.20" /> <PackageReference Include="AWSSDK.SecurityToken" Version="4.0.6.3" /> <PackageReference Include="AWSSDK.SQS" Version="4.0.2.28" /> <PackageReference Include="AWSSDK.SSO" Version="4.0.2.27" /> <PackageReference Include="AWSSDK.SSOOIDC" Version="4.0.3.28" /> <PackageReference Include="AWSSDK.StepFunctions" Version="4.0.2.23" /> <PackageReference Include="AWSSDK.Transfer" Version="4.0.9.1" />

Targeted .NET Platform

originally on net8.0, but when we are upgrading we are trying to go to net10.0

Operating System and version

EKS container running on Amazon Linux 2023.10.20260216

[dscpinheiro]

I don't know if this is doable, but could you enable debug logging in your EKS cluster [1] and share the details here? You're right that unfortunately we've had a few issues related to the credentials background refresh, but they should all be resolved in latest AWSSDK.Core.

One thing that could be happening though (and we'd need the logs to confirm) is: The SDK starts a background refresh (since the credentials are still valid but nearing expiration) and GenerateAuthToken signs the token with the current session token. If that expires before the connection is actually made to RDS, the token becomes invalid (resulting in the PAM error).

[1] Detailed logging:

using Amazon;
using Amazon.Runtime;

// Add these prior to making any AWS calls.
AWSConfigs.LoggingConfig.LogTo = LoggingOptions.Console;
AWSConfigs.LoggingConfig.LogMetrics = true;

RDSAuthTokenGenerator.GenerateAuthToken(region, hostName, portNumber, dbUserName);

[mrizzini]

We'll get to work on enabling those logs @dscpinheiro, thank you. Most likely won't be able to get anything back to you until sometime next week, since this is only happening in our production environment, so we need to carefully plan when we can deploy this. I'll be in touch with the results

[mrizzini]

Hello @dscpinheiro, silly question but where would i find those logs? I added the code snippet, and I'm testing in our dev environment and haven't been able to find them. Is there something I need to set on the EKS container as well?

[dscpinheiro]

Is your application using ASP.NET? If yes, the sample I wrote was for a console app but it may be easier to use https://github.com/aws/aws-sdk-net/tree/main/extensions/src/AWSSDK.Extensions.Logging.ILoggerAdaptor instead (the logs would flow through your existing logging pipeline)

[mrizzini]

@dscpinheiro we are not using ASP.NET. But I was able to add the AWSSDK.Extensions.Logging.ILoggerAdaptor as you suggested, I'm testing in my lower environment, and with that deployed, I see logs such as:

info: Amazon.Runtime.Internal.DefaultConfigurationProvider[0]
Resolved DefaultConfigurationMode for RegionEndpoint [us-east-1] to [Standard].

i have not seen these logs before in my app, so i assume these are the types of logs i should be looking for to make sure I know I set it up correctly? Since I'm not able to reproduce the error in this environment, I'm just looking for confirmation that if I see those types of logs above, that when I deploy this to prod, I know I'll see the error messages related to my issue. Thanks for your continued help on this, we appreciate it!

[normj]

Yes those are our SDK logs coming through. Here is the code in the SDK for logging the "Resolved DefaultConfiguationMode ..." line you see.

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/DefaultConfigurationProvider.cs

Line 108 in 40e39c2

$"Resolved {nameof(DefaultConfigurationMode)} for {nameof(RegionEndpoint)} [{clientRegion?.SystemName}] to [{defaultConfigurationModeName}].");

[mrizzini]

hey @dscpinheiro, @normj , we were finally able to get this logging in Prod. and here are the logs:

Worth noting that our application has logic that will check for the token last reset time, and if it's more than 10 minutes, then we will try to grab a new one. We put this in place previously when we had issues with PAM errors in the past. As part of that logic, we are seeing this log right before our error:

Jun 22 09:44:57 AM "Not getting a new token, elapsed time is 9.209972736666666 mins. Token last set time is 06/22/2026 13:35:44 UTC."

Then we have the error logs hopefully these are readable, but ive also attached a screenshot below too:

Jun 22 09:44:58 AM REDACTED REDACTED 2026-06-22 09:44:57.838 INFO Immediate Retry is going to retry message 'd5334f8c-ebb5-4f3d-bed5-b47100e2943c' because of an exception:
Jun 22 09:44:58 AM REDACTED REDACTED Npgsql.PostgresException (0x80004005): 28000: PAM authentication failed for user "REDACTED"
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.Internal.NpgsqlConnector.ReadMessageLong(Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage)
Jun 22 09:44:58 AM REDACTED REDACTED at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder1.StateMachineBox1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.Internal.NpgsqlConnector.Authenticate(String username, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.Internal.NpgsqlConnector.g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.Internal.NpgsqlConnector.g__OpenCore|209_0(NpgsqlConnector conn, String username, SslMode sslMode, GssEncryptionMode gssEncMode, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.Internal.NpgsqlConnector.Open(NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.PoolingDataSource.OpenNewConnector(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.PoolingDataSource.g__RentAsync|33_0(NpgsqlConnection conn, NpgsqlTimeout timeout, Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.NpgsqlConnection.g__OpenAsync|42_0(Boolean async, CancellationToken cancellationToken)
Jun 22 09:44:58 AM REDACTED REDACTED at Npgsql.NpgsqlConnection.Open()
REDACTED LOGS REDACTED LOGS REDACTED LOGS REDACTED LOGS REDACTED LOGS REDACTED LOGS REDACTED LOGS
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.InvokeHandlerTerminator.Terminate(IInvokeHandlerContext context) in //src/NServiceBus.Core/Pipeline/Incoming/InvokeHandlerTerminator.cs:line 33
Jun 22 09:44:58 AM REDACTED REDACTED at REDACTED
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.LoadHandlersConnector.Invoke(IIncomingLogicalMessageContext context, Func2 stage) in /_/src/NServiceBus.Core/Pipeline/Incoming/LoadHandlersConnector.cs:line 40 Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.DeserializeMessageConnector.Invoke(IIncomingPhysicalMessageContext context, Func2 stage) in //src/NServiceBus.Core/Pipeline/Incoming/DeserializeMessageConnector.cs:line 32
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.MutateIncomingTransportMessageBehavior.InvokeIncomingTransportMessagesMutators(IIncomingPhysicalMessageContext context, Func2 next) in /_/src/NServiceBus.Core/MessageMutators/MutateTransportMessage/MutateIncomingTransportMessageBehavior.cs:line 61 Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.ProcessingStatisticsBehavior.Invoke(IIncomingPhysicalMessageContext context, Func2 next) in //src/NServiceBus.Core/Performance/Statistics/ProcessingStatisticsBehavior.cs:line 25
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.TransportReceiveToPhysicalMessageConnector.Invoke(ITransportReceiveContext context, Func2 next) in /_/src/NServiceBus.Core/Pipeline/Incoming/TransportReceiveToPhysicalMessageConnector.cs:line 35 Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.RetryAcknowledgementBehavior.Invoke(ITransportReceiveContext context, Func2 next) in //src/NServiceBus.Core/ServicePlatform/Retries/RetryAcknowledgementBehavior.cs:line 25
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.MainPipelineExecutor.Invoke(MessageContext messageContext, CancellationToken cancellationToken) in //src/NServiceBus.Core/Pipeline/MainPipelineExecutor.cs:line 49
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.MainPipelineExecutor.Invoke(MessageContext messageContext, CancellationToken cancellationToken) in //src/NServiceBus.Core/Pipeline/MainPipelineExecutor.cs:line 68
Jun 22 09:44:58 AM REDACTED REDACTED at NServiceBus.Transport.RabbitMQ.MessagePump.Process(AsyncEventingBasicConsumer consumer, BasicDeliverEventArgs message, CancellationToken messageProcessingCancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 414
Jun 22 09:44:58 AM REDACTED REDACTED Exception data:
Jun 22 09:44:58 AM REDACTED REDACTED Severity: FATAL
Jun 22 09:44:58 AM REDACTED REDACTED SqlState: 28000
Jun 22 09:44:58 AM REDACTED REDACTED MessageText: PAM authentication failed for user "REDACTED"
Jun 22 09:44:58 AM REDACTED REDACTED File: auth.c
Jun 22 09:44:58 AM REDACTED REDACTED Line: 324
Jun 22 09:44:58 AM REDACTED REDACTED Routine: auth_failed
Jun 22 09:44:58 AM REDACTED REDACTED Exception details:
Jun 22 09:44:58 AM REDACTED REDACTED Severity: FATAL
Jun 22 09:44:58 AM REDACTED REDACTED InvariantSeverity: FATAL
Jun 22 09:44:58 AM REDACTED REDACTED SqlState: 28000
Jun 22 09:44:58 AM REDACTED REDACTED MessageText: PAM authentication failed for user "REDACTED"
Jun 22 09:44:58 AM REDACTED REDACTED File: auth.c
Jun 22 09:44:58 AM REDACTED REDACTED Line: 324
Jun 22 09:44:58 AM REDACTED REDACTED Routine: auth_failed
Jun 22 09:44:58 AM REDACTED REDACTED Message type: REDACTED
Jun 22 09:44:58 AM REDACTED REDACTED Handler type: REDACTED
Jun 22 09:44:58 AM REDACTED REDACTED Handler start time: 2026-06-22 13:44:56:948142 Z
Jun 22 09:44:58 AM REDACTED REDACTED Handler failure time: 2026-06-22 13:44:57:832931 Z
Jun 22 09:44:58 AM REDACTED REDACTED Handler canceled: False
Jun 22 09:44:58 AM REDACTED REDACTED Message ID: d5334f8c-ebb5-4f3d-bed5-b47100e2943c
Jun 22 09:44:58 AM REDACTED REDACTED Pipeline canceled: False

We just have the default logging on for this application, but we can also turn on Debug logging if that would be helpful too.