[bug] sts get caller identity does not work immediately

[hjkatz]

Describe the bug

See: #2093

Expected Behavior

I expect the call to sts.GetCallerIdentity() to succeed immediately after authenticating/receiving new sts credentials.

Current Behavior

The calls to sts.GetCallerIdentity() seem to be inconsistently failing.

Reproduction Steps

See: #2093

Possible Solution

See: #2093

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

Compiler and Version used

go version go1.21.6 linux/amd64

Operating System and version

ubuntu

[lucix-aws]

@hjkatz --

This is ultimately a fault in server-side behavior. The IAM state change associated with retrieving credentials (via AssumeRole or whatever else) does not propagate immediately after credentials are returned.

[RanVaknin]

Hi @hjkatz ,

Just to chime in as well. This is not an SDK issue and is not unique to SSO. What you are experiencing is a propagation delay that is solved with a retry. This is not an issue with the SDK itself but just a nature of a distributed system.

Thanks,
Ran~

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[hjkatz]

Hi @hjkatz ,

Just to chime in as well. This is not an SDK issue and is not unique to SSO. What you are experiencing is a propagation delay that is solved with a retry. This is not an issue with the SDK itself but just a nature of a distributed system.

Thanks,
Ran~

@RanVaknin Kindly review the linked discussion. Retrying 5 times with a gradual backoff from 500ms to 2s still reproduces the issue. (I can provide a mvp example if desired too.)

I understand that a distributed system will require time to propogate the new token. So I'll start with my goal instead: I want to verify that the SSO session via the SDK is valid. How can I test that reliably?

[hjkatz]

@RanVaknin I realize that the discussion doesn't have as clear of an example as I'm suggesting. I'll upload a clear example later today.

[hjkatz]

// Wrapper for sts.GetCallerIdentity() that supports retries
//
// Use the context to set a maximum time that retries can be performed.
// When the context is canceled then the final response will be returned.
//
// See: https://github.com/aws/aws-sdk-go-v2/discussions/2093#discussioncomment-8455830
func StsGetCallerIdentity(ctx context.Context, client *sts.Client) (result *sts.GetCallerIdentityOutput, err error) {
    if client == nil {
        return nil, errs.New("cannot call StsGetCallerIdentity with nil client!")
    }

    attempt := 0
    retries := 5
    // internal lib that implements a backoff between start -> end, without jitter (false)
    backoff := reliable.NewBackoff(100*time.Millisecond, 1*time.Second, false)

    for attempt < retries {
        attempt++

        select {
        case <-ctx.Done():
            // ran out of time, return whatever we got last
            return
        default:
            // continue below
        }

        result, err = client.GetCallerIdentity(ctx, &sts.GetCallerIdentityInput{})
        if err == nil {
            // success
            return // final values
        }

        // See: https://github.com/aws/aws-sdk-go-v2/discussions/2093
        if !strings.Contains(err.Error(), "api error InvalidClientTokenId") {
            // non-retryable error
            return // last result + error
        }

        // error, backoff and try again
        be := backoff.Wait(ctx) // Wait() == time.Sleep(backoff.NextDuration())
        if be != nil {
            // context canceled
            return // last values
        }
    }

    return // last values
}

I hope this helps reproduce what we're seeing.

I'm also hopeful we can find a way via the SDK to verify the session credentials returned by the SSO credentials provider are valid. Some thoughts to get at this information:

• sts get-caller-identity (as above)
• reading a profile from ~/.aws/config that depends on sso_session = my-session and see if it errors or not (relying on internal aws CLI's boto implementation)
• reading refresh token from the SSO provider directly? (not yet tried) see: https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/credentials/ssocreds
• Some new implementation in ssocreds that provides this ability

Happy to discuss alternatives too!

[lucix-aws]

@hjkatz On average, how long does it take for GetCallerIdentity to recognize the token?

This really seems like it should be something that sts models as @waitable. That would enable automatic code generation of a waiter API, in every SDK, that polls GetCallerIdentity until a success response is returned. In the absence of that trait, you are forced to re-solve that problem, as you've basically implemented your own waiter above.

In case you're not familiar with waiters -- https://aws.github.io/aws-sdk-go-v2/docs/making-requests/#using-waiters.

The most ubiquitous example in my mind would be dynamodb.TableExistsWaiter, since ddb table creates are technically asynchronous.

tl;dr waiting for async state changes is a problem we've solved at large, if I understand correctly it's just a question of pushing for sts to add some additional modeling to solve this specific case