Kinesis Sink - AWS Security Token Handling when using ECS Task Role

[corleyscotte]

Hey Sequin team,

I discovered what appears to be a bug with the handling of expired security tokens when using the Kinesis sink.
In the logs I am getting the following error:

[warning] [21:47:50.670] Failed to deliver messages to sink: [aws_kinesis]: Error from AWS: The security token included in the request is expired (status=400) line=506 pid=<0.13227210.0> file=lib/sequin/runtime/sink_pipeline.ex domain=elixir application=sequin account_id=00418827-4530-498c-9aaa-ed754dbb7c05 consumer_id=5ac7a484-9599-4063-b17b-b4bdee53243d

The UI also displays the sink as being unhealthy. Initially I thought this may be a permissions issue with the ECS Task Role, however I don't believe that's the case.

Here are the permissions that the role I am using has:

Policies:
        - PolicyName: KinesisPublisherTaskPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - kinesis:DescribeStream
                  - kinesis:DescribeStreamSummary
                  - kinesis:ListShards
                  - kinesis:PutRecord
                  - kinesis:PutRecords
                Resource: '*'
        - PolicyName: SequinTaskPolicy
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - ssmmessages:CreateControlChannel
                  - ssmmessages:CreateDataChannel
                  - ssmmessages:OpenControlChannel
                  - ssmmessages:OpenDataChannel
                Resource: '*'

The issue can typically be resolved by going through and hitting the refresh icon on the healthcheck for a sink.
I also typically have to go through and hit "redeliver all" to the messages in queue after the healthcheck has confirmed the updated credentials within the container are valid.

Please let me know if you need any additional information, logs, testing, etc.

Thanks!

[dosubot]

Hi @corleyscotte! I'm Dosu and I’m helping the sequin team.

Based on what I found, Sequin’s Kinesis sink delegates message delivery and error handling to its pipeline modules, but there’s no explicit logic for automatically refreshing AWS credentials when they expire. The error you’re seeing (“The security token included in the request is expired”) is logged as a failed delivery, and the sink is marked unhealthy until you manually refresh the healthcheck, which likely triggers a reinitialization and reloads the credentials within the container. This manual step appears necessary because the pipeline doesn’t automatically recover from expired tokens or refresh credentials on its own [source].

If you’re running on ECS with a Task Role, the AWS SDK should be able to refresh credentials automatically, but Sequin may not be reinitializing the client or pipeline when a token expires. For now, using the healthcheck refresh is the workaround. If you want more robust handling, you might need to look into whether the Kinesis pipeline can be extended to detect expired tokens and trigger a credential reload or pipeline restart automatically.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[corleyscotte]

@dosu - Is there an undocumented API endpoint that I can hit to trigger a healthcheck or credential reload? I could use that as a temporary workaround if so.

[dosubot]

There isn’t an API endpoint—documented or undocumented—that you can hit to trigger a healthcheck or credential reload for sinks like Kinesis in Sequin. These actions are only available through the authenticated browser UI and are handled via LiveView events, not HTTP API calls or REST endpoints (source).

If you need to automate this as a workaround, you’d have to simulate browser actions (for example, using browser automation tools), since there’s no programmatic way to trigger a sink healthcheck or credential reload externally. Let me know if you want details on possible operational workarounds or have other questions about sink health or AWS credential management.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[vagelim]

I noticed the same issue in the SQS implementation. I temporarily patched it for myself with vagelim#1 but it is pretty heavy-handed and would be better if it only refreshed when expiry was near.

[corleyscotte]

Update: was able to get a screenshot of the error I reported

[md5]

The issue here seems to be that there is a bit of a disconnect between how aws-elixir and aws_credentials work.

The aws_credentials README clearly states that applications should not cache the result of calling get_credentials:

It is best practice to not cache these credentials inside of your own application. Always use the aws_credentials library to retrieve them - this way you will always get "fresh" credentials from the internal state of the application.

However, AWS.Client.create basically requires applications to do just that. Ideally, AWS.Client would instead have a way to call aws_credentials:get_credentials() whenever it needs credentials since aws_credentials already handles checking for expiration and refreshing credentials as needed. I'm not familiar enough with Elixir or Erlang to suggest what the idiomatic way to do that would be, but it does seem like it's idiomatic to work this way since aws_credentials is implemented with the GenServer behavior.

Working around this in Sequin does not seem like the best way to fix this issue.