Fail the AWS Kinesis input on unrecoverable AWS authorization denials

[patrickmann]

Closes Graylog2/graylog-plugin-enterprise#15073

Description

When the IAM role an AWS Kinesis input uses is missing a DynamoDB permission, the Kinesis Client Library retries the denied call on a fixed schedule for as long as the input runs and only logs the failure. The result is an endless ERROR loop with a full stack trace, while the input consumes no records and still reports RUNNING.

The concrete case that prompted this is KCL 3.x lease discovery, which queries a global secondary index on the lease table. dynamodb:Query on table/<app>/index/* is a new requirement in KCL 3.x, and an index is a separate IAM resource, so a policy that grants Query on the table alone still denies it:

DynamoDBLeaseCoordinator - Failed to execute lease discovery
DynamoDbException: User: ... is not authorized to perform: dynamodb:Query on resource:
arn:aws:dynamodb:...:table/graylog-aws-plugin-<stream>/index/LeaseOwnerToLeaseKeyIndex

KCL exposes no callback for lease-management failures and no setting to bound them. It does, however, use AWS clients that we build ourselves (we bypass KinesisClientUtil to keep proxy support), so an ExecutionInterceptor on those clients observes every denial the service returns.

AWSAuthorizationFailureDetector reports a failure and stops the KCL scheduler once a call the consumer cannot work without has been denied for two minutes.

Notes:

• Only the calls the consumer cannot work without are reported, because a denial alone does not mean the input is broken. KCL absorbs several denials and keeps delivering records from the leases it already holds: a stalled single-table migration retries TransactWriteItems about twice a second, the lease-assignment Scan runs every 20s, and the DescribeTable used only for scan sizing never caches on failure. None of those has a permitted sibling call to clear its streak, so reporting on any denial stops inputs that are ingesting normally. Two operations qualify, both of which KCL retries forever while surfacing nothing: DynamoDB Query, which is lease discovery and the call denied in the reported case, and Kinesis GetRecords, which is the read path itself and how an encrypted stream's KMS failures arrive. Everything else is logged and left alone.
• Keying on the operation rather than on whether records are flowing is deliberate. An earlier revision required the consumer to have gone quiet as well, measured through completed KCL record-processing tasks. Those exist only for leases the worker already holds, which makes the signal wrong in both directions. A worker restarting with leases still stamped with its own worker id re-adopts them through DynamoDBLeaseRenewer.initialize(), which is a permitted Scan, so it keeps processing and a denied lease discovery is never reported - including on the 7.1 to 7.2 upgrade path that produced this incident. Conversely a worker holding no leases never reports progress at all, and since AWS inputs are global and KCL leadership is a DynamoDB lock with no lease affinity, a cluster with more nodes than shards would still lose one consumer at a time to the leader-only schedules above.
• Terminal codes are an allowlist, not "any 403". Expired session credentials and throttling arrive as authorization-shaped errors but recover on their own. The KMS codes an encrypted stream returns through GetRecords are included where they need an operator to act - KMSAccessDeniedException, KMSNotFoundException, KMSOptInRequired and KMSDisabledException, the last because AWS's model says the key "isn't enabled", which stays true until someone re-enables it. PrefetchRecordsPublisher swallows all of them and re-polls every 1.5s forever, so each produces this same bug. KMSInvalidStateException is excluded because its documentation does not say which key states produce it, and an allowlist has to fail safe. Matching is exact equality, which a test pins, because the allowlist carries the short code AccessDenied and that is a substring of unrelated error codes.
• A rejected credential is reported differently from a missing permission. A rotated or revoked secret key is just as terminal, but "grant it" is the wrong remedy, so the message names the credentials instead.
• Denials are tracked per operation, not per client. One DynamoDbAsyncClient carries around a dozen KCL schedules at very different rates: the denied lease-discovery Query runs every ~10s, while lease renewal writes every ~3.3s and the leader polls the migration state every ~1s. Any state shared across operations is cleared by that healthy traffic long before a threshold is reached. Note this keys on the operation name while IAM authorizes on (action, resource), so a permitted call to a different table under the same operation name does clear the streak. That does not affect the reported case: Query has exactly one call site in KCL 3.5.0 and it always targets the index.
• The threshold is a duration, not a number of attempts, because cadences on one client differ by more than an order of magnitude, so a fixed count would mean seconds for one operation and many minutes for another. The streak-reset gap is deliberately twice the reporting threshold: when the two were equal, an operation retried at just over two minutes restarted its streak on every attempt and could never be reported at all. Both watched operations are retried far faster than the threshold - lease discovery every ~10s, record fetching every 1.5s - so reaching it takes many attempts rather than two.
• CloudWatch is excluded. A cloudwatch:PutMetricData denial is non-fatal and must not fail an input that is otherwise ingesting.
• The stop is handed off to another thread. stop() blocks for up to 20s, and the callback runs on the client's shared SDK response-completion pool (sdk-async-response), or on that pool's rejection path directly on the HTTP thread, so occupying one for that long would stall unrelated completions on the same client. The thread is named per stream so concurrent failures are distinguishable in a dump.
• The terminal failure replaces an earlier transient one, and is published. InputFailureRecorder.setFailing keeps the first message once an input is already FAILING, so a preceding TaskOutcome.FAILURE would hide the denied action and resource permanently. A new setTerminallyFailing replaces it and blocks any later setRunning(). IOState.setState also had to publish its event when only the message changes: the notification, the system message and the persisted runtime state are all written by IOStateChangedEvent subscribers, so the actionable message was otherwise reaching only callers reading the input state directly. InputStateListener is the only subscriber in core or enterprise.
• Terminality lives in InputFailureRecorder, under the same lock as every state write. KinesisConsumer previously read its own flag before the failure was written, so a KCL task completing in between could revert the input to RUNNING.
• UPGRADING.md now names the DynamoDB permissions KCL 3.5 added, in its own section rather than inside the single-table migration topic, because Query on the index and UpdateTable to create it are required by every 7.2 input including new ones. It also splits the legacy -CoordinatorState and -WorkerMetricStats tables by input age: DescribeTable alone is enough only for an input created on 7.2, while an input that has not completed the migration keeps its leader lock and worker metrics there and needs the same item-level actions as the lease table. The 7.2 note deferred all of this to AWS's documentation and listed none of it, which is what let the incident happen.

Known limits, stated rather than fixed:

• A denial of an operation KCL absorbs is logged and left alone by design, so its share of the log volume continues. That is the trade for not stopping inputs that are ingesting normally, and the operator still learns of it from server.log and from AWS.
• Each node decides independently, so on a cluster the input reports FAILING per node as each node's own streak matures. Neither watched operation is leader-gated and neither needs a held lease, so unlike the earlier revision detection no longer depends on which node holds what.
• dynamodb:UpdateTable is not detectable through this seam. If it is denied the index is never created, and a KCL idempotence bug means the second initialization attempt skips index creation entirely, so the input starts and the discoverer then fails with ResourceNotFoundException rather than AccessDenied. UPGRADING.md covers it instead.
• ListShards and GetShardIterator are deliberately not watched. ListShards post-init runs at the 120s periodic-sync cadence, close enough to the threshold that a streak could mature on two samples, and a denied GetShardIterator aborts its shard consumer and already reaches the input through TaskExecutionListener, though only with the generic message.

How Tested

• New AWSAuthorizationFailureDetectorTest (40 cases): reports once a denial run spans two minutes at the real 9975ms lease-discovery cadence; a success on a different operation on the same client does not clear the run, which is the property that makes the fix work; twenty minutes of continuous denial of an operation KCL absorbs changes nothing, over Scan, DescribeTable, TransactWriteItems, UpdateItem, GetItem and PutItem; a denied Kinesis record fetch is reported; a success on the denied operation itself clears the run; the threshold boundaries either side of two minutes; an operation retried more slowly than the threshold is still reported; a gap longer than the reset window starts a new run; unwraps a denial nested in a cause chain but not one buried deeper than the limit; treats all nine terminal codes as terminal and matches them exactly; never reports self-healing codes, non-AWS exceptions, or AWS exceptions without error details; drives the real onExecutionFailure/afterExecution hooks; and fails closed when the SDK reports no operation name.
• New KinesisConsumerTest (7 cases): the headline one drives a denial through the interceptor actually installed on the DynamoDB client, so the feature cannot be left unwired; plus the terminal failure and off-thread stop with the per-stream thread name, at-most-once handling, the credential-specific message, the per-stream shutdown thread name taken from production code, one detector each on DynamoDB and Kinesis and none on CloudWatch, distinct instances, and a successful task clearing an ordinary failure.
• New InputFailureRecorderTest (9 cases): setFailing keeps the first message, setTerminallyFailing replaces it, keeps its own first message, and survives a later setRunning() or setFailing(), an event is published when only the message changes and not when nothing changes.
• New InputStateListenerTest (4 cases): the same-state failure retires the stale notification before re-raising it, for both INPUT_FAILING and INPUT_FAILED_TO_START; a real state transition retires nothing, since that would discard a notification another node may have raised; and the current state and message are persisted. This branch is what gets the actionable message onto the notification, which is the only surface a Cloud tenant can see, and nothing covered it before.
• KinesisConsumerIT now also captures setTerminallyFailing, or a terminal failure inside the IT would be swallowed until its deadline.
• Full org.graylog.integrations.aws suite plus InputFailureRecorderTest, InputStateListenerTest and IOStateTest: 116 tests green, including KinesisConsumerIT. forbiddenapis clean on both source and test scans.
• The load-bearing properties were confirmed genuinely red, not just green. Neutering the essential-operation check fails all six absorbed-operation cases and nothing else in the 40-case detector suite; neutering the same-state branch in InputStateListener fails exactly its two notification tests and leaves the other two green. Earlier in the branch, reverting the IOState message check, dropping the terminal flag from setRunning() and replacing the detector callback with a no-op each failed exactly one corresponding test.
• No integration test was added for the denial itself: KinesisConsumerIT runs against an emulator with no IAM enforcement, so it cannot produce a real 403.

Manual testing against real AWS, which is the only way to produce a genuine denial:

1. Create an AWS Kinesis input whose IAM role has the KCL 2.x-era DynamoDB permissions (CreateTable, DescribeTable, GetItem, PutItem, Scan, UpdateItem, UpdateTable) but not dynamodb:Query on arn:aws:dynamodb:<region>:<account>:table/graylog-aws-plugin-*/index/*. UpdateTable matters: without it the index is never created and you reproduce a different failure.
2. Start the input. Confirm it moves to a failed state after roughly two minutes of denials, which at the ~10s lease-discovery cadence is the 14th attempt at about 130s, that the failure message names dynamodb:Query and the LeaseOwnerToLeaseKeyIndex ARN, and that Failed to execute lease discovery stops recurring. This no longer depends on the state of the lease table, so it reproduces on a first start and on any later start alike.
3. Confirm the message also reaches the system notification and survives a restart of the page, not just the Input Diagnosis view.
4. Confirm nothing restarts the input on its own.
5. Grant the missing permission, then stop and start the input, and confirm normal ingestion.
6. Regression: with a fully permitted role, remove only cloudwatch:PutMetricData and confirm the input keeps running.
7. Regression: with a fully permitted role and an input that already holds leases, confirm ingestion is unaffected and no failure is reported.
8. Regression for the operation scope: remove dynamodb:DescribeTable on the lease table. KCL falls back to a default scan parallelism and keeps delivering records, and DescribeTable is not a watched operation, so the input must not be stopped. Worth checking on a node that holds no leases as well as on the leaseholder, since that is where a progress-based rule went wrong.
9. Point a static access key at the input and rotate the secret in AWS without updating Graylog. Confirm the failure message names the credentials rather than telling you to grant a permission.

Also validated end to end against a local emulator driving the real KCL 3.x lease discoverer, with a proxy in front of DynamoDB that denies only Query (the GSI call) and forwards everything else, mirroring the customer's policy. Query is exactly the operation the shipped rule watches, so that run still describes the shipped behaviour. Note the harness ran with failoverTimeMillis=500, which compresses KCL's coordination timings by about 21x, so it validates the window and the interleaving but not the production cadence: it recorded 254 denied queries at ~473ms apart against 1,086 permitted calls on the same client (UpdateItem 872, Scan 197, DescribeTable 10, GetItem 4, PutItem 3, and no permitted Query). That interleaving is precisely what a counter shared across operations cannot survive. At production defaults the equivalent run is 13 denials against roughly 145 permitted calls, and the input is reported at 129.7s. The reported cause was the SDK-unmarshalled AccessDeniedException naming dynamodb:Query and the LeaseOwnerToLeaseKeyIndex ARN, and stop() ran on the auth-failure shutdown thread. The harness is not committed: it needs Docker, waits out the real two-minute window, and its in-process HTTP proxy trips the jdk-non-portable forbidden-API check.

Note this behaviour also reaches the Cloud Forwarder, since KinesisTransport is bound in configureUniversalBindings() and the input is forwarder- and Cloud-compatible. The forwarder UI does render FAILING and offer Stop, so the "stop and start the input" remedy is actionable there too.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactoring (non-breaking change)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have requested a documentation update.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.