[TT-14473] - support for encrypted aws kinesis

[LLe27]

Description

• Added the support to pass a kms_key_id to enable server side encryption for Amazon Kinesis.

Related Issue

TT-14473

Motivation and Context

How This Has Been Tested

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• Make sure you are requesting to pull a topic/feature/bugfix branch (right side). If pulling from your own
  fork, don't request your master!
• Make sure you are making a pull request against the master branch (left side). Also, you should start
  your branch off our latest master.
• My change requires a change to the documentation.
  • If you've changed APIs, describe what needs to be updated in the documentation.
• I have updated the documentation accordingly.
• Modules and vendor dependencies have been updated; run go mod tidy && go mod vendor
• I have added tests to cover my changes.
• All new and existing tests passed.
• Check your code additions will not fail linting checks:
  • go fmt -s
  • go vet

[probelabs]

🔍 Code Analysis Results

1. Change Impact Analysis

What this PR Accomplishes

This pull request introduces a significant security enhancement to the Tyk Kinesis Pump by adding support for server-side encryption (SSE). Users can now specify an AWS Key Management Service (KMS) key ID in the pump's configuration. The pump will then ensure that the target Kinesis stream is encrypted using this key, protecting data at rest and helping to meet compliance requirements.

Key Technical Changes Introduced

1. New Configuration Option: The KinesisConf struct in pumps/kinesis.go is extended with a KMSKeyID field. This allows users to specify the KMS key via configuration files (kms_key_id) or environment variables (TYK_PMP_PUMPS_KINESIS_META_KMSKEYID). The README.md has been updated to document this new setting.
2. Idempotent Encryption Setup: During initialization (Init function), if a KMSKeyID is provided, the pump first checks the stream's current encryption status using the DescribeStream API call.
3. Conditional Encryption Enforcement:
   • If the stream is not encrypted, the pump calls StartStreamEncryption to enable it with the specified key.
   • If the stream is already encrypted with the correct key, the pump logs this and continues, avoiding unnecessary API calls.
   • If the stream is encrypted with a different key, the pump logs a fatal error and terminates. This "fail-secure" approach prevents running with a critical misconfiguration.
4. Secure Logging: The KMS Key ID is partially masked in log outputs to prevent accidental exposure of sensitive information.
5. Unit Testing: A new test file, pumps/kinesis_test.go, has been added to validate the new configuration options and associated logic.

Affected System Components

• Kinesis Pump (pumps/kinesis.go): The core logic is modified to include the new encryption setup during its initialization phase.
• Configuration (README.md): The configuration schema for the Kinesis pump is updated. Users deploying this pump will need to be aware of the new kms_key_id option.
• IAM Permissions: To use this feature, the IAM role or user associated with the Tyk Pump will require additional permissions: kinesis:DescribeStream and kinesis:StartStreamEncryption for the target Kinesis stream. Note: This requirement is not yet documented in the README.md and should be added.

2. Architecture Visualization

The following sequence diagram illustrates the updated initialization flow of the Kinesis pump. It highlights the new conditional logic for checking and enabling server-side encryption.

sequenceDiagram
    participant P as Tyk Pump
    participant KP as KinesisPump
    participant AWS_SDK as AWS Kinesis Client
    participant Kinesis as AWS Kinesis API

    P->>KP: Init(config)
    KP->>KP: Parse config, load StreamName & KMSKeyID

    alt KMSKeyID is provided
        KP->>AWS_SDK: DescribeStream(StreamName)
        AWS_SDK->>Kinesis: API Call: DescribeStream
        Kinesis-->>AWS_SDK: returns StreamDescription
        AWS_SDK-->>KP: returns StreamDescription

        alt Stream is NOT encrypted
            KP->>AWS_SDK: StartStreamEncryption(StreamName, KMSKeyID)
            AWS_SDK->>Kinesis: API Call: StartStreamEncryption
            Kinesis-->>AWS_SDK: API Response (Success or Error)
            
            alt API Call Succeeds
                AWS_SDK-->>KP: returns success
                KP->>KP: Log "Server-side encryption enabled"
            else API returns Error
                AWS_SDK-->>KP: returns error
                KP->>P: Log.Fatalf("Failed to enable encryption")
            end
        else Stream is encrypted with a different key
            KP->>P: Log.Fatal("Encryption enabled with wrong key")
        else Stream is already encrypted correctly
            KP->>KP: Log "Encryption already enabled"
        end
    end

    KP-->>P: Initialization Complete

Loading

---

Powered by Visor from Probelabs

Last updated: 2025-10-09T14:40:06.877Z | Triggered by: synchronize | Commit: bb13c40

[probelabs]

🔍 Code Analysis Results

Security Issues (1)

Severity	Location	Issue
🟡 Warning	README.md:1363	The documentation for the new `kms_key_id` feature does not mention the required AWS IAM permissions (`kinesis:DescribeStream` and `kinesis:StartStreamEncryption`). This omission can lead to deployment failures and may cause users to grant overly broad permissions to resolve the issue, violating the principle of least privilege. 💡 Suggestion Update the `README.md` to explicitly state the required IAM permissions. This ensures users can configure their infrastructure securely and correctly from the start. 🔧 Suggested Fix `kms_key_id` - The AWS KMS key ID used to encrypt the records in the Kinesis stream. If not provided, the records will not be encrypted (optional). **Note:** Using this option requires the `kinesis:DescribeStream` and `kinesis:StartStreamEncryption` IAM permissions for the target stream.

Performance Issues (1)

Severity	Location	Issue
🟡 Warning	pumps/kinesis.go:94-129	The new logic to enable server-side encryption introduces blocking network calls (`DescribeStream` and `StartStreamEncryption`) within the `Init` function. This will increase the application's startup time whenever a `KMSKeyID` is configured, as the pump must wait for responses from the AWS API before it can start processing data. In environments with slow network connectivity to AWS, this could lead to noticeable delays in service availability during deployments or restarts. 💡 Suggestion While the current fail-safe approach is robust, consider adding a log entry at the beginning of the check to indicate that the pump is verifying Kinesis encryption. This would make it clear to operators why startup might be delayed. For example: `p.log.Info("Verifying Kinesis stream encryption...")`. No major architectural change is recommended as the current approach is safe, but awareness of the startup delay is important.

Quality Issues (3)

Severity	Location	Issue
🟠 Error	pumps/kinesis_test.go:1	Critical functionality is completely untested. The core logic for checking and enabling server-side encryption, which involves multiple AWS API calls (`DescribeStream`, `StartStreamEncryption`) and complex conditional branching, has no unit test coverage. The new test file only validates configuration parsing and does not exercise any of the new AWS interaction logic within the `Init` function. 💡 Suggestion To address this, the `KinesisPump` should be refactored to allow for dependency injection of the Kinesis client. This would enable the use of mocks to write unit tests that cover the following scenarios: 1. Stream is already encrypted with the correct KMS key. 2. Stream is encrypted with a different KMS key (expecting a fatal error). 3. Stream is not encrypted, and the call to `StartStreamEncryption` is successful. 4. The call to `DescribeStream` fails. 5. The call to `StartStreamEncryption` fails with a `ResourceInUseException`. 6. The call to `StartStreamEncryption` fails with a generic error.
🟠 Error	system:0	Global failure condition met
🟡 Warning	pumps/kinesis.go:120-122	Potentially misleading log message on `ResourceInUseException`. When a `ResourceInUseException` is caught, the code logs `"Server-side encryption is already enabled for the Kinesis stream."`. According to AWS documentation, this exception indicates the stream is currently being updated (i.e., in an `UPDATING` state) and cannot be modified. It does not guarantee that encryption is already enabled. This log message could be misleading during troubleshooting. 💡 Suggestion Change the log message to more accurately reflect the situation, for example: `Stream is currently being updated; could not enable server-side encryption at this time. Will retry on next startup.`

✅ Style Check Passed

No style issues found – changes LGTM.

---

Powered by Visor from Probelabs

Last updated: 2025-10-09T14:40:08.023Z | Triggered by: synchronize | Commit: bb13c40