Skip to content

[receiver/awslambda] Add multi-format S3 log routing#47237

Draft
MichaelKatsoulis wants to merge 8 commits intoopen-telemetry:mainfrom
MichaelKatsoulis:feature/awslambdareceiver-s3-multi-format
Draft

[receiver/awslambda] Add multi-format S3 log routing#47237
MichaelKatsoulis wants to merge 8 commits intoopen-telemetry:mainfrom
MichaelKatsoulis:feature/awslambdareceiver-s3-multi-format

Conversation

@MichaelKatsoulis
Copy link
Copy Markdown
Contributor

Description

This PR adds support for routing S3 objects to different encoding extensions based on their key prefix within a
single Lambda deployment.
This is useful when a Lambda receives events from S3 buckets that
store multiple log types (e.g. VPC Flow Logs and CloudTrail in the same bucket, or across
multiple buckets with different log types).

We introduce a new encodings field in the S3 receiver config that can be used like this:

extensions:
  awslogs_encoding/vpcflow:
    format: vpcflow
    vpcflow:
      file_format: plain-text
  awslogs_encoding/cloudtrail:
    format: cloudtrail

receivers:
  awslambda:
    s3:
      encodings:
        - name: vpcflow
          encoding: awslogs_encoding/vpcflow     # decode VPC Flow Log fields into structured records
        - name: cloudtrail
          encoding: awslogs_encoding/cloudtrail  # decode CloudTrail JSON events into structured records
          path_pattern: "myorg/*/CloudTrail"     # optional: override default (AWSLogs/*/CloudTrail); omit to use the default
        - name: catchall
          path_pattern: "*"                      # forward anything else as raw bytes

The existing encoding field is unchanged. encoding and encodings are mutually
exclusive.

Link to tracking issue

Part of #46458

Testing

Unit Testing.
Pending E2E testing.

Documentation

Readme has been updated.

@MichaelKatsoulis
Copy link
Copy Markdown
Contributor Author

@axw This is the first implementation of the multi-format log support in awslambda receiver.
It focuses only in logs stored in S3.

One thing that I am not sure about is the multiFormatS3LogsHandler which lives alongside to the pre-existing S3LogsHandler.
When we add support for multi-format for logs coming from CloudWatch we could maybe create a single routedLogsHandler or something similar to be used for both cases of S3 and CW.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants