Skip to content

Latest commit

 

History

History
201 lines (155 loc) · 6.88 KB

clp-config.md

File metadata and controls

201 lines (155 loc) · 6.88 KB

Configuring CLP

To use AWS S3 with CLP, follow the steps below to configure each use case you require.

:::{note} If CLP is already running, shut it down, update its configuration, and then start it again. :::

Configuration for input logs

To configure CLP to compress logs from AWS S3, update the logs_input key in <package>/etc/clp-config.yaml with the values in the code block below, replacing the fields in angle brackets (<>) with the appropriate values:

logs_input:
  type: "s3"
  aws_authentication:
    type: "<type>"
    # type-specific settings

<type> and the type-specific settings are described in the configuring AWS authentication section.

:::{note} Unlike archive and stream storage, the S3 bucket and key prefix for input logs are not specified in the configuration file. Instead, they are specified at compression time via object URLs passed to sbin/compress-from-s3.sh. :::

Configuration for archive storage

To configure CLP to store archives on AWS S3, update the archive_output.storage key in <package>/etc/clp-config.yaml with the values in the code block below, replacing the fields in angle brackets (<>) with the appropriate values:

archive_output:
  storage:
    type: "s3"
    staging_directory: "var/data/staged-archives"  # Or a path of your choosing
    s3_config:
      region_code: "<region-code>"
      bucket: "<bucket-name>"
      key_prefix: "<key-prefix>"
      aws_authentication:
        type: "<type>"
        # type-specific settings

  # archive_output's other config keys
  • staging_directory is the local filesystem directory where archives will be temporarily stored before being uploaded to AWS S3.
  • s3_config configures both the S3 bucket where archives should be stored and the authentication method for accessing it.
    • <region-code> is the AWS region code for the bucket.
    • <bucket-name> is the bucket's name.
    • <key-prefix> is the "directory" where all archives will be stored within the bucket and must end with a trailing forward slash (e.g., archives/).
    • <type> and the type-specific settings are described in the configuring AWS authentication section.

Configuration for stream storage

To configure CLP to cache stream files on AWS S3, update the stream_output.storage key in <package>/etc/clp-config.yaml with the values in the code block below, replacing the fields in angle brackets (<>) with the appropriate values:

stream_output:
  storage:
    type: "s3"
    staging_directory: "var/data/staged-streams"  # Or a path of your choosing
    s3_config:
      region_code: "<region-code>"
      bucket: "<bucket-name>"
      key_prefix: "<key-prefix>"
      aws_authentication:
        type: "<type>"
        # type-specific settings

  # stream_output's other config keys
  • staging_directory is the local filesystem directory where streams will be temporarily stored before being uploaded to AWS S3.
  • s3_config configures both the S3 bucket where streams should be stored and the authentication method for accessing it.
    • <region-code> is the AWS region code for the bucket.
    • <bucket-name> is the bucket's name.
    • <key-prefix> is the "directory" where all streams will be stored within the bucket and must end with a trailing forward slash (e.g., streams/).
    • <type> and the type-specific settings are described in the configuring AWS authentication section.

:::{note} CLP currently doesn't explicitly delete the cached streams. This limitation will be addressed in a future release. :::

Configuring AWS authentication

For each use case above, you can configure its AWS authentication method through the aws_authentication config object, which includes the authentication method type to use and any additional settings necessary for the chosen authentication type.

:::{note} The code blocks below show aws_authentication as a top-level key, but it should be nested under logs_input, archive_output.storage.s3_config, or stream_output.storage.s3_config depending on the use case. :::

Settings for each type are described below:

credentials

Settings for this type are shown below. Replace fields in angle brackets (<>) with the appropriate values:

aws_authentication:
  type: "credentials"
  credentials:
    access_key_id: "<aws-access-key-id>"
    secret_access_key: "<aws-secret-access-key>"

<aws-access-key-id> and <aws-secret-access-key> should be replaced with long-term credentials for an IAM user.

profile

Settings for this type are shown below. Replace fields in angle brackets (<>) with the appropriate values:

aws_authentication:
  type: "profile"
  profile: "<profile-name>"

<profile-name> should be the name of an existing AWS CLI profile.

In addition, the top-level config aws_config_directory must be set to the directory containing the profile configurations (typically ~/.aws):

aws_config_directory: "<aws-config-dir>"

:::{note} If profiles are not used for AWS authentication, aws_config_directory should be commented or set to null. :::

env_vars

Settings for this type are shown below.

aws_authentication:
  type: "env_vars"

The environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY should be used to specify a set of long-term IAM user credentials.

default

Settings for this type are shown below.

aws_authentication:
  type: "default"

This authentication method uses the AWS SDK's default credential provider chain, which automatically discovers credentials from multiple sources in priority order. See the official documentation for details:

Credentials are resolved in priority order. Common sources include:

  • Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  • AWS config files (~/.aws/credentials, ~/.aws/config)
  • IRSA web identity tokens (on Amazon EKS)
  • Container credentials (on Amazon ECS)
  • EC2 instance metadata / IAM roles attached to an instance