This guide explains how to configure and use CLP with AWS S3.
:::{note}
Currently, only clp-json supports object storage. Support for clp-text will be
added in a future release.
:::
:::{note} This guide focuses on AWS S3. For S3-compatible storage services (e.g., MinIO, Ceph), see the S3-compatible storage guide. :::
:::{tip} If you're using object storage because the host(s) on which you're running CLP are ephemeral, consider also using external databases for metadata storage (to ensure data persistence in case of host replacements). :::
Table 1 shows the supported uses of AWS S3 in CLP.
(table-1)= ::::{card}
:::{table} :align: left
| Use case | Supported |
|---|---|
| Compress logs from AWS S3 | |
| Store archives on AWS S3 | |
| Cache stream files on AWS S3 |
:::
+++ Table 1: The supported uses of AWS S3 in CLP. ::::
:::{note} You can choose to use AWS S3 for any combination of the three use cases above (e.g., compress logs from AWS S3 and cache the stream files on AWS S3, but store archives on the local filesystem). :::
-
This guide assumes you're able to configure, start, stop, and use CLP as described in the clp-json quick-start guide.
-
Depending on which use cases you require:
- An S3 bucket and key prefix containing the logs you wish to compress.
- An S3 bucket and key prefix where you wish to store compressed archives.
- An S3 bucket and key prefix where you wish to cache stream files.
-
A supported AWS authentication method configured with the necessary permissions to access the S3 buckets and prefixes mentioned above.
:::{note} You may use a single authentication method for all the use cases above, or a separate one for each. :::
clp-json currently supports the AWS authentication methods described below.
:::{caution} Short-term STS credentials (which include a Session Token) are not supported directly. Instead, use named profiles (with IAM Identity Center authentication or IAM role assumption) which provide the required permissions and don't require specifying credentials directly. :::
clp-json can authenticate using long-term credentials for an IAM user.
- To create a user, follow this guide.
- You don't need to assign any groups or policies to the user at this stage since we will attach policies in later steps, depending on which object storage use cases you require.
- To generate the credentials, follow this guide.
- Choose the "Other" use case to generate long-term credentials.
clp-json can authenticate using AWS CLI named profiles. Named profiles can themselves make use of a variety of AWS authentication mechanisms, including:
- IAM Identity Center authentication
- Assuming an IAM role
- Long-term IAM user credentials
Follow this guide for more information on configuring profiles with the AWS CLI.
:::{note}
Profile configurations are stored in your AWS config directory (typically ~/.aws).
:::
clp-json can authenticate using long-term IAM user credentials
specified through the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
clp-json can authenticate using the AWS SDK's default credential provider chain, which automatically discovers credentials from multiple sources in priority order. This includes:
- IRSA (IAM Roles for Service Accounts) web identity tokens (on Amazon EKS)
- Container credentials (on Amazon ECS)
- EC2 instance IAM roles (follow this guide to attach a role)
The subsections below explain how to configure your AWS S3 bucket and CLP for each use case:
::::{grid} 1 1 1 1 :gutter: 2
:::{grid-item-card} :link: aws-s3-config Configuring AWS S3 ^^^ Configuring your AWS S3 bucket and IAM permissions for each use case. :::
:::{grid-item-card} :link: clp-config Configuring CLP ^^^ Configuring CLP to use AWS S3 for each use case. ::: ::::
The subsection below explains how to use CLP with AWS S3 for each use case:
::::{grid} 1 1 1 1 :gutter: 2
:::{grid-item-card} :link: using-clp-with-aws-s3 Using CLP with AWS S3 ^^^ Using CLP to compress, search, and view log files from AWS S3. ::: ::::
:::{toctree} :hidden:
aws-s3-config clp-config eks-irsa-setup using-clp-with-aws-s3 :::