Allow the AWS SDK to get auth from the environment (fixes #28)

[Makeshift]

The AWS SDK can automatically gather auth from various sources in the environment, but this couldn't happen because the zod schema validation would throw if you didn't provide STORAGE_S3_ACCESS_KEY and STORAGE_S3_SECRET_KEY.

This PR makes several of the env vars optional, and deprecates them in favour of the env vars AWS SDK will infer by default.

I've tested these changes in my EKS cluster with IRSA set up for the pod to assume a role for S3 access, and it seems to work as expected.

Fixes #28, and should still be backwards compatible.

[Makeshift]

This being separate is required, because having secretAccessKey and accessKeyId keys set to undefined in a credential object will cause the AWS SDK to throw and not actually grab from the env

[axelfontaine]

This would also be very useful on EC2 for automatically using instance profiles.

Is there anything preventing this from being merged?

[DrJume]

Thank you for your PR! As we don't have experience with AWS, we appreciate you helping to include this platform.
The changes proposed improve AWS usage, but it seems, a little bit at the cost of the manual configuration.
Not everyone using S3 is on AWS, so it would be nice to combine the two approaches equally.
A compromise would be to keep the current manual documentation and validation, but to add a new Auto-Config parameter, making it easier for people using AWS. This extra parameter would add your logic, instead of replacing the current manual way.

[Makeshift]

Thank you for your PR! As we don't have experience with AWS, we appreciate you helping to include this platform. The changes proposed improve AWS usage, but it seems, a little bit at the cost of the manual configuration. Not everyone using S3 is on AWS, so it would be nice to combine the two approaches equally. A compromise would be to keep the current manual documentation and validation, but to add a new Auto-Config parameter, making it easier for people using AWS. This extra parameter would add your logic, instead of replacing the current manual way.

If you're using the AWS SDK, the AWS_ vars work for all S3 providers, irrelevant of the actual S3 backend in use. For example, if ran on my branch, this compose file sets all the needed config options via env var for AWS-SDK to communicate successfully with local minio:

services:
  postgres:
    image: postgres:15
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres

  mysql:
    image: mysql:8
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: mysql

  minio:
    image: quay.io/minio/minio:latest
    entrypoint: sh
    command: -c 'mkdir -p /data/test && /usr/bin/minio server /data'
    ports:
      - 9000:9000
    environment:
      MINIO_ROOT_USER: access_key
      MINIO_ROOT_PASSWORD: secret_key

  cache-server:
    build:
      dockerfile: Dockerfile
      context: .
    ports:
      - '3000:3000'
    environment:
      API_BASE_URL: http://localhost:3000

      STORAGE_DRIVER: s3
      STORAGE_S3_BUCKET: test
###### vvvvv
      AWS_ACCESS_KEY_ID: access_key
      AWS_SECRET_ACCESS_KEY: secret_key
      AWS_ENDPOINT_URL: http://minio:9000

They're functionally identical to passing the vars in when invoking the SDK, like you are currently, so I wouldn't say it's an AWS-centric change - merely unlocking the rest of the functionality that AWS SDK exposes rather than just a limited sliver.

The SDK already has many different auth mechanisms whether you're using AWS or not, and promoting using them rather than adding a limited custom config setup on top reduces maintenance overhead for you and improves adoption by allowing for more advanced use-cases that would be non-feasible for you to support (Such as OIDC authentication with minio).

Would some better documentation, eg examples and links to the SDK environment variables list help? (this is actually the CLI variable list, but they're very intentionally identical to the SDK variable list)

[LouisHaftmann]

I would probably completely replace our custom env vars in favor of aws defaults and only validate vars which are always required. The basic env vars should also be mentioned in the docs so people don't have to look it up elsewhere.

[DrJume]

Thank you for your PR! As we don't have experience with AWS, we appreciate you helping to include this platform. The changes proposed improve AWS usage, but it seems, a little bit at the cost of the manual configuration. Not everyone using S3 is on AWS, so it would be nice to combine the two approaches equally. A compromise would be to keep the current manual documentation and validation, but to add a new Auto-Config parameter, making it easier for people using AWS. This extra parameter would add your logic, instead of replacing the current manual way.

If you're using the AWS SDK, the AWS_ vars work for all S3 providers, irrelevant of the actual S3 backend in use. For example, if ran on my branch, this compose file sets all the needed config options via env var for AWS-SDK to communicate successfully with local minio:

services:

  postgres:

    image: postgres:15

    ports:

      - 5432:5432

    environment:

      POSTGRES_USER: postgres

      POSTGRES_PASSWORD: postgres

      POSTGRES_DB: postgres



  mysql:

    image: mysql:8

    ports:

      - 3306:3306

    environment:

      MYSQL_ROOT_PASSWORD: root

      MYSQL_DATABASE: mysql



  minio:

    image: quay.io/minio/minio:latest

    entrypoint: sh

    command: -c 'mkdir -p /data/test && /usr/bin/minio server /data'

    ports:

      - 9000:9000

    environment:

      MINIO_ROOT_USER: access_key

      MINIO_ROOT_PASSWORD: secret_key



  cache-server:

    build:

      dockerfile: Dockerfile

      context: .

    ports:

      - '3000:3000'

    environment:

      API_BASE_URL: http://localhost:3000



      STORAGE_DRIVER: s3

      STORAGE_S3_BUCKET: test

###### vvvvv

      AWS_ACCESS_KEY_ID: access_key

      AWS_SECRET_ACCESS_KEY: secret_key

      AWS_ENDPOINT_URL: http://minio:9000

They're functionally identical to passing the vars in when invoking the SDK, like you are currently, so I wouldn't say it's an AWS-centric change - merely unlocking the rest of the functionality that AWS SDK exposes rather than just a limited sliver.

The SDK already has many different auth mechanisms whether you're using AWS or not, and promoting using them rather than adding a limited custom config setup on top reduces maintenance overhead for you and improves adoption by allowing for more advanced use-cases that would be non-feasible for you to support (Such as OIDC authentication with minio).

Would some better documentation, eg examples and links to the SDK environment variables list help? (this is actually the CLI variable list, but they're very intentionally identical to the SDK variable list)

You're right. I didn't know that the AWS SDK handles so many credentials environments cases.

[axelfontaine]

I just want to add that this isn't just nice to have, it's essential. When using EC2 instance profiles, the credentials in the instance metadata are temporary and refresh regularly. Passing them to the cache server doesn't help as they will fail as soon as the first refresh happens. The AWS SDK handles all this automatically. This PR is really essential for making things work.

[LouisHaftmann]

Alright, let's just remove our custom env var validation and just let the aws sdk handle configuration.

todos:

• remove custom env validation logic
• update docs with link to aws sdk docs
• update examples in docs

[reegnz]

@Makeshift can you do what @LouisHaftmann asked? I think there's a couple of us that would be grateful if this awesome tool would work with the AWS SDK default credential chain out of the box.

[Makeshift]

@LouisHaftmann I think that should do it, if you're willing to re-review.

[LouisHaftmann]

Sorry for the delay, was on vacation for the last 3 weeks.

We just need to update the test env vars. Then we can merge. @Makeshift

.env.s3.storage

STORAGE_DRIVER=s3
STORAGE_S3_BUCKET=test
AWS_ENDPOINT_URL=http://localhost:9000
AWS_ACCESS_KEY_ID=minioadmin
AWS_SECRET_ACCESS_KEY=minioadmin

[Makeshift]

Done - Also fixed .env in the root since I'd missed that before too.