S3 connector can't read from public s3-compatible object stores

[sotojn]

The createS3Client() configuration requires that we have both accessKeyId and secretAccessKey keys to be valid. The issue is that this configuration requirement disables the ability to read from public s3-compatible s3-stores. This is because when we include these values the SDK does not omit signing information from the requests.

There isn't any docs discussing a specific way in the config to disable signing but I believe setting credentials: false or not providing credentials at all should fix this issue. This validation for checking for the access keys lives here:

file-assets/packages/file-asset-apis/src/s3/createS3Client.ts

Lines 135 to 149 in f6f215c

	export function createCredentialsObject(
	config: S3ClientConfig
	): S3ClientCredentials {
	if (!config.accessKeyId) {
	throw new Error('S3 accessKeyId must be defined in S3ClientConfig');
	}
	if (!config.secretAccessKey) {
	throw new Error('S3 secretAccessKey must be defined in S3ClientConfig');
	}
	const { accessKeyId, secretAccessKey } = config;
	return {
	accessKeyId,
	secretAccessKey
	};
	}

I did not find any logic of validation of the keys anywhere in the teraslice repo or spaces.

[sotojn]

After creating a patched version that allows this I propose a couple solutions on how to resolve this:

1. Add a new field to the s3 connector called no-sign-request and by default it's false. When false all current logic is the same, but when true, we allow the accessKeyId and secretAccessKey to not be set or ignore it. And append the following signer key to the s3ClientConfig:

  signer: {
    sign: async (request) => request,
  }

2. We still append the signer to the config but only when accessKeyId and secretAccessKey are not provided or are empty in the connector config.

These both achieve the same goal at the end of the day.

[sotojn]

Here is the link to an issue that gave me this workaround:

aws/aws-sdk-js-v3#2321