credential_process parsing failure

[guillaumekh]

Describe the bug

I use the following credential_process command inside an ~/.aws/credentials file to source AWS credentials. It uses 1password-cli & jq to extract the credentials.

This command worked reliably for years but is now broken. Likely root cause is the new ini2 parser.

[default]
credential_process = sh -c "op --account COMPANYNAME item get ITEMNAME --vault Private --fields 'AccessKeyId','SecretAccessKey' --format json | jq -c '{ Version: 1, AccessKeyId: .[] | select(.label | test(\"AccessKeyId\")).value, SecretAccessKey: .[] | select(.label | test(\"SecretAccessKey\")).value }'"

Unless I'm misreading something, the command above is compliant with documentation.

Expected Behavior

SDK authenticates successfully using credentials sourced from credential_process

Current Behavior

SDK fails to authenticate and outputs the following error message:

jq: error: AccessKeyId/0 is not defined at <top-level>, line 1:
{ Version: 1, AccessKeyId: .[] | select(.label | test(AccessKeyId)).value, SecretAccessKey: .[] | select(.label | test(SecretAccessKey)).value }                                                      
jq: error: SecretAccessKey/0 is not defined at <top-level>, line 1:
{ Version: 1, AccessKeyId: .[] | select(.label | test(AccessKeyId)).value, SecretAccessKey: .[] | select(.label | test(SecretAccessKey)).value }                                                                                                                       
jq: 2 compile errors
2024/01/15 13:07:58 operation error S3: ListObjectsV2, get identity: get credentials: failed to refresh cached credentials, process provider error: error in credential_process: exit status 3
exit status 1

Reproduction Steps

The following ~/.aws/credentials files can be used to reproduce the failure. I have removed the 1password-cli call since it's not necessary to reproduce the bug.

[default]
credential_process = sh -c "echo '[{\"label\":\"AccessKeyId\",\"value\":\"foo\"},{\"label\":\"SecretAccessKey\",\"value\":\"bar\"}]' | jq -c '{ Version: 1, AccessKeyId: .[] | select(.label | test(\"AccessKeyId\")).value, SecretAccessKey: .[] | select(.label | test(\"SecretAccessKey\")).value }'  "

This ~/.aws/credentials file is valid for the aws-cli. It works fine w/ aws-cli v2.15.10.

It fails with aws-sdk-go-v2 however.

Here is a test go file, to run w/ go run

package main

import (
	"context"
	"log"
	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	// Load the Shared AWS Configuration (~/.aws/config)
	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		log.Fatal(err)
	}

	// Create an Amazon S3 service client
	client := s3.NewFromConfig(cfg)

	// Get the first page of results for ListObjectsV2 for a bucket
	output, err := client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
		Bucket: aws.String("spectre.tests"),
	})
	if err != nil {
		log.Fatal(err)
	}

	log.Println("first page results:")
	for _, object := range output.Contents {
		log.Printf("key=%s size=%d", aws.ToString(object.Key), object.Size)
	}
}

Output is:

jq: error: AccessKeyId/0 is not defined at <top-level>, line 1:
{ Version: 1, AccessKeyId: .[] | select(.label | test(AccessKeyId)).value, SecretAccessKey: .[] | select(.label | test(SecretAccessKey)).value }                                                      
jq: error: SecretAccessKey/0 is not defined at <top-level>, line 1:
{ Version: 1, AccessKeyId: .[] | select(.label | test(AccessKeyId)).value, SecretAccessKey: .[] | select(.label | test(SecretAccessKey)).value }                                                                                                                       
jq: 2 compile errors
2024/01/15 13:07:58 operation error S3: ListObjectsV2, get identity: get credentials: failed to refresh cached credentials, process provider error: error in credential_process: exit status 3
exit status 1

Possible Solution

Either documentation or the ini parser should probably be updated.

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.24.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.4 // indirect
github.com/aws/aws-sdk-go-v2/config v1.26.3 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.16.14 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.7.2 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.10 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.10 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.10 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.10 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.48.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.18.6 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.21.6 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.26.7 // indirect

Compiler and Version used

go version go1.21.6 darwin/arm64

Operating System and version

macOS 14.2.1

[RanVaknin]

Hi @guillaumekh ,

I can confirm the behavior and that by checking an older commit this went away.

Will look into it with priority.

Thanks,
Ran~

[lucix-aws]

Will be resolved by #2461. I've removed the attempt at recreating the "escaping" routine we had from before ini2 which I evidently failed to match behaviorally.

Readings of the test value before/after ini2 and with the new patch:

1.6.0  : sh -c "echo '[{\"label\":\"AccessKeyId\",\"value\":\"foo\"},{\"label\":\"SecretAccessKey\",\"value\":\"bar\"}]' | jq -c '{ Version: 1, AccessKeyId: .[] | select(.label | test(\"AccessKeyId\")).value, SecretAccessKey: .[] | select(.label | test(\"SecretAccessKey\")).value }'  "
patched: sh -c "echo '[{\"label\":\"AccessKeyId\",\"value\":\"foo\"},{\"label\":\"SecretAccessKey\",\"value\":\"bar\"}]' | jq -c '{ Version: 1, AccessKeyId: .[] | select(.label | test(\"AccessKeyId\")).value, SecretAccessKey: .[] | select(.label | test(\"SecretAccessKey\")).value }'  "
1.7.2  : sh -c "echo '[{"label":"AccessKeyId","value":"foo"},{"label":"SecretAccessKey","value":"bar"}]' | jq -c '{ Version: 1, AccessKeyId: .[] | select(.label | test("AccessKeyId")).value, SecretAccessKey: .[] | select(.label | test("SecretAccessKey")).value }'  "