Support for GCP Quota Project

[sabre1041]

Google Cloud Platform includes support for a Quota Project where the ID of the Google Project can be specified for billing purposes. #1697 first introduced support for this feature where the option.WithQuotaProject(id) property is set when authenticating to Google Cloud.

However, the new integration caused a regression for users not leveraging the feature as it requires additional permissions (serviceusage.services.use) to be granted. In order to support SOPS users as a whole, this discussion intends to determine the best path forward:

Motivations

1. Support existing users of SOPS without modifying any additional permissions or actions
2. Introduce support for Project Quotas to enable users to take advantage of the capability while providing an ease of use with options/guidance for enabling the feature

Implementation of Project Quotas

A Project Quota can be specified in several ways:

1. In the request
2. Associated with an API key
3. CLI Credentials
4. Service Account
5. Workload Identity Federation

Property within the creation_rules

Include a new property within the creation_rules which allows for specifying the ID of the Quota Project that should be used for each operation.

Example:

creation_rules:
  - quota_project_id: <id>

Pros:

• Explicit in how the property is defined/provided

Cons:

• Potential challenges similar to how aws_profile is specified as documented here

Environment Variable

Specify the Quota Project via the environment variable that is leveraged via the Google clients (GOOGLE_CLOUD_QUOTA_PROJECT)

Pros:

• Aligns with existing tooling and usage

Cons:

• Precedence where multiple methods of specifying a Quota project is defined

Defined on Credentials

A Quota Project can be defined on the credential that is used to authenticate with Google Cloud. Based on the credential being used, extract the value.

Pros:

• Leverage assets that are already defined ad provided

Cons:

• Misinterpreting the value that is intended to be used

[felixfontein]

Having this as a property within the creation_rules means that the value would be part of the SOPS file metadata, similar to aws_profile. I guess it could also be defined per GCP KMS key (instead of once per file), to be more similar to aws_profile (and to avoid introducing new special cases for KMSes). Maybe a boolean flag would make more sense though (since the project ID can be extracted from the key itself if I understand it correctly?), especially if this is a global option.

Can the quota project ID always be extracted from the key, or might it be different from the key's project? (I don't use GCP KMS, so 🤷)

I guess another question is whether it is possible to need a different project ID depending on the key in use. This seems to be the case for AWS profiles for AWS KMS keys (which is one reason why the aws_profile situation isn't that easy to improve). If that's the case, we cannot simply have one quota project ID for all keys.