Feature/add quota project support

[Ankurdeewan]

Add quota/billing project support for Google Cloud providers

Currently, Airflow's Google Cloud providers lack support for specifying a quota/billing project when using Google services. This PR adds the ability to configure a separate project for API quota and billing purposes, which is particularly useful for organizations using shared service accounts.

Changes:

• Added quota_project_id parameter to GoogleBaseHook
• Added support for quota project configuration via connection extras
• Implemented proper validation of quota project IDs
• Added comprehensive error handling with helpful messages
• Updated UI with tooltips and descriptions
• Added documentation and examples

The quota project can be specified in two ways:

1. Via connection configuration (in extras):
   "quota_project_id": "billing-project-123"

2. Via operator parameter:
   task = BigQueryExecuteQueryOperator(
   quota_project_id="billing-project-123",
   ...
   )

The implementation uses Google Auth library's with_quota_project() method to properly set the x-goog-user-project header required by Google Cloud APIs.

Testing:

• Added unit tests for parameter and connection-based configuration
• Added validation tests
• Tested with actual Google Cloud services
• Verified backward compatibility

Documentation has been updated to reflect the new functionality.

Closes: #56311

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: [email protected]
  Slack: https://s.apache.org/airflow-slack

[MaksYermak]

@Ankurdeewan thank you for your PR!
I have several questions about this. Why can't users use the impersonation_chain parameter to achieve the same? What is the difference between impersonation_chain and quota_project_id, and could it work together? What is the reason to use the quota_project_id parameter in __init__ ? Why can't we add it as an extra parameter for google_cloud_platform Connection?

[Ankurdeewan]

Thanks for the review @MaksYermak!
Let me try to clarify a bit. The impersonation_chain and quota_project_id actually solve two different problems. Impersonation is about who is making the API call (identity), while the quota project is about which project is being billed for the request. For example, you could impersonate a service account from project A but still need the requests to be billed against project B. So they’re not overlapping, they’re complimentary and can definetly be used together.

On the placement side, I added support for quota_project_id both at the connection level (through extras) and in the operator __init__. My thinking was that connection extras work well as a default, but there are orgs where some tasks in a DAG need different quota billing, so having the param in __init__ makes that possible. In the current code, the operator param takes precedence if both are set, which is the same pattern used for project_id in other GCP operators.

That was the reasoning I went with, but of course happy to adjust if you feel it should be connection-only or if precedence should be handled differently.

[adamides]

I would think that having it at the Operator level is beneficial since different operations in different areas of a google project can be billed differently. And google's API/CLI/SDKs support specifying the quota project at the call-level as well: https://cloud.google.com/docs/quotas/set-quota-project

[Ankurdeewan]

@adamides
Yeah, that’s exactly the reasoning I had as well. Different tasks in the same DAG can need different billing contexts, so connection extras alone felt limiting. That’s why I added support both at the connection level (for defaults) and at the Operator level (for overrides), similar to how project_id works in other GCP operators.

[VladaZakharova]

@Ankurdeewan Hi there!
Thank you for the changes, I think it is really handy
I have only several concerns about the implementation itself

• the approach with writing system tests as unit tests with "_system.py" prefix is a very old approach, please check other PR to google provider for the examples how to do that
• as I can see you have created new folder for the documentation for this new parameter you have introduced. This shouldn't be in the new file, instead I would suggest moving this docs to the one that is connected to GoogleBaseHook documentation providers/Google/docs/connections/GCP.rst file.
• I think there is no need to create any new folder, like "/common" for everything connected to this. Instead, you can move also unit tests to the file that already exists: test_base_google.py file
• here -> https://github.com/apache/airflow/pull/56324/files#diff-8982cd72c0faea336f97868e0f69198e2a5b86b9e686a32db6a4b220cb7b1459R419 - this method raises exception if it is not implemented. and we have an option when user can create anonymous credentials and for this type of creds you also use this credentials. So maybe you can add some logic for this case too? With anonymous = True and quota_project_id = "blabla" rn your code will raise an exception, right?

[Ankurdeewan]

yeah thanks @VladaZakharova for the detailed feedback, i went through all the points you mentioned. moved the tests into test_base_google.py (removed the old _system style), cleaned up the extra /common folder, moved the docs into gcp.rst, and added logic for the anonymous creds case too. just pushed everything, let me know if i missed something or if there’s a cleaner way to do any of these :)