Make `google-cloud-auth` work in virtualized environments

[coryan]

Objective

Run google-cloud-rust in environments where std::env::var, std::fs::File, or reqwest::Client may not work. The first step is running google-cloud-auth.

Overview

Add dyn-compatible traits to represent std::env::var, std::fs::File, and reqwest::Client. Use these traits in the implementation of google-cloud-auth. Define some configuration flags to google_cloud_auth::credentials::Builder to replace the default traits with custom ones.

Details

The traits for std::env::var and std::fs::File are more or less straightforward.

Abstracting the HTTP client requires more machinery. We need to use a dyn-compatible trait for the asynchronous methods, so the usual abstractions around async_trait apply.

Alternatives Considered

Use traits that are not dyn-compatible.

This would require breaking changes and modify every API. Not really a good option, just documenting it here.

[coryan]

I have questions:

1. What happens after google-cloud-auth? How does anybody use google-cloud-storage or google-cloud-secretmanager-v1 if the standard reqwest::Client does not work?

2. Why do we need to abstract away http::StatusCode and http::HeaderName and friends?

[codeshaunted]

Hi, thanks for making this issue!

1. What happens after `google-cloud-auth`? How does anybody use `google-cloud-storage` or `google-cloud-secretmanager-v1` if the standard `reqwest::Client` does not work?

I think you could have methods like with_http_client_provider, with_env_provider, etc on ClientBuilder that store dyn impls of provider traits from a shared crate to be used on that client. AWS uses a very similar pattern for the HTTP client and similar methods for env and fs just got merged upstream.

2. Why do we need to abstract away `http::StatusCode` and `http::HeaderName` and friends?

Especially since gax uses types from http I don't think this is necessary.

[coryan]

I think you could have methods like with_http_client_provider, with_env_provider, etc on ClientBuilder that store
dyn impls of provider traits from a shared crate to be used on that client.

Hmmm... that could work for services that use pure HTTP+JSON, but we also have services that use gRPC, and some use (or will use) a combination of HTTP+JSON and gRPC. Maybe about 10% of the client libraries will require hybrid clients, too many too ignore.

Currently we use Tonic for gRPC, but that will change in the future. I am not sure we know enough about the future implementation to define a dyn interface.

And we also need to think about how to represent a streaming body for PUT, and streaming GET responses. We use both. We may need to wrap the streaming functions in some kind of dyn-compatible wrapper.

Before we embark on the changes to google-cloud-auth I would like to make sure we can complete the journey. It would be sad to have only auth work and the rest of the services stuck.

2. Why do we need to abstract away `http::StatusCode` and `http::HeaderName` and friends?

Especially since gax uses types from http I don't think this is necessary.

Okay, so I need to think if exposing http in the public API was one of my most terrible ideas and we should start reducing its impact or if if was just regular terrible and we should continue.... 🤔 I believe it was just regular terrible.

[codeshaunted]

Hmmm... that could work for services that use pure HTTP+JSON, but we also have services that use gRPC, and some use (or will use) a combination of HTTP+JSON and gRPC. Maybe about 10% of the client libraries will require hybrid clients, too many too ignore.

Forgive me if this is naive as I'm not too experienced with gRPC, but isn't gRPC backed by HTTP2? Can we thread the same implementation we provide for HTTP+JSON implementation through to a generic gRPC implementation (requiring it supports HTTP2, which seems reasonable in 2026). A very cursory look at Tonic looks like it may support this sort of dependency injection.

Okay, so I need to think if exposing http in the public API was one of my most terrible ideas and we should start reducing its impact or if if was just regular terrible and we should continue.... 🤔 I believe it was just regular terrible.

With ~50M downloads per month I think it can basically be considered standard library; seems like just regular terrible to me :)

[coryan]

Hmmm... that could work for services that use pure HTTP+JSON, but we also have services that use gRPC, and some use (or will use) a combination of HTTP+JSON and gRPC. Maybe about 10% of the client libraries will require hybrid clients, too many too ignore.

Forgive me if this is naive as I'm not too experienced with gRPC, but isn't gRPC backed by HTTP2? Can we thread the same implementation we provide for HTTP+JSON implementation through to a generic gRPC implementation (requiring it supports HTTP2, which seems reasonable in 2026). A very cursory look at Tonic looks like it may support this sort of dependency injection.

You are probably quite right about Tonic. But as I mentioned, we are planning to replace the gRPC implementation. The next implementation is still under construction under:

https://github.com/hyperium/tonic/tree/master/grpc

I need to ask the folks at Google that are working on it.

[codeshaunted]

Could we start with env/fs without worrying about HTTP/gRPC for now since those are less likely to need to be mocked by end-users? At least right now, env/fs are the main issue for us; we can easily abuse cargo to patch a custom implementation of reqwest in for our needs, but patching in a custom std is much more involved and fragile.

[coryan]

Could we start with env/fs without worrying about HTTP/gRPC for now since those are less likely to need to be mocked by end-users?

Yes! Good idea. Let's start with those.

[codeshaunted]

Could we start with env/fs without worrying about HTTP/gRPC for now since those are less likely to need to be mocked by end-users?

Yes! Good idea. Let's start with those.

Ok! I can start working on a PR for this, I'm assuming the traits should live in google-cloud-gax? Even just with env/fs it would still likely be a rather large PR in terms of number of files changed in google-cloud-auth (lots of single fields added to builders, std::env::var changed to env.var, etc), so let me know if/how you'd like me to split it further.

[suzmue]

Ok! I can start working on a PR for this, I'm assuming the traits should live in google-cloud-gax?

Thanks for taking this on! A new module within google-cloud-gax sounds like the right home for these traits.

Even just with env/fs it would still likely be a rather large PR in terms of number of files changed in google-cloud-auth (lots of single fields added to builders, std::env::var changed to env.var, etc), so let me know if/how you'd like me to split it further.

Let's aim for smaller, incremental PRs where possible to keep the reviews manageable. I'd suggest starting strictly with std::env::var and choosing a single google-cloud-auth use case for the first PR. That way we can align on the trait design and what the google-cloud-auth wiring should look before adding it across the entire library.

We can then handle the fs trait and broader integration in follow-up PRs!