Anonymous auth to GCS

[TomAugspurger]

I'm trying to make an anonymous request to a Google Cloud Storage bucket. I'm not too familiar with GCS, but I did find google.auth.credentials.AnonymousCredentials: https://googleapis.dev/python/google-auth/1.7.0/reference/google.auth.credentials.html#google.auth.credentials.AnonymousCredentials. However, the following raises an error:

import obstore
import obstore.auth.google
import google.auth.credentials


def main():
    g_credentials = google.auth.credentials.AnonymousCredentials()
    credential_provider = obstore.auth.google.GoogleCredentialProvider(g_credentials)
    store = obstore.store.GCSStore(
        bucket="weatherbench2",
        prefix="datasets/era5/1959-2023_01_10-full_37-1h-0p25deg-chunk-1.zarr",
        credential_provider=credential_provider)

    xs = store.list_with_delimiter()
    print(next(xs))


if __name__ == "__main__":
    main()

with

Traceback (most recent call last):
  File "torch_dataloader.py", line 19, in <module>
    main()
  File "torch_dataloader.py", line 14, in main
    xs = store.list_with_delimiter()
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.12/site-packages/obstore/store.py", line 364, in list_with_delimiter
    return obs.list_with_delimiter(
           ^^^^^^^^^^^^^^^^^^^^^^^^
obstore.exceptions.UnauthenticatedError: The operation lacked valid authentication credentials for path External GCP credential provider: InvalidOperation: Anonymous credentials cannot be refreshed.

Debug source:
Unauthenticated {
    path: "External GCP credential provider",
    source: PyErr {
        type: <class 'google.auth.exceptions.InvalidOperation'>,
        value: InvalidOperation('Anonymous credentials cannot be refreshed.'),
        traceback: Some(
            "Traceback (most recent call last):\n  File \".venv/lib/python3.12/site-packages/obstore/auth/google.py\", line 76, in __call__\n    self.credentials.refresh(self.request)\n  File \.venv/lib/python3.12/site-packages/google/auth/credentials.py\", line 322, in refresh\n    raise exceptions.InvalidOperation(\"Anonymous credentials cannot be refreshed.\")\n",
        ),
    },
}

[kylebarron]

Well one option is to install from this branch: #404. Once the upstream PR apache/arrow-rs-object-store#322 is merged, we can merge it into main here.

To fix this issue at hand, we probably want to add a check

if isinstance(self.credentials, AnonymousCredentials):
    return {"bearer": ""}

But I'm not sure whether that will actually work in the current behavior of object_store, because it will send the bearer token with every request (except for get) even if the bearer token is "". We really need apache/arrow-rs-object-store#322 to be merged here.

[kylebarron]

You can install from that branch with

pip install git+https://github.com/developmentseed/obstore@5b7f5b9304edcaed31fccbed088fe89835dd0821#subdirectory=obstore

as long as you have Rust installed on your system.

[kylebarron]

In particular, with #404 , you can construct the store with

GCSStore("bucket", skip_signature=True)

without needing to use a credential provider

[kylebarron]

Yeeah I played around with this a little more and I don't think this will be fixed until #404 gets merged (I'll merge it as soon as the upstream object_store PR gets merged)

[TomAugspurger]

Thanks for testing that out! I'll keep an eye on those two.