Description
The following
gcloud auth application-default login
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/application_default_credentials.json
python -c "from cloudpathlib import GSPath; GSPath('gs://private_file').download_to('.')"
fails with
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Users/joe/cloudpathlib/cloudpathlib/cloudpath.py", line 171, in __call__
cls.__init__(new_obj, cloud_path, *args, **kwargs) # type: ignore[type-var]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joe/cloudpathlib/cloudpathlib/cloudpath.py", line 230, in __init__
client = self._cloud_meta.client_class.get_default_client()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joe/cloudpathlib/cloudpathlib/client.py", line 104, in get_default_client
cls._default_client = cls()
^^^^^
File "/Users/joe/cloudpathlib/cloudpathlib/gs/gsclient.py", line 88, in __init__
self.client = StorageClient.from_service_account_json(application_credentials)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 109, in from_service_account_json
return cls.from_service_account_info(credentials_info, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/cloud/client/__init__.py", line 76, in from_service_account_info
credentials = service_account.Credentials.from_service_account_info(info)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/oauth2/service_account.py", line 240, in from_service_account_info
signer = _service_account_info.from_dict(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/joe/cloudpathlib/venv/lib/python3.11/site-packages/google/auth/_service_account_info.py", line 50, in from_dict
raise exceptions.MalformedError(
google.auth.exceptions.MalformedError: Service account info was not in the expected format, missing fields token_uri, client_email.
This is because the logic in the GSClient init assumes that if a GOOGLE_APPLICATION_CREDENTIALS file exists, it is in the format of a service account JSON key (i.e. the call to from_service_account_json).
When using workload identity federation GOOGLE_APPLICATION_CREDENTIALS is in a different format (see here).
It is possible to work around this with existing functionality, e.g. explicitly creating a google.storage.client.Client or credentials object. However it would be nice if GSClient and GSPath "just work" with workload identity federation. I've been monkeypatching GSClient and GSPath to achieve this in a few projects.
The simplest workaround is probably to replace the call to from_service_account_json with a call to google.auth.load_credentials_from_file.