zstash globus_sdk.services.auth.errors.AuthAPIError

[BunnyVon]

Question criteria

• I added a descriptive title here.
• I searched the zstash GitHub Discussions to find a similar question and didn't find it.
• I searched the zstash documentation.

What is the deadline?

ASAP?

Describe your question

Using the latest E3SM Unified on Chrysalis and running into an error "globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')" using zstash through Globus.

Are there are any possible answers you came across?

No response

What machine were you running on?

Chrysalis

Environment

source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh

Minimal Complete Verifiable Example (MCVE)

No response

Relevant log output

=== Archiving 20240315.v2.LR.BGC-LNDATM.CONTRL ===
For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
DEBUG: Running zstash create
DEBUG: Local path : /lcrc/group/e3sm2/ac.sfeng1/E3SM_Simulations/20240315.v2.LR.BGC-LNDATM.CONTRL.ne30pg2_r05_EC30to60E2r2.chrysalis/archive
DEBUG: HPSS path  : globus://nersc/home/f/feng809/E3SMv2/20240315.v2.LR.BGC-LNDATM.CONTRL
DEBUG: Max size  : 137438953472
DEBUG: Keep local tar files  : True
DEBUG: Making sure input path exists and is a directory
INFO: Creating client of type <class 'globus_sdk.services.auth.client.native_client.NativeAppAuthClient'> for service "auth"
INFO: Finished initializing AuthLoginClient. client_id='6c1629cf-446c-49e7-af95-323c6412397f', type(authorizer)=<class 'globus_sdk.authorizers.base.NullAuthorizer'>
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140737097216448]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "5e18e65ef8183bfd4ba3bd447fb15895dd618eb2d6ca7b3b6f79d015421b39a6"
INFO: Executing token refresh without client credentials
INFO: Fetching new token from Globus Auth
INFO: request done (success)
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140737097216448]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "3e26dc4390f5f7dfd8369c1e9e3ab3d236f68a2b4a440326e5b943f1833116cf"
INFO: Executing token refresh without client credentials
INFO: Fetching new token from Globus Auth
INFO: request done (success)
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140737097216448]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "5e18e65ef8183bfd4ba3bd447fb15895dd618eb2d6ca7b3b6f79d015421b39a6"
INFO: Executing token refresh without client credentials
INFO: Fetching new token from Globus Auth
INFO: request done (success)
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140737097216448]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "3e26dc4390f5f7dfd8369c1e9e3ab3d236f68a2b4a440326e5b943f1833116cf"
INFO: Executing token refresh without client credentials
INFO: Fetching new token from Globus Auth
INFO: request done (success)
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140737097216448]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "5e18e65ef8183bfd4ba3bd447fb15895dd618eb2d6ca7b3b6f79d015421b39a6"
INFO: Setting up RefreshTokenAuthorizer with auth_client=[instance:140737097216448]
INFO: Setting up a RenewingAuthorizer. It will use an auth type of Bearer and can handle 401s.
INFO: RenewingAuthorizer will start by using access_token with hash "3e26dc4390f5f7dfd8369c1e9e3ab3d236f68a2b4a440326e5b943f1833116cf"
INFO: Creating client of type <class 'globus_sdk.services.transfer.client.TransferClient'> for service "transfer"
INFO: TransferClient.endpoint_autoactivate(15288284-7006-4041-ba1a-6b52501e49f1)
INFO: Executing token refresh without client credentials
INFO: Fetching new token from Globus Auth
INFO: request done (success)
Traceback (most recent call last):
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/bin/zstash", line 10, in <module>
    sys.exit(main())
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/zstash/main.py", line 65, in main
    create()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/zstash/create.py", line 58, in create
    globus_activate(hpss)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/zstash/globus.py", line 141, in globus_activate
    r = transfer_client.endpoint_autoactivate(ep_id, if_expires_in=600)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/services/transfer/client.py", line 364, in endpoint_autoactivate
    return self.post(
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/client.py", line 156, in post
    return self.request(
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/client.py", line 278, in request
    r = self.transport.request(
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/transport/requests.py", line 310, in request
    self._set_authz_header(authorizer, req)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/transport/requests.py", line 246, in _set_authz_header
    authz_header = authorizer.get_authorization_header()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 168, in get_authorization_header
    self.ensure_valid_token()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 162, in ensure_valid_token
    self._get_new_access_token()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/authorizers/renewing.py", line 122, in _get_new_access_token
    res = self._get_token_response()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/authorizers/refresh_token.py", line 85, in _get_token_response
    return self.auth_client.oauth2_refresh_token(self.refresh_token)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/services/auth/client/native_client.py", line 134, in oauth2_refresh_token
    return self.oauth2_token(form_data, body_params=body_params)
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/services/auth/client/base_login_client.py", line 396, in oauth2_token
    self.post(
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/client.py", line 156, in post
    return self.request(
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.3_login/lib/python3.10/site-packages/globus_sdk/client.py", line 296, in request
    raise self.error_class(r)
globus_sdk.services.auth.errors.AuthAPIError: ('POST', 'https://auth.globus.org/v2/oauth2/token', None, 400, 'Error', 'Bad Request')

real	0m0.744s
user	0m0.150s
sys	0m0.060s

Anything else we need to know?

No response

[forsyth2]

@BunnyVon These Globus authentication issues have been proving quite troublesome for developers and users alike. I checked the zstash discussions board for specifically AuthAPIError or Bad Request but didn't get other relevant results.

A few ideas to try:
-Make you're authenticated at both source and destination on the Globus website
-Delete rm ~/.globus-native-apps.cfg

"Bad Request" almost makes it sound like you're requesting data that doesn't exist.

If none of the above helps, let me know, and I'll try a more thorough debugging.

[BunnyVon]

@forsyth2 , it works now after removing ~/.globus-native-apps.cfg file. Thank you @forsyth2

[forsyth2]

Glad it was a simple fix, thanks for letting me know @BunnyVon!