Full Support for Globus

[whwjiang]

This achieves the following milestones:

• Undoes disabling of writes and ls capabilities for Globus origins
• Requests and persists a new token for the [https://docs.globus.org/api/transfer/overview/](transfer API), which is used for LS and STAT
• Automates XRootD config passthrough
• Transform for the database, adding the transfer token column
• Adds a test for persisting Globus Transfer API token
• Fixes a bug related to going through the OAuth flow twice

To test, follow the instructions https://github.com/PelicanPlatform/pelican/pull/1336 to set up a Globus-Pelican Origin, and run your favorite Pelican object commands against the object store.

[jhiemstrawisc]

A first round of review feedback. Once we figure out the goose migration issue I was having that prevented me from starting an Origin with this code, I'll run through it more thoroughly to see what the new experience is like. Thanks for all the work that went into this, it's exciting how close this is!

[h2zh]

Hey William, could you switch to the centralized DB (default filename: pelican.sqlite; location: Server.DbLocation). Then we don't need a seperate DB for globus in the origin. We are pursuing the goal of "one universal database" in Pelican. In other words, a Pelican server can run multiple services (origin/cache/...), and they share the same database. Now would be a good time because Globus is not in prod yet, meaning no backward compatibility needed to take care of.

Tips: If globus is for Origin only, then you could update this function to generate Origin-specific schema.

[jhiemstrawisc]

This is getting pretty close. I ran into one issue setting up my local Globus Origin. I used this config to attach to my CHTC staging (at least, I think that's what I did... I'm still pretty confused about the "correct" way to create the OAuth clients via Globus's web UI):

  StorageType: "globus"
  GlobusClientIDFile: "/workspaces/pelican/globus-client-id" # Where your Globus OAuth client ID is stored
  GlobusClientSecretFile: "/workspaces/pelican/globus-client-secret" # Where your Globus OAuth client secret is stored
  Exports:
    - StoragePrefix: "/jhiemstra"     
      FederationPrefix: "/my-prefix"
      GlobusCollectionID: "6561bd4c-8dc3-42f2-8442-4907d41265a6" # The Globus collection UUID
      GlobusCollectionName: "Justin's CHTC projects" # A human-readable name for this collection
      Capabilities: ["PublicReads", "DirectReads", "Writes", "Listings"]  

From there I started my Origin and successfully activated the collection via Pelican's web UI.

Finally, I tried a pelican object ls against the Pelican namespace:

$ pelican object ls pelican://`hostname`:8445/my-prefix/            
ERROR[2025-11-06T17:05:47Z] Failure getting pelican://3f9de108225a:8445/my-prefix/: failed to perform list request: failed to stat remote path: PROPFIND /my-prefix/: 500 

It looks like this caused a segfault in the Origin's XRootD process:

ERROR[2025-11-06T17:06:05Z] xrootd.origin process failed unexpectedly: signal: segmentation fault (core dumped) 
Error: context canceled
ERROR[2025-11-06T17:06:05Z] Fatal error occurred at the start of the program. Cleanup started: context canceled 
INFO[2025-11-06T17:06:05Z] Periodic update of Globus access token is stopped. 
INFO[2025-11-06T17:06:05Z] Gracefully stopping origin TTL cache eviction... 
INFO[2025-11-06T17:06:05Z] Origin TTL cache eviction has been stopped   
INFO[2025-11-06T17:06:05Z] Director health test timeout loop has been terminated 
INFO[2025-11-06T17:06:05Z] Xrootd metrics cache eviction has been stopped 
ERROR[2025-11-06T17:06:05Z] Fatal error occurred that lead to the shutdown of the process: xrootd.origin process failed unexpectedly: signal: segmentation fault (core dumped) 

I restarted the Origin and tried a second time -- this time the list operation worked just fine and I could see my objects from /staging.

Have you run into anything like this? Unfortunately I don't see that XRootD yielded a coredump 😞

[jhiemstrawisc]

'theAuthorization' --> missing a space

[whwjiang]

@jhiemstrawisc, wrt the seg fault issue, I opened this PR for the plugin: https://github.com/PelicanPlatform/xrootd-s3-http/pull/120/files

[h2zh]

Looks good in general! Some small improvements requested for a nice codebase

[h2zh]

The database part looks good to go.

Btw, I'd recommend using git rebase main over git merge in the future PR. The former provides a cleaner state by stacking the commits you made on the top of existing main, rather than intertwining your commits with existing commits in main chronologically.