Skip to content

Revisit deferred URL output transfer #222

Open
@JoshKarpel

Description

@JoshKarpel

8d91538 enables a worse form of URL output files, where users must specify their remaps ahead-of-time via MapOptions.

This PR will also contain a turned-off implementation of late-binding URL output transfer via a custom plugin. It's not turned on for two reasons:

  • It doesn't work with s3 file transfer, because s3 file transfer is done by pre-signed URLs generated by the shadow. We have no easy way to do this in the plugin right now. We could use HTChirp to fetch the user's s3 credentials, but that connection may not be secure, so sending tokens over it is bad.
  • It requires Python 3 and the Python 3 HTCondor bindings to be installed on bare metal (the plugin cannot run inside the container). This is easy to guarantee in testing, but hard/impossible to guarantee on, say, the OSG.

I think its day will still come, but that day is not today...

Originally posted by @JoshKarpel in #216 (comment)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions