ENH: Upload resulting auspice.jsons to nextstrain-staging for quicker inspection

[corneliusroemer]

Context

Pathogen CI is useful to quickly check something works as expected. While artefacts can be downloaded from the Github actions tab, this requires a few steps, locating the zip folder, extract, upload to auspice.us to view.

Description

It would be nice if we uploaded all Pathogen CI builds to some path at e.g. nextstrain-staging/pathogen-ci/... and include a link in the appropriate step to make visual checks faster.

Bonus: Nextstrain bot automatically posts link to staging build like Vercel does

[tsibley]

A good way to surface the links would be in a job summary. (Example of what this looks like for our conda-base CI.)

I think a bot posting links would be a bit noisy if it happens on every new push. Better might be registering the links as a "deployment" like the Heroku preview apps do. (These in addition to the job summaries in CI run details.)

I've long thought it would be useful to have explicitly temporary unique URLs for ad-hoc dataset sharing on nextstrain.org, e.g.

https://nextstrain.org/tmp/2Q58jzs6SkqSGcs8I8YuUe

The idea would be a destination (e.g. nextstrain.org/tmp) that any authenticated user could upload to where all uploads would be immutable (write-once) and automatically purged after a few/several days.

[corneliusroemer]

When working on an ncov PR I was reminded again how useful this feature here would be. The temporary destination sounds a bit like a "Nextstrain Gist", akin to "Github Gists".

[tsibley]

Nod. GitHub's Gist follows in the footsteps of a long line of "pastebin" (or "nopaste") services.

One difference I think between what I propose above and Gist, though, is that Gists are expected to live indefinitely. I think we'd want to auto-purge after a relatively short time (days to weeks, not months).

[tsibley]

A couple weeks back, I did start tinkering with what it would take to make nextstrain.org/tmp a reality. Enough to show myself that it's doable and to identity a few minor structural/design hurdles to clear in the current codebase (unforeseen until I started). I stopped there as I was intending for only a brief diversion.

[corneliusroemer]

Autopurge after a few days-weeks is reasonable. Would it make sense to allow overwriting of the purge time?

Maybe we could put the time its valid in the URL: /tmp/2023-05-10/xa324ha lives until 2023-05-10, purged after that.

To reduce space requirement, we could just always zstd compress when uploading.

As a slight diversion: @AngieHinrichs currently uses her own temporary auspice storage for trees generated by Usher: https://next.nextstrain.org/fetch/genome-test.gi.ucsc.edu/trash/ct/subtreeAuspice1_genome_test_5ec3d_5e4250.json. These trees are publicly viewable, links often end up in pango designation issues and it's a bit of a shame that the files expire after two days or so.

Adding the expiry date to the URL would save one from clicking in vain.

[tsibley]

I don't think we'd want to make the purge time customizable. Sounds like unnecessary complexity to me.

Putting the expected "valid until" time in the URL would be doable… but the simplest approach here would have the system accepting uploads (and determining URLs) as separate from the system doing the purging, and the system doing the purging wouldn't be basing its decisions on the dates in the URLs. That means the date in the URL would be advisory only.

We could consider not auto-purging—indefinite lifetimes are obviously desirable for some use cases—but I'm not sure that aligns with our goals here (and elsewhere).