Add a mechanism to recompute asset metadata for uploaded assets

[waxlamp]

For instance, when the asset metadata schema changes, this kind of recomputation would become necessary.

dandi-cli currently performs this computation; abstracting it out into the dandi package (or making use of the functionality if it is already so abstracted) would enable us to write a serverside task which performs the same function.

Also needed: a way to trigger this recomputation for all assets in the system. This might be a special purpose button we add to the admin console views, or we could possibly work with the existing admin app to trigger an action on a selection of assets.

@yarikoptic, @satra: does this mean that dandi-cli should no longer perform this function at all, leaving it for the server to do during upload and finalization?

[satra]

this is when changes are introduced either through dandi schema, cli extraction updates or other processes. so the cli process for upload would still exist, but an update could be triggered by us or dandiset owners. this task would still use the CLI to do this, but a specific task that doesn't involve downloading all the data.

[yarikoptic]

@satra , to clarify on "recompute" wouldn't we possibly want 2 modes of operation:

• "upgrade" -- take metadata record, pass through dandischema.metadata.migrate to follow the coded in upgrade path. No access to actual data is needed. Probably could be just a straight 1 line Python call to dandischema, thus no need for dandi-cli.
• "re-extract" -- access actual data (at S3), given its path (and might need entire dandiset layout/access to other assets due to BIDS inheritance goodness!!! yikes!!! not yet though AFAIK, attn @TheChymera ) to re-extract metadata. This one is likely to be done via fsspec, may be indeed providing "access" entire dandiset hierarchy through FUSE like interface at least at the level of dandi-cli so it could "navigate" dandiset instance. ... "involved"

Then there is a question on interfacing that in dandi-archive indeed ;) but which of the above two we actually need ATM?

[satra]

thanks for separating these out.

1. re-extract will likely in all instances also upgrade.
2. in this instance we need "re-extract" all the NWB files that are part of draft datasets. this will also involve being able to do this "lightly" on streaming data (via fsspec or s3fs - note that s3fs is the python library not s3 over fuse).

[dandibot]

🚀 Issue was released in v0.3.106 🚀