nightly internal blob stat

[satra]

given that people can upload without association to a dandiset, i think it's important for us to extract how many blobs don't have associated assets on a nightly basis and who uploaded them. this may help track people who uploaded but didn't pay attention to errors (would happen), but also any unintended uploads.

this would then serve as precursor to garbage collection.

[yarikoptic]

I wonder if we should

• add /blobs api entry point to get listing as known to the dandi-api DB. Script should also just ls the bucket and provide stat on differences (might happen if some got added/removed meanwhile) but those never should be large (if any). This would provide consistency check of the DB to the bucket
• incorporate such script into https://github.com/dandi/dandi-api-webshots/ CI run (and probably rename it into dandi-healthcheck)
  • should send out that README.md as a daily email. not sure if could be done by CI but I guess I could just set a cron job on drogon
• add there runs of https://github.com/dandi/dandi-cli/blob/master/tools/s3-gc-stats for all 3 categories on /blobs so we also know how many got removed (but still retained by versioned bucket) or (never should happen) if there is one with multiple versions

[waxlamp]

I'm not sure I understand this. Don't asset uploads have to be done per dandiset? I think I am unclear on what "without association to a dandiset" means.

[satra]

Don't asset uploads have to be done per dandiset?

currently assets are, but not blob uploads. so i can use the api to upload blobs to the archive, without associating with any dandiset.

[waxlamp]

Ok thanks for clarifying this for me.

[waxlamp]

Asset upload now requires an associated Dandiset, so I'm closing this as completed.