Add a mechanism to "delete" published dandisets

[mvandenburgh]

This issue will require some heavy planning, including guidelines for policies/procedures around deleting published data (i.e., how dandiset owners should flag this situation for admin attention, whether they should be able to hide (and for how long) the dandiset from the public on their own authority, questions about scrubbing data permanently from the S3 bucket, etc.), and UI features to enable any/all of this.

• write a design doc

[mvandenburgh]

@satra when deleting a published dandiset, do we have to actually delete the data from our database/s3, or is simply hiding it from public view sufficient?
If just hiding it is good enough, an immediate idea i have is to

• Add a "deleted" flag to the Dandiset or Version model (depending on if we want the ability to "delete" individual versions, or just the entire dandiset)
• In the dandiset info endpoint, check if this flag is true and if so return a response indicating the dandiset has been removed from the archive
• Display a relevant message on the DLP saying the dandiset was deleted

[yarikoptic]

As data from S3 is also accessible via datalad or directly from S3 using assets dumps we have

example

$> s3cmd -c ~/.s3cfg-dandi-backup ls s3://dandiarchive/dandisets/000027/0.210831.2033/ 
2021-08-31 20:34         2142  s3://dandiarchive/dandisets/000027/0.210831.2033/assets.jsonld
2021-08-31 20:34         2210  s3://dandiarchive/dandisets/000027/0.210831.2033/assets.yaml
2021-08-31 20:34          220  s3://dandiarchive/dandisets/000027/0.210831.2033/collection.jsonld
2021-08-31 20:34         2590  s3://dandiarchive/dandisets/000027/0.210831.2033/dandiset.jsonld
2021-08-31 20:34         2540  s3://dandiarchive/dandisets/000027/0.210831.2033/dandiset.yaml

and typically the requests come for data which must not be made public. We are doomed to really delete that data.
That should be done with awareness that

• due to the deduplication of blobs across the archive, those assets might have been already used/referenced in other dandisets! We must not remove assets blindly from other dandisets. Some assets might be legit open assets or even just some rudimentary "the same content" files. So, IMHO by default only the assets which are not used in other dandisets should be removed. There must be a way to get a list of assets (ideally with some basic information e.g. size) to assess the situation. May be even by default we should just error out if some assets found to be used in other dandisets, and add an option to instruct either to skip them, or remove and which.
• due to versioning of the bucket removal of blobs from the S3 would also be more involved, and should remove DeleteMarkers etc.
• for now I think it is ok to not bother implementing this for zarrs since current immediate use-case AFAIK doesn't have zarrs. But there should be explicit check/error out if dandiset in question has zarrs.
• Depending on where we annotate about "why" previously published data was removed, we might need to make sure that such annotation somehow available within s3://dandiarchive/dandisets/ dumps

[bendichter]

Yes, as a free data storage platform, we should also prepare for the case where a user uploads illicit material and tries to use DANDI to distribute it