Add Asset garbage collection design doc #2367
Conversation
|
@satra @yarikoptic PTAL and let me know if you have any questions or objections to this. I've also included the current counts of garbage-collectible assets and their total file sizes; it looks like we will be able to clean up a significant amount of stale data (almost 50%!). |
| - AssetBlobs: 88 (375,936,742,348 bytes / 350.12 GB) | ||
| - Uploads: 962 | ||
|
|
||
| **Note that around 50% of the existing data in DANDI (About 329 TB out of 787 TB) consists of orphaned Assets and will be garbage collected.** |
There was a problem hiding this comment.
not sure i'm following this section since assets don't have bytes associated with them, just blobs do. an orphaned asset may be attached to a blob that is not orphaned (i.e. connected to another asset)
There was a problem hiding this comment.
In this case, "Assets" refers to the bytes taken up by the blobs that would become orphaned if Asset GC was run. Whereas, "AssetBlobs" refers to blobs that are already orphaned as-is.
I could have worded that slightly better, I'll update the doc to clarify.
There was a problem hiding this comment.
but where are the numbers coming from? for example it says X of Y. i see X as the bytes associated, but where does Y come from?
Outlines a design for Asset garbage collection. Implementing this will also unblock garbage collection from being run in general, so the idea is that we will begin running the
collect_garbagescript for assets/uploads/asset blobs after this feature lands.