how to delete orphaned link and upload file after gc?

[microyahoo]

I use harbor as registory for testing. I cleanup harbor after test finished, and run Garbage Collection, then run registry garbage-collect cmd/registry/config-harbor.yml -m to cleanup residual data in s3 bucket.
But I found that even after executing gc, there is still a lot of residual data, mainly the link files of layers. How can I clear this data?

[root@cicd-node01 ~]# rclone ls test-harbor:test-harbor/docker/registry/v2/
       ...........
     108 repositories/project-006/repository-1711047458397/_uploads/f9a625ad-5383-4c00-855b-02ecc5f0284e/hashstates/sha256/524288
       20 repositories/project-006/repository-1711047458397/_uploads/f9a625ad-5383-4c00-855b-02ecc5f0284e/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/f9b7bcf0-d0d4-4922-82d9-ea8bf70fd4c2/hashstates/sha256/0
      108 repositories/project-006/repository-1711047458397/_uploads/f9b7bcf0-d0d4-4922-82d9-ea8bf70fd4c2/hashstates/sha256/524288
       20 repositories/project-006/repository-1711047458397/_uploads/f9b7bcf0-d0d4-4922-82d9-ea8bf70fd4c2/startedat
       20 repositories/project-006/repository-1711047458397/_uploads/f9fb5ec1-c0a5-430c-8bab-cd1b3f76b08c/startedat
   524288 repositories/project-006/repository-1711047458397/_uploads/fa22079c-540d-41dc-9305-7f5c520f4767/data
      108 repositories/project-006/repository-1711047458397/_uploads/fa22079c-540d-41dc-9305-7f5c520f4767/hashstates/sha256/0
      108 repositories/project-006/repository-1711047458397/_uploads/fa22079c-540d-41dc-9305-7f5c520f4767/hashstates/sha256/524288
       20 repositories/project-006/repository-1711047458397/_uploads/fa22079c-540d-41dc-9305-7f5c520f4767/startedat
       20 repositories/project-006/repository-1711047458397/_uploads/fa7dd186-521c-4c7c-abc6-d2b186770244/startedat
       20 repositories/project-006/repository-1711047458397/_uploads/fabae351-f1be-40e9-9cb4-6ee4c6aa8de8/startedat
       47 repositories/project-006/repository-1711047458397/_uploads/fac0b10a-e59a-46bc-af3c-13d2eb5af9bf/data
      108 repositories/project-006/repository-1711047458397/_uploads/fac0b10a-e59a-46bc-af3c-13d2eb5af9bf/hashstates/sha256/0
      108 repositories/project-006/repository-1711047458397/_uploads/fac0b10a-e59a-46bc-af3c-13d2eb5af9bf/hashstates/sha256/47
       20 repositories/project-006/repository-1711047458397/_uploads/fac0b10a-e59a-46bc-af3c-13d2eb5af9bf/startedat
   524288 repositories/project-006/repository-1711047458397/_uploads/fb48d9f8-5628-4dce-ac8c-7b3eeb3c02d2/data
      108 repositories/project-006/repository-1711047458397/_uploads/fb48d9f8-5628-4dce-ac8c-7b3eeb3c02d2/hashstates/sha256/0
       20 repositories/project-006/repository-1711047458397/_uploads/fb48d9f8-5628-4dce-ac8c-7b3eeb3c02d2/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/fb4a6f60-2459-4f8f-980a-6c3197697216/hashstates/sha256/0
       20 repositories/project-006/repository-1711047458397/_uploads/fb4a6f60-2459-4f8f-980a-6c3197697216/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/fb5ad944-966f-4f30-aa65-74abbe114054/hashstates/sha256/0
       20 repositories/project-006/repository-1711047458397/_uploads/fb5ad944-966f-4f30-aa65-74abbe114054/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/fd6c2672-3400-4594-b8aa-d517ad04512d/hashstates/sha256/47
      108 repositories/project-006/repository-1711047458397/_uploads/fe6bacad-02fe-41be-a7a7-ab4b2f25cc30/hashstates/sha256/0
       20 repositories/project-006/repository-1711047458397/_uploads/fe6bacad-02fe-41be-a7a7-ab4b2f25cc30/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/fe8cc2e5-5bba-4ed6-88fa-ed39cfa86cbd/hashstates/sha256/0
      108 repositories/project-006/repository-1711047458397/_uploads/fe8cc2e5-5bba-4ed6-88fa-ed39cfa86cbd/hashstates/sha256/47
       20 repositories/project-006/repository-1711047458397/_uploads/fe8cc2e5-5bba-4ed6-88fa-ed39cfa86cbd/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/fee0d0a5-c075-4fb0-b51e-2252dd72b575/hashstates/sha256/0
      108 repositories/project-006/repository-1711047458397/_uploads/fee0d0a5-c075-4fb0-b51e-2252dd72b575/hashstates/sha256/524288
       20 repositories/project-006/repository-1711047458397/_uploads/fee0d0a5-c075-4fb0-b51e-2252dd72b575/startedat
      108 repositories/project-006/repository-1711047458397/_uploads/ffb7f2b2-6293-46d4-b82c-3b576156643f/hashstates/sha256/0
      108 repositories/project-006/repository-1711047458397/_uploads/ffb7f2b2-6293-46d4-b82c-3b576156643f/hashstates/sha256/47
       20 repositories/project-006/repository-1711047458397/_uploads/ffb7f2b2-6293-46d4-b82c-3b576156643f/startedat
       ...........
       71 repositories/test/mysql/_layers/sha256/1ae11d1b9d69faaee9664ecf939a2c7fc88ed73d135269e353163b39934f4876/link
       71 repositories/test/mysql/_layers/sha256/1ef92b20c8ec10ade9d1228ef999a5ff3afd443bea12717145c725d96c41e6d1/link
       71 repositories/test/mysql/_layers/sha256/210e58b61fa3726a89826dcbcd43a93e07c6ae9b8257082c1d15e89b878b05fd/link
       71 repositories/test/mysql/_layers/sha256/3c9be74ddc84185eafd024e29063043eee015ec174ac612c1250d3d7ebe47b44/link
       71 repositories/test/mysql/_layers/sha256/508ada0981f60fda3708effe5ca1e896bd17b0e307b951d76696b8d6cad59466/link
       71 repositories/test/mysql/_layers/sha256/558b7d69a2e576e93c7cb18ecd087a92959e912b116323c188183d5cf8ab5b17/link
       71 repositories/test/mysql/_layers/sha256/ac42afee6c9cb6ee9efac8fb6bf9505d3d3c42fd6ea8ef12007237f7946777b9/link
       71 repositories/test/mysql/_layers/sha256/b6188b3dc37c427f01aac2e8fc2b153044f2861becb8fd0f402bcabb429d71c7/link
       71 repositories/test/mysql/_layers/sha256/c7e714ea5470c150d90ee69da1f028674313e515f5b3c54fe94e3896b3aa5524/link
       71 repositories/test/mysql/_layers/sha256/d4a59c7182522928e2dc091099a5f00ac4d95cbfa080182995eb8175bc35854d/link
       71 repositories/test/mysql/_layers/sha256/d9719ad703de58410c05950e5f3b8d291ab3dce45ab456dcf1fbe3e1f232105f/link
       71 repositories/test/mysql/_layers/sha256/dcf1f586b2e2abf128a177a2e00b54fecbaca5eb23a9c6b6e8a29bffb6527f6a/link
       71 repositories/test/smartctl-exporter/_layers/sha256/00c9a2f5729c07cb6a9315344ffbe92cb0bb1fa3d388b68028084ab96223e215/link
       71 repositories/test/smartctl-exporter/_layers/sha256/05c37727fb47d7ccfec9503227a31955b8bf318d48b5fbd9a87fb9d8cdb882fd/link
       71 repositories/test/smartctl-exporter/_layers/sha256/28214c68a3d07690193c79c8e5cfec99f959f31933c276bd607234b6cbf383c8/link
       71 repositories/test/smartctl-exporter/_layers/sha256/4c1db3320c3f1482569514f92ddb80e7c975b389fc5b69c8a5f0a4d893abc519/link
       71 repositories/test/smartctl-exporter/_layers/sha256/577f92c8b679661c141ba125d89658d946290db6b293213f23c90b3b896824b7/link
       71 repositories/test/smartctl-exporter/_layers/sha256/6fb1b38f6655111c598e34b9275640ae1dd4117e264512b8385711ac821420f5/link
       71 repositories/test/smartctl-exporter/_layers/sha256/abea4a286e15c16ba543caf6864d8ce93d381ad115c7d801ce48edda01309f4e/link
       71 repositories/test/smartctl-exporter/_layers/sha256/afe31fdfe06bcf132d77f47635d25602a0fc467beea92b2ff774686a05d4c3a8/link
       71 repositories/test/smartctl-exporter/_layers/sha256/be9a1a61565ae10b48307c6fd321cdddd12d5dd9074192e0a99ec434ea3176ec/link
       71 repositories/test/smartctl-exporter/_layers/sha256/c1b18d4f3999365485ab1f1ceed7718354a7a8db839719fdec2086a6acaea3c1/link
       71 repositories/test/smartctl-exporter/_layers/sha256/d1d610c7533a6cfa7851be19557e403244aaecc0ffad89d6688aac1dc6f8aecc/link

[root@cicd-node01 ~]# rclone size test-harbor:test-harbor/docker/registry/v2/
Total objects: 207.656k (207656)
Total size: 5.438 GiB (5838625500 Byte)

[zyyw]

Currently, Harbor only removes tags and revisions of a manifest and also deletes the blob during the GC process, while the links under layers are not touched. It seems it is not defined in the oci spec either:

• https://github.com/opencontainers/distribution-spec/blob/main/spec.md#endpoints

[microyahoo]

Thanks for your quick response, I will resort to distribution.

[zyyw]

more details:

• Why doesn't deleting a repository remove all files? #3334

[microyahoo]

The two links mentioned in the issue have already invalidated, thank you for sharing this information.

[chlins]

By default, there should be a purge job executed periodically to clean the upload folder by distribution, see

harbor/make/harbor.yml.tmpl

Line 283 in dee73a4

upload_purging:

for more details.

[microyahoo]

Thanks for your sharing, I have try it, but it can only delete partial data.

[root@cicd-node01 ~]# rclone ls test-harbor:test-harbor/docker/registry/v2/ |grep _uploads | wc -l                                
484                                                                                                                                          
[root@cicd-node01 ~]# rclone ls test-harbor:test-harbor/docker/registry/v2/ |grep _layers | wc -l                             
187490                                                                                                                                       
[root@cicd-node01 ~]# rclone size test-harbor:test-harbor/docker/registry/v2/                                                   
Total objects: 188.547k (188547)                                                                                                             
Total size: 2.966 GiB (3184321730 Byte) 

[shgninc]

I have the same problem. I run GC and show 0 to delete, but my registry volume occupied and close to be full !!
Is there any solution for orphan blobs?