Description
Task description
The Problem
Over the years, some older stuff was partially removed/not used anymore, but not fully cleaned up. Those files and leftovers in the code still live in the repository, leading to confusions (what is this? is this still used? can this be deleted?) and misleading search/grep results (e.g. for packages/extras-buildpkgs/htop or packages/extras-buildpkgs/hostapd which included changelogs and therefore lots of unrelated text).
Besides that, many blobs were added to the build repo, only some of them still remain. But even the deleted blobs still remain in the repo since Git saves all the history: The history size is huge, even though the current size of the packages/blobs folder is only 55MB. This leads to a unnecesarily bloated repository, increasing the size for everybody.
I remember one person often having to visit a local library or university to download/update their Armbian repo since the size was too large on their slow or resticted internet connection at home.
For comparison:
- Size of the whole U-Boot repository (since 2005!): ~ 228 MB
- Size of the current Armbian/build repository: ~ 636 MB
I don't believe that Armbian/build is a bigger project than U-Boot, but it is amlost triple the size in MB. Vastly reducing the repo size (TODO: calculate actual size before/after blob purge) will make contributions more inclusive overall and save time on many occasions.
The Solutions
Removing all known leftover code and moving all blobs to a separate blob repository, like already done with the Rockchip blobs in the Armbian/rkbin repo. After this is done, purge the build repository's history from all the blobs (original idea by @rpardini I believe). The goal is to have a completely blobless Arbian/build repo while blobs are only pulled from other repositories.
Task List
Leftover code:
- Remove unused
packages/extras-buildpkgs/hostapdplus its Realtek part and its traces [Cleanup] Clean leftovers inpackages/extras-buildpkgs#6830 - Remove unused
packages/extras-buildpkgs/htopand its traces --> same PR - Remove unused
packages/extras-buildpkgs/sunxi-toolsand its traces --> same PR - Remove old compiler torrents from
config/torrents/[Cleanup] Remove old torrents #6829 - Remove 'external-toolchains' code path
- [Task / Story]: Clean up old kernels/kernel patch leftovers #6821
- Check for other leftover stuff in the code (HELP NEEDED: Please let me know or add to this task list if you know of any unused stuff 😄)
Blobs:
- Create new repository for blobs (e.g. Armbian/blobs)
- Copy all existing blobs to the new blob repo
- Change all references in the code to point to the new repo for the blobs
- Remove all existing blobs from the Armbian/build repo
- Purge all blobs from the Armbian/build Git history by rewriting history (this is where the actual size reduction happens)
- Add an Actions workflow to check new PRs for blobs and if detected, kindly auto-remind them to commit the blobs to the blob repo instead of the build repo
This task list will be extended with new findings. PRs solving specific tasks will be linked.
This task/story is open for ideas and discussions! 😄
Some statistics for fun and to compare the impact of this cleanup:
| Before | After | Difference | |
|---|---|---|---|
| Lines of code | 6 458 280 | TBD | |
| # of files | 6503 | TBD | |
| Repo size | ~ 636 MB | TBD |
Commands used:
- Lines of code:
git diff --shortstat 4b825dc642cb6eb9a060e54bf8d69288fbee4904 - # of files:
git ls-files | wc -l - Repo size: Queried on GitHub