|
1 | | -# .stone format |
| 1 | +# How AerynOS delivers software to OS installs |
2 | 2 |
|
3 | | -## manifest.*.bin |
| 3 | +## Software package metadata: manifest.*.bin |
4 | 4 |
|
5 | | -The `manifest.${ARCH}.bin` files are the only files the AerynOS tooling reads. The `manifest.${ARCH}.jsonc` files are only there for git diff purposes and human-readable insight. They are completely ignored by the tooling. |
| 5 | +The `manifest.${ARCH}.bin` files contain all the metadata needed by the AerynOS tooling. For context, the `manifest.${ARCH}.jsonc` files are only there for git diff purposes and human-readable insight. They are completely ignored by the tooling. |
6 | 6 |
|
7 | | -**Ikey Doherty** |
8 | | -> our manifest.*.bin format is just a .stone in disguise |
9 | | -> containing only a metadata payload with special fields |
10 | | -> and the stone archive type flag is set to buildmanifest |
11 | | -> sneaksy |
12 | | -> (in fact, our repo format is also just a set of meta payloads in a stone file..) |
13 | | -> but its also strongly typed, fixed headers, version agnostic header unpack and compressed with zstd with CRC checks |
14 | | -> soo. a little less weak than sounding |
15 | | -> crc is actually xxh64 iirc |
| 7 | + **Ikey Doherty** |
| 8 | + > our manifest.*.bin format is just a .stone in disguise |
| 9 | + > containing only a metadata payload with special fields |
| 10 | + > and the stone archive type flag is set to buildmanifest |
| 11 | + > sneaksy |
| 12 | + > (in fact, our repo format is also just a set of meta payloads in a stone file..) |
| 13 | + > but its also strongly typed, fixed headers, version agnostic header unpack and compressed with zstd with CRC checks |
| 14 | + > soo. a little less weak than sounding |
| 15 | + > crc is actually xxh64 iirc |
16 | 16 |
|
17 | | -## *.stone |
| 17 | +## Software distribution via *.stone packages |
18 | 18 |
|
19 | 19 | AerynOS distributes software via its custom `stone` format. This format was explicitly built to enable fast, deduplicated transmission and installation of software artefacts on target OS installs. |
20 | 20 |
|
21 | | -> **Ikey Doherty** |
22 | | -> Context: we dont mix layout + metadata (unlike in alpine, where tar records are used for metadata) |
23 | | -> in fact we explicitly separate them |
24 | | -> so a "normal" stone file has a meta payload with strongly typed/tagged key value pairs/sets |
25 | | -> a content payload which is every unique file concatanated into a "megablob" and compressed singly |
26 | | -> an index payload which is a jump table into offsets in the unpacked content payload |
27 | | -> to allow the xxhash128 keying |
28 | | -> ie "position one is hash xyz" |
29 | | -> and lastly there is the layout payload which is a meta-ish payload containing a set of records that define how the package is laid out on disk when installed |
30 | | -> so the paths, file types, modes, link targets, permissions |
31 | | -> and optionally for regular files, the xxh128 hash |
32 | | -> so when we "cache" / install a package, in reality we're ripping the content payload out, then using the index payload to shard it into the unique assets in the store to build up the content addressable storage |
33 | | -> we then merge the entries from metapayload + layoutpayload into the DBs |
34 | | -> and we use the unique package "id" to key it, ie the hash for the `.stone` |
35 | | -> internally moss has a notion of "State" whereby it maps your explicitly / transitive selections for a transaction into those pkgids |
36 | | -> and during composition ("blit") we load all the required ids/etc/ and produce an in memory VFS structure using those "LayoutEntry" |
37 | | -> and then blit in optimal order into a tree using linkat, mkdirat, etc. |
38 | | -> we also do some graph ordering and reparenting to detect filesystem conflicts ahead of time |
39 | | -> and to solve symlinked directories chicken/egg ordering |
40 | | -> and detect those conflicts too.. |
41 | | -> anyway, the main sauce is then linkat for all the $hash -> $root/$path |
42 | | -> and is delta-friendly (in future) by not locking paths to contents |
43 | | -> In summary: "A lot more than a single tar file can do." |
44 | | -> the vfs crate is actually borderline devil magic |
45 | | -> https://github.com/AerynOS/os-tools/blob/main/crates/vfs/src/tree/mod.rs |
46 | | -> https://github.com/AerynOS/os-tools/blob/main/crates/vfs/src/tree/builder.rs |
47 | | -> but it does mean we can bake the view of the applied/installed OS ahead of time in memory and organise/optimise it |
48 | | -> and we use that to make "new" installs each time under /.moss/root/staging |
49 | | -> we then do some magic shit with linux containers (kernel namespaces) to enter the new system and run some triggers in an ephemeral copy |
50 | | -> and eventually we swap that staging system with /usr using renameat2 / atomic exchange / rename |
51 | | -> and ofc swap the one now at staging into an archived ID |
52 | | -> as well as handling the boot management via blsforme.. |
53 | | -> i mean its basically systemd of package managers. |
54 | | -> on crack |
55 | | -> so when folks say "hey apk is super fast" im like "is it any fucking wonder" |
56 | | -> it does nothing |
57 | | -> look at us |
58 | | -> only thing moss isnt doing is delivering pizzas on the side |
| 21 | + > **Ikey Doherty** |
| 22 | + > Context: we dont mix layout + metadata (unlike in alpine, where tar records are used for metadata) |
| 23 | + > in fact we explicitly separate them |
| 24 | + > so a "normal" stone file has a meta payload with strongly typed/tagged key value pairs/sets |
| 25 | + > a content payload which is every unique file concatanated into a "megablob" and compressed singly |
| 26 | + > an index payload which is a jump table into offsets in the unpacked content payload |
| 27 | + > to allow the xxhash128 keying |
| 28 | + > ie "position one is hash xyz" |
| 29 | + > and lastly there is the layout payload which is a meta-ish payload containing a set of records that define how the package is laid out on disk when installed |
| 30 | + > so the paths, file types, modes, link targets, permissions |
| 31 | + > and optionally for regular files, the xxh128 hash |
| 32 | + > so when we "cache" / install a package, in reality we're ripping the content payload out, then using the index payload to shard it into the unique assets in the store to build up the content addressable storage |
| 33 | + > we then merge the entries from metapayload + layoutpayload into the DBs |
| 34 | + > and we use the unique package "id" to key it, ie the hash for the `.stone` |
| 35 | + > internally moss has a notion of "State" whereby it maps your explicitly / transitive selections for a transaction into those pkgids |
| 36 | + > and during composition ("blit") we load all the required ids/etc/ and produce an in memory VFS structure using those "LayoutEntry" |
| 37 | + > and then blit in optimal order into a tree using linkat, mkdirat, etc. |
| 38 | + > we also do some graph ordering and reparenting to detect filesystem conflicts ahead of time |
| 39 | + > and to solve symlinked directories chicken/egg ordering |
| 40 | + > and detect those conflicts too.. |
| 41 | + > anyway, the main sauce is then linkat for all the $hash -> $root/$path |
| 42 | + > and is delta-friendly (in future) by not locking paths to contents |
| 43 | + > In summary: "A lot more than a single tar file can do." |
| 44 | + > the vfs crate is actually borderline devil magic |
| 45 | + > https://github.com/AerynOS/os-tools/blob/main/crates/vfs/src/tree/mod.rs |
| 46 | + > https://github.com/AerynOS/os-tools/blob/main/crates/vfs/src/tree/builder.rs |
| 47 | + > but it does mean we can bake the view of the applied/installed OS ahead of time in memory and organise/optimise it |
| 48 | + > and we use that to make "new" installs each time under /.moss/root/staging |
| 49 | + > we then do some magic shit with linux containers (kernel namespaces) to enter the new system and run some triggers in an ephemeral copy |
| 50 | + > and eventually we swap that staging system with /usr using renameat2 / atomic exchange / rename |
| 51 | + > and ofc swap the one now at staging into an archived ID |
| 52 | + > as well as handling the boot management via blsforme.. |
| 53 | + > i mean its basically systemd of package managers. |
| 54 | + > on crack |
| 55 | + > so when folks say "hey apk is super fast" im like "is it any fucking wonder" |
| 56 | + > it does nothing |
| 57 | + > look at us |
| 58 | + > only thing moss isnt doing is delivering pizzas on the side |
0 commit comments