Skip to content

Commit d57b5be

Browse files
yockgenyockgen
authored
Create container-pipeline.md
1 parent 4247af8 commit d57b5be

1 file changed

Lines changed: 158 additions & 0 deletions

File tree

‎docs/architecture/container-pipeline.md‎

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# Multi-Stage and Multi-Container OS Image Composer
2+
3+
## Introduction: Why Multi-Stage and Multi-Container OS Builds
4+
5+
Building a modern Linux operating system distribution is no longer a simple linear process. It requires dependency isolation, reproducibility, toolchain flexibility, and support for secure artifacts like signed RPMs, Secure Boot Unified Kernel Images (UKIs), and verifiable system images. A multi-stage, multi-container build pipeline is the industry-proven approach used by modern Linux projects such as Fedora CoreOS, Flatcar, CBL-Mariner (Azure Linux), and Talos Linux.
6+
7+
In this approach, the OS is not built in a single environment. Instead, each stage of the build process — such as toolchain setup, root filesystem creation, package management, image assembly, and secure artifact signing — is executed inside separate, purpose-built containers. Each container has only the tools required for its stage.
8+
9+
For example, just concept wide:
10+
11+
```
12+
+-------------------------------------------+
13+
| Build Host (Any Linux) |
14+
| (Ubuntu, RHEL, Debian – supports CI) |
15+
+-----------------------+-------------------+
16+
|
17+
-----------------------------------------------------------------------------------------------------------------------------------
18+
| | | | |
19+
+--------------------------------+ +--------------------------------+ +--------------------------------+ +--------------------------------+
20+
| Container #1 | | Container #2 | | Container #3 | | Container #4 |
21+
| Package Fetch Stage | | RootFS Build Stage | | System Config Stage | | UKI Build & Secure Boot Stage |
22+
| (Download Debian/RPM | | (debootstrap or rpm + chroot) | | (Immutable + overlay setup) | | (ukify + sign) |
23+
| packages to local cache) | | | | (/etc, /var, /home persistence)| | (EFI+initrd+kernel packaged) |
24+
+--------------------------------+ +--------------------------------+ +--------------------------------+ +--------------------------------+
25+
| | |
26+
------------------------------------------------------------------
27+
|
28+
+-----------------------------+
29+
| Container #5 |
30+
| Image Assembly Stage |
31+
| (Partitions + systemd-boot |
32+
| + RAW image output) |
33+
+-----------------------------+
34+
|
35+
----------------------------------------------------------------------------------
36+
Output Artifacts:
37+
- Local package cache (.rpm/.deb + dependencies)
38+
- Root filesystem directory
39+
- Immutable OS root with writable overlays
40+
- Signed Unified Kernel Image (UKI)
41+
- Bootable RAW disk image
42+
43+
44+
45+
```
46+
47+
---
48+
49+
### Problems with Existing Chroot-Based Build System
50+
51+
Our current OS build tooling uses a monolithic workflow where a single chroot workspace is created and all tooling (dnf/yum, mkfs, rpm, UKI tools, signing tools, etc.) is installed inside this same environment. While this works for simple builds, it introduces serious scalability, maintenance, and reliability problems.
52+
53+
#### Key Limitations of Chroot Approach
54+
55+
- **Poor Tool Isolation**
56+
- All build dependencies must coexist inside the same chroot.
57+
- Conflicting package requirements (RPM + DEB tools cannot co-exist cleanly).
58+
- Example: `ukify` cannot be installed in a RHEL-based chroot.
59+
60+
- **Environment Contamination**
61+
- Installing tools inside the chroot modifies it continuously.
62+
- Hard to guarantee reproducible builds.
63+
- Chroot often becomes a "snowball mess" over time.
64+
65+
- **Host Dependency Problems**
66+
- Chroot still depends heavily on host kernel + userspace compatibility.
67+
- Differences in host OS versions break chroot behavior.
68+
- Example: `systemd-nspawn`, loop devices, `mkfs` behavior vary by host.
69+
70+
- **Security Risks**
71+
- Chroot is not a security boundary.
72+
- Tools installed inside chroot may accidentally modify host filesystems.
73+
- No isolation from build host; unsafe for CI.
74+
75+
- **Difficult to Maintain Toolchains**
76+
- Updating a tool version requires rebuilding and validating the entire chroot.
77+
- No easy package version pinning inside chroot.
78+
- Hard to maintain multiple toolchains for different OS targets.
79+
80+
- **Not Scalable for Complex Pipelines**
81+
- Cannot mix Debian tools in RHEL chroot or vice versa.
82+
- No modularity—every new build capability adds complexity inside the same environment.
83+
- No reuse of tooling per stage (rootfs vs. signing vs. UKI vs. ISO build).
84+
85+
---
86+
87+
### Why Containers Solve These Problems
88+
89+
| Problem in Chroot Model | Solved by Multi-Stage Containers |
90+
|------------------------|----------------------------------|
91+
| Single environment polluted by tools | Each stage uses a clean container |
92+
| Hard to cross RPM/DEB tool usage | Use different base images per stage |
93+
| Reproducibility issues | Tool versions locked by container images |
94+
| Unsafe for CI | Containers provide isolation |
95+
| Hard to debug and extend | Modular stages, easier pipeline |
96+
| Host dependency issues | Host only needs Docker/Podman |
97+
98+
---
99+
100+
### Conclusion
101+
102+
> Chroot-based builds are fragile, hard to maintain, and unsuitable for secure, reproducible OS build pipelines. Containers provide isolation, cross-platform tool support, reproducibility, security, and clean pipeline modularity. This makes them the modern industry standard for OS image composition (used by Fedora CoreOS, Flatcar, Talos Linux, CBL-Mariner, and Ubuntu Core pipelines).
103+
104+
105+
### ✅ Key Advantages of Multi-Stage Containers
106+
107+
- **No Host Dependency Issues**
108+
- Each build stage runs inside its own container with the correct toolchain.
109+
- Example: Rootfs uses RHEL/Fedora-based container with `dnf`; UKI uses Debian container with `ukify`.
110+
111+
- **Guaranteed Tool Availability**
112+
- All required build tools exist inside their container stage.
113+
- Host never needs `dnf`, `debootstrap`, `dracut`, `ukify`, or `sbsign` installed.
114+
115+
- **Supports Cross-Distro Pipelines**
116+
- Can mix **RPM-based target OS** with **Debian-based UKI tooling** in different build stages.
117+
- No longer blocked by missing cross-distro packages.
118+
119+
- **Reproducible and Deterministic**
120+
- Container versions freeze build dependencies.
121+
- Ensures consistent builds across developers and CI pipelines.
122+
123+
- **Clean Separation of Responsibilities**
124+
- Each build step has a dedicated container:
125+
- Package download/cache
126+
- Rootfs assembly
127+
- OS configuration (immutable setup, overlays, fstab, users)
128+
- Kernel + UKI build & signing
129+
- Raw disk image creation
130+
131+
- **Security and Compliance Friendly**
132+
- Reduced attack surface compared to installing tools on host.
133+
- Easier SBOM tracking and supply chain auditing.
134+
135+
- **CI/CD Friendly and Scalable**
136+
- Works with GitLab CI, GitHub Actions, Jenkins, etc.
137+
- Only requirement: Docker or Podman available on build runner.
138+
139+
---
140+
141+
### ✅ Example Problem Solved: `ukify` Missing on RPM Hosts
142+
143+
| Single Host Build (Current) | Multi-Stage Container Build (Proposed) |
144+
|-----------------------------|----------------------------------------|
145+
| UKI build fails because `ukify` not available on RHEL host | UKI stage uses Debian container with `ukify` installed |
146+
| Requires mixing DEB and RPM tools on host | No cross-tool contamination; clean separation |
147+
| Host dependency hell | Host only runs containers |
148+
| Inconsistent developer environments | Identical builds across all machines |
149+
150+
---
151+
152+
### ✅ Summary
153+
154+
> The multi-stage container approach guarantees a portable, reproducible, host-independent OS build pipeline and eliminates tool conflict issues permanently.
155+
156+
157+
158+

0 commit comments

Comments
 (0)