diff --git a/README.md b/README.md index 73fb910f..6edd2377 100644 --- a/README.md +++ b/README.md @@ -3,612 +3,84 @@ [![License](https://img.shields.io/badge/License-MIT-blue.svg)](./LICENSE) [![Go Lint Check](https://github.com/open-edge-platform/os-image-composer/actions/workflows/go-lint.yml/badge.svg)](https://github.com/open-edge-platform/os-image-composer/actions/workflows/go-lint.yml) [![Unit and Coverage](https://github.com/open-edge-platform/os-image-composer/actions/workflows/unit-test-and-coverage-gate.yml/badge.svg)](https://github.com/open-edge-platform/os-image-composer/actions/workflows/unit-test-and-coverage-gate.yml) [![Security zizmor 🌈](https://github.com/open-edge-platform/os-image-composer/actions/workflows/zizmor.yml/badge.svg)](https://github.com/open-edge-platform/os-image-composer/actions/workflows/zizmor.yml) [![Fuzz test](https://github.com/open-edge-platform/os-image-composer/actions/workflows/fuzz-test.yml/badge.svg)](https://github.com/open-edge-platform/os-image-composer/actions/workflows/fuzz-test.yml) [![Trivy scan](https://github.com/open-edge-platform/os-image-composer/actions/workflows/trivy-scan.yml/badge.svg)](https://github.com/open-edge-platform/os-image-composer/actions/workflows/trivy-scan.yml) -OS Image Composer is a command-line tool that uses a simple toolchain to build mutable or immutable Linux distributions from the pre-built packages from different OS distribution repositories. -Developed in the Go programming language, or Golang, the tool initially builds custom images for [Edge Microvisor Toolkit](https://github.com/open-edge-platform/edge-microvisor-toolkit), [Azure Linux](https://github.com/microsoft/azurelinux) and [Wind River eLxr](https://www.windriver.com/blog/Introducing-eLxr). +A command-line tool for building custom Linux images from pre-built packages. Define what you need in a YAML template, run one command, and get a bootable RAW or ISO image. -## Get Started +## Supported Distributions -The initial release of the OS Image Composer tool has been tested and validated to work with Ubuntu 24.04, which is the recommended distribution for running the tool. Other standard Linux distributions should also work but haven't been validated. The plan for later releases is to include a containerized version to support portability across operating systems. +| OS | Distribution | Architecture | +|----|-------------|--------------| +| Azure Linux | azl3 | x86_64 | +| Edge Microvisor Toolkit | emt3 | x86_64 | +| Wind River eLxr | elxr12 | x86_64 | +| Ubuntu | ubuntu24 | x86_64 | -* Download the tool by cloning and checking out the latest tagged release on [GitHub](https://github.com/open-edge-platform/os-image-composer/). Alternatively, you can download the [latest tagged release](https://github.com/open-edge-platform/os-image-composer/releases) of the ZIP archive. +## Quick Start -* Install version 1.22.12 or later of the Go programming language before building the tool; see the [Go installation instructions](https://go.dev/doc/manage-install) for your Linux distribution. +### 1. Build the Tool - -### Build the Tool - -Build the OS Image Composer command-line utility by using Go directly or by using the Earthly framework: - -#### Development Build (Go) - -For development and testing purposes, you can use Go directly: +Requires Go 1.22.12+ on Ubuntu 24.04 (recommended). ```bash -# Build the tool: +git clone https://github.com/open-edge-platform/os-image-composer.git +cd os-image-composer go build -buildmode=pie -ldflags "-s -w" ./cmd/os-image-composer - -# Build the live-installer (Required for ISO image): -go build -buildmode=pie -o ./build/live-installer -ldflags "-s -w" ./cmd/live-installer - -# Or run it directly: -go run ./cmd/os-image-composer --help ``` -**Note**: Development builds using `go build` will show default version information (e.g., `Version: 0.1.0`, `Build Date: unknown`). This is expected during development. +This produces `./os-image-composer` in the repo root. -To include version information in a development build, use ldflags with git commands: +Alternatively, use [Earthly](https://earthly.dev/) for reproducible production +builds (output: `./build/os-image-composer`): ```bash -VERSION=$(git describe --tags --always --dirty 2>/dev/null || echo "dev") -COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") -BUILD_DATE=$(date -u '+%Y-%m-%d') - -go build -buildmode=pie \ - -ldflags "-s -w \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Version=$VERSION' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Toolname=Image-Composer' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Organization=Open Edge Platform' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.BuildDate=$BUILD_DATE' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.CommitSHA=$COMMIT'" \ - ./cmd/os-image-composer - -# Required for ISO image -go build -buildmode=pie \ - -o ./build/live-installer \ - -ldflags "-s -w \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Version=$VERSION' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Toolname=Image-Composer' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Organization=Open Edge Platform' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.BuildDate=$BUILD_DATE' \ - -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.CommitSHA=$COMMIT'" \ - ./cmd/live-installer -``` - -#### Production Build (Earthly) - -For production and release builds, use the Earthly framework, which produces a reproducible build that automatically includes the version number (from git tags), the build date (the current UTC date), and the Git commit SHA: - -```bash -# Default build (uses latest git tag for version) earthly +build - -# Build with custom version metadata -earthly +build --VERSION=1.2.0 - -### Install via Debian Package (Ubuntu/Debian) - -For Ubuntu and Debian systems, you can build and install OS Image Composer as a Debian package. This method provides a cleaner installation with proper package management. - -#### Build the Debian Package - -Use the Earthly `+deb` target to create a `.deb` package: - -```bash -# Build with default parameters (latest git tag, amd64) -earthly +deb - -# Build with custom version and architecture -earthly +deb --VERSION=1.2.0 --ARCH=amd64 - -# Build for ARM64 -earthly +deb --VERSION=1.0.0 --ARCH=arm64 -``` - -The package will be created in the `dist/` directory as `os-image-composer__.deb`. A companion file `dist/os-image-composer.version` captures the resolved version when the package was built. - -#### Install the Package - -```bash -# Install using apt (recommended - automatically resolves and installs dependencies) -sudo apt install /os-image-composer_1.0.0_amd64.deb - -# Or using dpkg (requires manual dependency installation) -# First install required dependencies: -sudo apt-get update -sudo apt-get install -y bash coreutils unzip dosfstools xorriso grub-common -# Then install the package: -sudo dpkg -i dist/os-image-composer_1.0.0_amd64.deb -# Optionally install bootstrap tools: -sudo apt-get install -y mmdebstrap || sudo apt-get install -y debootstrap -``` - -**Note:** Using `apt install` is strongly recommended as it automatically handles all dependencies. If you use `dpkg -i` and encounter dependency errors, run `sudo apt-get install -f` to fix them. - -#### Verify Installation - -```bash -# Check if package is installed -dpkg -l | grep os-image-composer - -# View installed files -dpkg -L os-image-composer - -# Verify the binary works -os-image-composer version ``` -#### Package Contents - -The Debian package installs the following files: - -* **Binary:** `/usr/local/bin/os-image-composer` - Main executable -* **Configuration:** `/etc/os-image-composer/` - Default configuration and OS variant configs - - `/etc/os-image-composer/config.yml` - Global configuration with system paths - - `/etc/os-image-composer/config/` - OS variant configuration files -* **Examples:** `/usr/share/os-image-composer/examples/` - Sample image templates -* **Documentation:** `/usr/share/doc/os-image-composer/` - README, LICENSE, and CLI specification -* **Cache Directory:** `/var/cache/os-image-composer/` - Package cache storage - -After installation via the Debian package, you can use `os-image-composer` directly from any directory. The configuration is pre-set to use system paths, and you can reference the example templates from `/usr/share/os-image-composer/examples/`. - -#### Package Dependencies - -The Debian package automatically installs the following runtime dependencies: +See the [Installation Guide](./docs/tutorial/installation.md) for Debian +package installation, prerequisite setup, and other options. -**Required Dependencies:** -* `bash` - Shell for script execution -* `coreutils` - Core GNU utilities -* `unzip` - Archive extraction utility -* `dosfstools` - FAT filesystem utilities -* `xorriso` - ISO image creation tool -* `grub-common` - Bootloader utilities - -**Recommended Dependencies (installed if available):** -* `mmdebstrap` - Debian bootstrap tool (preferred, version 1.4.3+ required) -* `debootstrap` - Alternative Debian bootstrap tool - -**Important:** `mmdebstrap` version 0.8.x (included in Ubuntu 22.04) has known issues. For Ubuntu 22.04 users, you must install `mmdebstrap` version 1.4.3+ manually as described in the [prerequisite documentation](./docs/tutorial/prerequisite.md#mmdebstrap). - -#### Uninstall the Package +### 2. Install Prerequisites ```bash -# Remove package (keeps configuration files) -sudo dpkg -r os-image-composer - -# Remove package and configuration files -sudo dpkg --purge os-image-composer +# Required for image composition +sudo apt install systemd-ukify mmdebstrap ``` -### Install the Prerequisites for Composing an Image - -Before you compose an operating system image with the OS Image Composer tool, you need to install additional prerequisites: - -**Required Tools:** - -* **`ukify`** - Combines kernel, initrd, and UEFI boot stub to create signed Unified Kernel Images (UKI) - * **Ubuntu 23.04+**: Available via `sudo apt install systemd-ukify` - * **Ubuntu 22.04 and earlier**: Must be installed manually from systemd source - * See [detailed ukify installation instructions](./docs/tutorial/prerequisite.md#ukify) +> Specific versions and alternative installation methods are documented in the +> [Installation Guide](./docs/tutorial/installation.md#image-composition-prerequisites). -* **`mmdebstrap`** - Downloads and installs Debian packages to initialize a chroot - * **Ubuntu 23.04+**: Automatically installed with the Debian package (version 1.4.3+) - * **Ubuntu 22.04**: The version in Ubuntu 22.04 repositories (0.8.x) has known bugs and will not work - * **Required:** Manually install version 1.4.3+ following [mmdebstrap installation instructions](./docs/tutorial/prerequisite.md#mmdebstrap) - * **Alternative**: Can use `debootstrap` for Debian-based images - -**Note:** If you installed os-image-composer via the Debian package, `mmdebstrap` may already be installed. You still need to install `ukify` separately following the instructions above. - -### Compose or Validate an Image - -Now you're ready to compose an image from a built-in template or validate a template. +### 3. Compose an Image ```bash -# Build an image from template +# If built with go build: sudo -E ./os-image-composer build image-templates/azl3-x86_64-edge-raw.yml -# If installed via Debian package, use system paths: -sudo os-image-composer build /usr/share/os-image-composer/examples/azl3-x86_64-edge-raw.yml - -> Note: The default configuration at `/etc/os-image-composer/config.yml` is discovered automatically; no extra flags are required. - -# Validate a template: -./os-image-composer validate image-templates/azl3-x86_64-edge-raw.yml -``` - -After the image finishes building, check your output directory. The exact name of the output directory varies by environment and image but should look something like this: - +# If built with earthly: +sudo -E ./build/os-image-composer build image-templates/azl3-x86_64-edge-raw.yml ``` -/os-image-composer/tmp/os-image-composer/azl3-x86_64-edge-raw/imagebuild/Minimal_Raw -``` - -To build an image from your own template, see [Creating and Reusing Image Templates](./docs/architecture/os-image-composer-templates.md). For complete usage instructions, see the [Command-Line Reference](./docs/architecture/os-image-composer-cli-specification.md). - -## Configuration - -### Global Configuration - -The OS Image Composer tool supports global configuration files for setting tool-level parameters that apply across all image builds. Image-specific parameters are defined in YAML image template files. See [Understanding the OS Image Build Process](./docs/architecture/os-image-composer-build-process.md). +That's it. The output image will be under +`./workspace/--/imagebuild//` +(or `/tmp/os-image-composer/...` if installed via the Debian package). -### Configuration File Locations +### 4. Validate a Template (Optional) -The tool searches for configuration files in the following order: - -1. `os-image-composer.yaml` (current directory) -2. `os-image-composer.yml` (current directory) -3. `.os-image-composer.yaml` (hidden file in current directory) -4. `~/.os-image-composer/config.yaml` (user home directory) -5. `~/.config/os-image-composer/config.yaml` (XDG config directory) -6. `/etc/os-image-composer/config.yaml` (system-wide) - -**Note:** When installed via the Debian package, the default configuration is located at `/etc/os-image-composer/config.yml` and is pre-configured with system paths. - - -### Configuration Parameters - -```yaml -# Core tool settings -workers: 12 # Number of concurrent download workers (1-100, default: 8) -cache_dir: "/var/cache/os-image-composer" # Package cache directory (default: ./cache) -work_dir: "/tmp/os-image-composer" # Working directory for builds (default: ./workspace) -temp_dir: "" # Temporary directory (empty = system default) - -# Logging configuration -logging: - level: "info" # Log level: debug, info, warn, error (default: info) -``` - -### Configuration Management Commands +Check a template for errors before starting a build: ```bash -# Create a new configuration file -./os-image-composer config init - -# Create config file at specific location -./os-image-composer config init /path/to/config.yaml - -# Show current configuration -./os-image-composer config show - -# Use specific configuration file -./os-image-composer --config /path/to/config.yaml build template.yml -``` - -## Operations Requiring Sudo Access - -The OS Image Composer performs several system-level operations that require elevated privileges (sudo access). - -### System Directory Access and Modification - -The following system directories require root access for OS Image Composer operations: - -- **`/etc/` directory operations**: Writing system configuration files, modifying network configurations, updating system settings -- **`/dev/` device access**: Block device operations, loop device management, hardware access -- **`/sys/` filesystem access**: System parameter modification, kernel interface access -- **`/proc/` filesystem modification**: Process and system state changes -- **`/boot/` directory**: Boot loader and kernel image management -- **`/var/` system directories**: System logs, package databases, runtime state -- **`/usr/sbin/` and `/sbin/`**: System administrator binaries - -### Common Privileged Operations - -OS Image Composer typically requires sudo for: - -- **Block device management**: Creating loop devices, partitioning, filesystem creation -- **Mount/unmount operations**: Mounting filesystems, managing mount points -- **Chroot environment setup**: Creating and managing isolated build environments -- **Package installation**: System-wide package management operations -- **Boot configuration**: Installing bootloaders, managing EFI settings -- **Security operations**: Secure boot signing, cryptographic operations - -## Usage - -The OS Image Composer tool uses a command-line interface with various commands. Here are some examples: - -```bash -# Show help -./os-image-composer --help - -# Build command with template file as positional argument -sudo -E ./os-image-composer build image-templates/azl3-x86_64-edge-raw.yml - -# Override config settings with command-line flags -sudo -E ./os-image-composer build --workers 16 --cache-dir /tmp/cache image-templates/azl3-x86_64-edge-raw.yml - -# Validate a template file against the schema ./os-image-composer validate image-templates/azl3-x86_64-edge-raw.yml - -# Display version information -./os-image-composer version - -# Install shell completion for your current shell -./os-image-composer install-completion -``` - -### Commands - -The OS Image Composer tool provides the following commands: - -#### build - -Builds a Linux distribution image based on the specified image template file: - -```bash -sudo -E ./os-image-composer build [flags] TEMPLATE_FILE -``` - -Flags: - -- `--workers, -w`: Number of concurrent download workers (overrides the configuration file) -- `--cache-dir, -d`: Package cache directory (overrides the configuration file) -- `--work-dir`: Working directory for builds (overrides the configuration file) -- `--verbose, -v`: Enable verbose output -- `--dotfile, -f`: Generate a dependency graph for the merged template as a dot file (nodes are color-coded: essentials in pale yellow, template packages in green, kernel in blue, bootloader in orange) -- `--config`: Path to the configuration file -- `--log-level`: Log level (debug, info, warn, and error) -- `--log-file`: Override the log file path defined in the configuration - -Example: - -```bash -sudo -E ./os-image-composer build --workers 12 --cache-dir ./package-cache image-templates/azl3-x86_64-edge-raw.yml -``` - -#### config - -Manages the global configuration: - -```bash -# Show current configuration -./os-image-composer config show - -# Initialize new configuration file -./os-image-composer config init [config-file] -``` - -#### validate - -Validates a YAML template file against the schema without building an image: - -```bash -./os-image-composer validate TEMPLATE_FILE -``` - -The `os-image-composer validate` command is useful for verifying template configurations before starting the potentially time-consuming build process. - -#### version - -Displays the tool's version number, build date, and Git commit SHA: - -```bash -./os-image-composer version -``` - -**Note**: The version information depends on how the binary was built: -- **Earthly build** (`earthly +build`): Shows actual version from git tags, build date, and commit SHA -- **Simple Go build** (`go build`): Shows default development values unless ldflags are used -- For production releases, always use the Earthly build or equivalent build system that injects version information - -#### install-completion - -Installs the shell completion feature for your current shell or a specified shell: - -```bash -# Auto-detect shell and create completion file -./os-image-composer install-completion - -# Specify shell type -./os-image-composer install-completion --shell bash -./os-image-composer install-completion --shell zsh -./os-image-composer install-completion --shell fish -./os-image-composer install-completion --shell powershell - -# Force overwrite existing completion files -./os-image-composer install-completion --force -``` - -**Important**: The command creates completion files but additional activation steps are required: - -Bash: - -```bash -# Add to your ~/.bashrc -echo "source ~/.bash_completion.d/os-image-composer.bash" >> ~/.bashrc -source ~/.bashrc -``` - -Reload your shell configuration based on the shell that you are using: - -Bash: - -```bash -source ~/.bashrc -``` - -Zsh (May need fpath setup): - -```zsh -# Ensure completion directory is in fpath (add to ~/.zshrc if needed) -echo 'fpath=(~/.zsh/completion $fpath)' >> ~/.zshrc -source ~/.zshrc -``` - -Fish (Works automatically): - -```fish -# Just restart your terminal -``` - -PowerShell (May need execution policy): - -```powershell -# May need to allow script execution -Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser -. $PROFILE -``` - -Test the completion: - -```bash -os-image-composer [TAB] -os-image-composer b[TAB] -os-image-composer build --[TAB] ``` -### Image Template Format +## Image Templates -Written in the YAML format, templates define the requirements for building an operating system image. The template structure enables you to define key parameters, such as the operating system distribution, version, architecture, software packages, output format, and kernel configuration. The image template format is validated against a JSON schema to check syntax and semantics before building the image. +Templates are YAML files that define what goes into your image — OS, packages, +kernel, partitioning, and more. Ready-to-use examples are in +[`image-templates/`](./image-templates/). -If you're an entry-level user or have straightforward requirements, you can reuse the basic template and add the packages you require. If you're addressing an advanced use case with, for instance, robust security requirements, you can modify the template to define disc and partition layouts and other settings for security. +A minimal template looks like this: ```yaml image: - name: azl3-x86_64-edge - version: "1.0.0" - -target: - os: azure-linux # Target OS name - dist: azl3 # Target OS distribution - arch: x86_64 # Target OS architecture - imageType: raw # Image type: raw, iso - -systemConfigs: - - name: edge - description: Default configuration for edge image - - # Package Configuration - packages: - # Additional packages beyond the base system - - openssh-server # Remote access - - docker-ce # Container runtime - - vim # Text editor - - curl # HTTP client - - wget # File downloader - - # Kernel Configuration - kernel: - version: "6.12" - cmdline: "quiet splash" -``` - -#### Key Components - -Here are the key components of an image template. - -##### 1. `image` - -Basic image identification and metadata: -- `name`: Name of the resulting image -- `version`: Version for tracking and naming - -##### 2. `target` - -Defines the target OS and image configuration: -- `os`: Target OS (`azure-linux`, `emt`, and `elxr`) -- `dist`: Distribution identifier (`azl3`, `emt3`, and `elxr12`) -- `arch`: Target architecture (`x86_64`and `aarch64`) -- `imageType`: Output format (`raw`, `iso`) - -##### 3. `systemConfigs` - -Array of system configurations that define what goes into the image: -- `name`: Configuration name -- `description`: Human-readable description -- `packages`: List of packages to include in the OS build -- `kernel`: Kernel configuration with version and command-line parameters - -#### Supported Distributions - -| OS | Distribution | Version | Provider | -|----|-------------|---------|----------| -| azure-linux | azl3 | 3 | AzureLinux3 | -| emt | emt3 | 3.0 | EMT3.0 | -| elxr | elxr12 | 12 | eLxr12 | - -#### Package Examples - -You can include common packages: -- `cloud-init`: For initializing cloud instances -- `python3`: The Python 3 programming language interpreter -- `rsyslog`: A logging system for Linux OS -- `openssh-server`: SSH server for remote access -- `docker-ce`: Docker container runtime - -### Shell Completion Feature - -The OS Image Composer CLI supports shell auto-completion for the Bash, Zsh, Fish, and PowerShell command-line shells. This feature helps users discover and use commands and flags more efficiently. - -#### Generate Completion Scripts - -```bash -# Bash -./os-image-composer completion bash > os-image-composer_completion.bash - -# Zsh -./os-image-composer completion zsh > os-image-composer_completion.zsh - -# Fish -./os-image-composer completion fish > os-image-composer_completion.fish - -# PowerShell -./os-image-composer completion powershell > os-image-composer_completion.ps1 -``` - -#### Install Completion Scripts - -After you install the completion script for your command-line shell, you can use tab completion to navigate commands, flags, and arguments. - -**Bash**: - -```bash -# Temporary use -source os-image-composer_completion.bash - -# Permanent installation (Linux) -sudo cp os-image-composer_completion.bash /etc/bash_completion.d/ -# or add to your ~/.bashrc -echo "source /path/to/os-image-composer_completion.bash" >> ~/.bashrc -``` - -**Zsh**: - -```bash -# Add to your .zshrc -echo "source /path/to/os-image-composer_completion.zsh" >> ~/.zshrc -# Or copy to a directory in your fpath -cp os-image-composer_completion.zsh ~/.zfunc/_os-image-composer -``` - -**Fish**: - -```bash -cp os-image-composer_completion.fish ~/.config/fish/completions/os-image-composer.fish -``` - -**PowerShell**: - -```powershell -# Add to your PowerShell profile -echo ". /path/to/os-image-composer_completion.ps1" >> $PROFILE -``` - -#### Examples of Completion in Action - -Once the completion script is installed, the tool is configured to suggest YAML files when completing the template file argument for the build and validate commands, and you can see that in action: - -```bash -# Tab-complete commands -./os-image-composer -build completion config help validate version - -# Tab-complete flags -sudo -E ./os-image-composer build -- ---cache-dir --config --help --log-level --verbose --work-dir --workers - -# Tab-complete YAML files for template file argument -sudo -E ./os-image-composer build -# Will show YAML files in the current directory -``` - -## Template Examples - -Here are several example YAML template files. You can use YAML image templates to rapidly reproduce custom, verified, and inventoried operating systems; see [Creating and Reusing Image Templates](./docs/architecture/os-image-composer-templates.md). - -### Minimal Edge Device - -```yaml -image: - name: minimal-edge + name: my-edge-device version: "1.0.0" target: @@ -617,87 +89,44 @@ target: arch: x86_64 imageType: raw -systemConfigs: - - name: minimal - description: Minimal edge device configuration - packages: - - openssh-server - - ca-certificates - kernel: +systemConfig: + name: edge + packages: + - openssh-server + - ca-certificates + kernel: version: "6.12" cmdline: "quiet" ``` -### Development Environment - -```yaml -image: - name: dev-environment - version: "1.0.0" - -target: - os: azure-linux - dist: azl3 - arch: x86_64 - imageType: raw - -systemConfigs: - - name: development - description: Development environment with tools - packages: - - openssh-server - - git - - docker-ce - - vim - - curl - - wget - - python3 - kernel: - version: "6.12" - cmdline: "quiet splash" -``` - -### Edge Microvisor Toolkit - -```yaml -image: - name: emt-edge-device - version: "1.0.0" +To learn about template structure, variable substitution, and best practices, +see [Creating and Reusing Image Templates](./docs/architecture/os-image-composer-templates.md). -target: - os: emt - dist: emt3 - arch: x86_64 - imageType: raw +## Documentation -systemConfigs: - - name: edge - description: Edge Microvisor Toolkit configuration - packages: - - openssh-server - - docker-ce - - edge-runtime - - telemetry-agent - kernel: - version: "6.12" - cmdline: "quiet splash systemd.unified_cgroup_hierarchy=0" -``` +| Guide | Description | +|-------|-------------| +| [Installation Guide](./docs/tutorial/installation.md) | Build options, Debian packaging, prerequisites | +| [Usage Guide](./docs/tutorial/usage-guide.md) | CLI commands, configuration, shell completion | +| [CLI Reference](./docs/architecture/os-image-composer-cli-specification.md) | Complete command-line specification | +| [Image Templates](./docs/architecture/os-image-composer-templates.md) | Template system and customization | +| [Build Process](./docs/architecture/os-image-composer-build-process.md) | How image composition works, troubleshooting | +| [Secure Boot](./docs/tutorial/configure-secure-boot.md) | Configuring secure boot for images | +| [Multiple Repos](./docs/architecture/os-image-composer-multi-repo-support.md) | Using multiple package repositories | ## Get Help -* Run the following command in the command-line tool to see all the commands and options: `./os-image-composer --help` -* See the [CLI Specification and Reference](./docs/architecture/os-image-composer-cli-specification.md). -* Read the [documentation](https://github.com/open-edge-platform/os-image-composer/tree/main/docs). -* Troubleshoot by using the [Build Process documentation](./docs/architecture/os-image-composer-build-process.md#troubleshooting-build-issues). -* [Participate in discussions](https://github.com/open-edge-platform/os-image-composer/discussions). +- Run `./os-image-composer --help` +- [Browse the documentation](./docs/) +- [Start a discussion](https://github.com/open-edge-platform/os-image-composer/discussions) +- [Troubleshoot build issues](./docs/architecture/os-image-composer-build-process.md#troubleshooting-build-issues) ## Contribute -* [Open an issue](https://github.com/open-edge-platform/os-image-composer/issues). -* [Report a security vulnerability](./SECURITY.md). -* [Submit a pull request](https://github.com/open-edge-platform/os-image-composer/pulls). - +- [Open an issue](https://github.com/open-edge-platform/os-image-composer/issues) +- [Report a security vulnerability](./SECURITY.md) +- [Submit a pull request](https://github.com/open-edge-platform/os-image-composer/pulls) -## License Information +## License -See [License](./LICENSE). +[MIT](./LICENSE) diff --git a/docs/tutorial/installation.md b/docs/tutorial/installation.md new file mode 100644 index 00000000..0654bb6f --- /dev/null +++ b/docs/tutorial/installation.md @@ -0,0 +1,197 @@ +# Installation Guide + +This guide covers all the ways to install OS Image Composer and its +prerequisites. + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Development Build (Go)](#development-build-go) +- [Production Build (Earthly)](#production-build-earthly) +- [Install via Debian Package](#install-via-debian-package) +- [Image Composition Prerequisites](#image-composition-prerequisites) + +--- + +## Prerequisites + +- **OS**: Ubuntu 24.04 (recommended; other standard Linux distributions may + work but are not validated) +- **Go**: Version 1.22.12 or later — see + [Go installation instructions](https://go.dev/doc/manage-install) + +## Development Build (Go) + +For development and testing purposes, build directly with Go. +The binary is placed in the **repo root** as `./os-image-composer`. + +```bash +# Build the tool (output: ./os-image-composer) +go build -buildmode=pie -ldflags "-s -w" ./cmd/os-image-composer + +# Build the live-installer (required for ISO images) +go build -buildmode=pie -o ./build/live-installer -ldflags "-s -w" ./cmd/live-installer + +# Or run without building +go run ./cmd/os-image-composer --help +``` + +> **Note:** Development builds show default version information (e.g., +> `Version: 0.1.0`, `Build Date: unknown`). This is expected. + +### Including Version Information in Dev Builds + +To embed version metadata via ldflags: + +```bash +VERSION=$(git describe --tags --always --dirty 2>/dev/null || echo "dev") +COMMIT=$(git rev-parse --short HEAD 2>/dev/null || echo "unknown") +BUILD_DATE=$(date -u '+%Y-%m-%d') + +go build -buildmode=pie \ + -ldflags "-s -w \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Version=$VERSION' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Toolname=Image-Composer' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Organization=Open Edge Platform' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.BuildDate=$BUILD_DATE' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.CommitSHA=$COMMIT'" \ + ./cmd/os-image-composer + +# Required for ISO images +go build -buildmode=pie \ + -o ./build/live-installer \ + -ldflags "-s -w \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Version=$VERSION' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Toolname=Image-Composer' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.Organization=Open Edge Platform' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.BuildDate=$BUILD_DATE' \ + -X 'github.com/open-edge-platform/os-image-composer/internal/config/version.CommitSHA=$COMMIT'" \ + ./cmd/live-installer +``` + +## Production Build (Earthly) + +For reproducible production builds with automatic version injection. +The binary is placed in `./build/os-image-composer`. + +```bash +# Default build (output: ./build/os-image-composer) +earthly +build + +# Build with custom version metadata +earthly +build --VERSION=1.2.0 +``` + +## Install via Debian Package + +For Ubuntu and Debian systems, OS Image Composer can be installed as a `.deb` +package with proper package management. + +### Build the Debian Package + +```bash +# Build with default parameters (latest git tag, amd64) +earthly +deb + +# Build with custom version +earthly +deb --VERSION=1.2.0 +``` + +The package is created in the `dist/` directory as +`os-image-composer__.deb`. + +### Install the Package + +```bash +# Recommended — automatically resolves dependencies +sudo apt install /os-image-composer_1.0.0_amd64.deb +``` + +Alternatively, using `dpkg`: + +```bash +sudo apt-get update +sudo apt-get install -y bash coreutils unzip dosfstools xorriso grub-common +sudo dpkg -i dist/os-image-composer_1.0.0_amd64.deb +# Optional bootstrap tools: +sudo apt-get install -y mmdebstrap || sudo apt-get install -y debootstrap +``` + +> **Tip:** If `dpkg -i` reports dependency errors, run +> `sudo apt-get install -f` to resolve them. + +### Verify Installation + +```bash +dpkg -l | grep os-image-composer +os-image-composer version +``` + +### Package Contents + +| Path | Description | +|------|-------------| +| `/usr/local/bin/os-image-composer` | Main executable | +| `/etc/os-image-composer/config.yml` | Global configuration | +| `/etc/os-image-composer/config/` | OS variant configuration files | +| `/usr/share/os-image-composer/examples/` | Sample image templates | +| `/usr/share/doc/os-image-composer/` | README, LICENSE, CLI specification | +| `/var/cache/os-image-composer/` | Package cache storage | + +### Package Dependencies + +**Required:** + +- `bash`, `coreutils`, `unzip`, `dosfstools`, `xorriso`, `grub-common` + +**Recommended (installed if available):** + +- `mmdebstrap` (preferred, version 1.4.3+ required) +- `debootstrap` (alternative) + +> **Important:** `mmdebstrap` version 0.8.x (included in Ubuntu 22.04) has +> known issues. For Ubuntu 22.04, install version 1.4.3+ manually — see +> [mmdebstrap instructions](./prerequisite.md#mmdebstrap). + +### Uninstall + +```bash +# Remove package (keeps config files) +sudo dpkg -r os-image-composer + +# Remove package and config files +sudo dpkg --purge os-image-composer +``` + +## Image Composition Prerequisites + +Before composing images, install these additional tools: + +### ukify + +Combines kernel, initrd, and UEFI boot stub to create signed Unified Kernel +Images (UKI). + +- **Ubuntu 23.04+**: `sudo apt install systemd-ukify` +- **Ubuntu 22.04 and earlier**: Install manually from systemd source — see + [ukify installation instructions](./prerequisite.md#ukify) + +### mmdebstrap + +Downloads and installs Debian packages to initialize a chroot. + +- **Ubuntu 23.04+**: Available in system repositories (version 1.4.3+) +- **Ubuntu 22.04**: The repository version (0.8.x) will not work — install + 1.4.3+ manually per the + [mmdebstrap instructions](./prerequisite.md#mmdebstrap) +- **Alternative**: `debootstrap` can be used for Debian-based images + +--- + +## Next Steps + +- [Quick Start](../../README.md#quick-start) — build your first image +- [Usage Guide](./usage-guide.md) — CLI commands, configuration, and + shell completion +- [Image Templates](../architecture/os-image-composer-templates.md) — + creating and reusing templates diff --git a/docs/tutorial/usage-guide.md b/docs/tutorial/usage-guide.md new file mode 100644 index 00000000..7b8dd1ba --- /dev/null +++ b/docs/tutorial/usage-guide.md @@ -0,0 +1,254 @@ +# Usage Guide + +A practical guide for common OS Image Composer workflows. For the complete +command reference, see the +[CLI Specification](../architecture/os-image-composer-cli-specification.md). + +## Table of Contents + +- [Usage Guide](#usage-guide) + - [Table of Contents](#table-of-contents) + - [Binary Location](#binary-location) + - [Commands Overview](#commands-overview) + - [Building an Image](#building-an-image) + - [Build Output](#build-output) + - [Validating a Template](#validating-a-template) + - [Configuration](#configuration) + - [Operations Requiring Sudo](#operations-requiring-sudo) + - [Shell Completion](#shell-completion) + - [Template Examples](#template-examples) + - [Minimal Edge Device](#minimal-edge-device) + - [Development Environment](#development-environment) + - [Edge Microvisor Toolkit](#edge-microvisor-toolkit) + - [Related Documentation](#related-documentation) + +--- + +## Binary Location + +The path to the `os-image-composer` binary depends on how you built or +installed it: + +| Build method | Binary path | +|-------------|-------------| +| `go build ./cmd/os-image-composer` | `./os-image-composer` | +| `earthly +build` | `./build/os-image-composer` | +| Debian package | `os-image-composer` (installed to `/usr/local/bin/`) | + +The examples below use `./os-image-composer` (the `go build` location). +Substitute the path that matches your setup. + +## Commands Overview + +```bash +os-image-composer build # Build an image from a template +os-image-composer validate # Validate a template without building +os-image-composer inspect # Inspect a raw image's structure +os-image-composer compare # Compare two images +os-image-composer cache clean # Manage cached artifacts +os-image-composer config # Manage configuration (init, show) +os-image-composer version # Display version info +os-image-composer --help # Show all commands and options +``` + +For the full details on every command — including `inspect`, `compare`, and +`cache` — see the +[CLI Specification](../architecture/os-image-composer-cli-specification.md#commands). + +## Building an Image + +```bash +# go build — binary is in the repo root +sudo -E ./os-image-composer build image-templates/azl3-x86_64-edge-raw.yml + +# earthly +build — binary is in ./build/ +sudo -E ./build/os-image-composer build image-templates/azl3-x86_64-edge-raw.yml + +# Debian package — binary is on PATH +sudo os-image-composer build /usr/share/os-image-composer/examples/azl3-x86_64-edge-raw.yml + +# Override config settings with flags +sudo -E ./os-image-composer build --workers 16 --cache-dir /tmp/cache image-templates/azl3-x86_64-edge-raw.yml +``` + +Common flags: `--workers`, `--cache-dir`, `--work-dir`, `--verbose`, +`--dotfile`, `--config`, `--log-level`. +See the full +[build flag reference](../architecture/os-image-composer-cli-specification.md#build-command) +for descriptions and additional flags like `--system-packages-only`. + +### Build Output + +After the image finishes building, the output is placed under the configured +`work_dir`. The full path follows this pattern: + +``` +/--/imagebuild// +``` + +The default `work_dir` depends on how you installed the tool: + +| Install method | Default `work_dir` | Example output path | +|----------------|-------------------|---------------------| +| Cloned repo | `./workspace` (relative to repo root) | `./workspace/azure-linux-azl3-x86_64/imagebuild/edge/` | +| Debian package | `/tmp/os-image-composer` | `/tmp/os-image-composer/azure-linux-azl3-x86_64/imagebuild/edge/` | + +You can override it with `--work-dir` or by setting `work_dir` in your +configuration file. + +## Validating a Template + +Check a template for errors before starting a build: + +```bash +./os-image-composer validate image-templates/azl3-x86_64-edge-raw.yml +``` + +## Configuration + +The tool uses a layered configuration: config file values are overridden by +command-line flags. A config file is auto-discovered from several standard +locations (current directory, home directory, `/etc/`), or you can specify one +explicitly with `--config`. + +```bash +# Create a default configuration file +./os-image-composer config init + +# Show the active configuration +./os-image-composer config show + +# Use a specific configuration file +./os-image-composer --config /path/to/config.yaml build template.yml +``` + +Key settings: + +| Setting | Default (cloned repo) | Default (Debian pkg) | +|---------|----------------------|----------------------| +| `workers` | 8 | 8 | +| `cache_dir` | `./cache` | `/var/cache/os-image-composer` | +| `work_dir` | `./workspace` | `/tmp/os-image-composer` | + +For the complete search order and all configuration fields, see +[Configuration Files](../architecture/os-image-composer-cli-specification.md#configuration-files) +in the CLI Specification. + +## Operations Requiring Sudo + +The `build` command requires `sudo` because it performs system-level +operations: creating loop devices, mounting filesystems, setting up chroot +environments, installing packages, and configuring bootloaders. + +Always run builds with `sudo -E` to preserve your environment variables +(such as `$PATH` and proxy settings). + +## Shell Completion + +```bash +# Auto-detect shell and install completion +./os-image-composer install-completion + +# Or specify a shell: bash, zsh, fish, powershell +./os-image-composer install-completion --shell bash +``` + +After installing, reload your shell configuration (e.g., `source ~/.bashrc`). +For per-shell activation steps and manual completion script generation, see the +[Install-Completion Command](../architecture/os-image-composer-cli-specification.md#install-completion-command) +reference. + +## Template Examples + +Templates are YAML files that define the requirements for an image build. +For the full template system documentation, see +[Creating and Reusing Image Templates](../architecture/os-image-composer-templates.md). + +The `image-templates/` directory contains ready-to-use examples for all +supported distributions and image types. + +### Minimal Edge Device + +```yaml +image: + name: minimal-edge + version: "1.0.0" + +target: + os: azure-linux + dist: azl3 + arch: x86_64 + imageType: raw + +systemConfig: + name: minimal + description: Minimal edge device configuration + packages: + - openssh-server + - ca-certificates + kernel: + version: "6.12" + cmdline: "quiet" +``` + +### Development Environment + +```yaml +image: + name: dev-environment + version: "1.0.0" + +target: + os: azure-linux + dist: azl3 + arch: x86_64 + imageType: raw + +systemConfig: + name: development + description: Development environment with tools + packages: + - openssh-server + - git + - docker-ce + - vim + - curl + - wget + - python3 + kernel: + version: "6.12" + cmdline: "quiet splash" +``` + +### Edge Microvisor Toolkit + +```yaml +image: + name: emt-edge-device + version: "1.0.0" + +target: + os: edge-microvisor-toolkit + dist: emt3 + arch: x86_64 + imageType: raw + +systemConfig: + name: edge + description: Edge Microvisor Toolkit configuration + packages: + - cloud-init + - rsyslog + kernel: + version: "6.12" + cmdline: "console=ttyS0,115200 console=tty0 loglevel=7" +``` + +--- + +## Related Documentation + +- [CLI Specification and Reference](../architecture/os-image-composer-cli-specification.md) +- [Image Templates](../architecture/os-image-composer-templates.md) +- [Build Process](../architecture/os-image-composer-build-process.md) +- [Installation Guide](./installation.md)