Portage Engine

Portage Engine is a self-hosted Gentoo binary-package build and publishing system. It provisions isolated workers, runs native emerge builds, verifies GPKG artifacts and publishes a standard Portage binhost.

Status: trusted alpha. PVE + Terraform + native Gentoo is the reference backend and has been exercised on real infrastructure. The service is for a trusted private network. OIDC identity, project RBAC and PostgreSQL-atomic queued/active/UTC-daily admission plus per-attempt vCPU, memory and disk reservations, artifact-generation budgets, daily build-time/cloud-cost budgets, failure-storm cooldown, independently fenced provision/build/verify/publish execution, administrator step-up and cross-replica OIDC session revocation, exact executor routing and durable workload certificate/issuer revocation are implemented. Vault PKI external signing, listener-bound CA rollover and token recovery have a real-container Gate. Real community IdP callbacks, production Vault HA recovery and public-service hardening remain.

Pick your workflow

You are…	Normal workflow	Extra client?
Binpkg consumer	Configure the binhost once, then use emerge --getbinpkg	No
Developer	Submit an explicit remote build, then install with emerge	portage-client
Infra operator	Manage builders, images, signing, storage and release policy	Server/dashboard tools

Consume a published package with native Portage:

sudo ./bin/portage-client configure \
  -server=http://portage-engine.infra.lan:8080 \
  -profile-id=pe/amd64/glibc/systemd/base-v1

emerge --getbinpkg app-editors/vim

# Do not fall back to a local source build:
emerge --getbinpkgonly app-editors/vim

The configure command resolves the profile to an official-style namespace such as /binpkgs/releases/amd64/binpackages/23.0/x86-64_pe-systemd-base-v1. Different profiles have independent Packages indexes; /binpkgs/ itself is not an aggregate repository.

Developers request missing packages explicitly because Portage has no native “ask this binhost to build” protocol. In OIDC/hybrid mode, select an authorized project and use a short-lived token:

export PORTAGE_ENGINE_TOKEN='read-from-your-identity-provider'
export PORTAGE_ENGINE_PROJECT='project-name-or-uuid'

./bin/portage-client build \
  -server=http://portage-engine.infra.lan:8080 \
  -package=dev-lang/python -version=3.11 \
  -profile-id=pe/amd64/glibc/systemd/base-v1 \
  -resource-class=medium -wait

The legacy PORTAGE_ENGINE_API_KEY remains a system-administrator migration and break-glass path in legacy/hybrid mode. Trusted-LAN bring-up may use HTTP, but API keys and bearer tokens are plaintext on the wire. Add HTTPS before crossing an untrusted network.

The consumer path needs neither the CLI nor an overlay. An overlay is useful only for distributing the optional client/service ebuilds; it must not own the published binpkg trust key or image/profile policy.

Components

flowchart LR
    Client[CLI / Web] --> API[Server + scheduler]
    API -->|provision + transient SSH bootstrap| Worker[Disposable PVE / native worker]
    Worker -->|outbound TLS 1.3 + attempt certificate| Gateway[Worker Gateway]
    Gateway --> API
    Worker --> Build[Native emerge]
    Build --> Stage[Unsigned quarantine]
    Stage --> Verify[Unsigned install / GUI verification]
    Verify --> Queue[(PostgreSQL signing task)]
    Queue --> Signer[Isolated outbound-pull signer]
    Signer --> SignedVerify[Signed install verification]
    SignedVerify --> Publish[Immutable signed binhost]
    Publish --> Portage[User emerge]

Loading

• portage-server: API, job state, scheduling, IaC and binhost publication.
• portage-builder: native Gentoo execution and install checks.
• portage-signer: digest-bound queue worker and the only private-key owner.
• portage-dashboard: builds, nodes, logs and read-only factory evidence.
• portage-client: optional developer CLI.
• portage-desktop-runner: deterministic native-GUI verification.
• image-factory/: offline Packer/Catalyst image and release gates.

Development quick start

Requires Go 1.26.5; real builds require a native Gentoo disposable root or VM.

git clone https://github.com/slchris/portage-engine.git
cd portage-engine
go mod download
make build
go test ./...

Run the development topology:

cp .env.compose.example .env.compose
scripts/check-compose-ports.sh .env.compose
# Safe to repeat; applies embedded reviewed SQL with the database owner.
docker compose --env-file .env.compose run --rm portage-migrate
docker compose --env-file .env.compose up -d
docker compose --env-file .env.compose ps
scripts/verify-compose.sh .env.compose

To exercise PVE workers without exposing a builder listener, generate the worker-only PKI and enable the dedicated gateway in .env.compose:

WORKER_GATEWAY_HOST=portage-engine.infra.lan \
  scripts/generate-worker-pki.sh .local/worker-pki
# Set PORTAGE_WORKER_GATEWAY_ENABLED=true, its LAN bind address, and
# PORTAGE_WORKER_GATEWAY_ADVERTISE_URL=https://portage-engine.infra.lan:19444
# Keep PORTAGE_PHASE_EXECUTOR_MODE=shadow for rollout; active additionally
# requires PostgreSQL authority, the Worker Gateway, shared state and
# capability-equivalent replicas.

This TLS requirement applies only to the worker identity channel. The normal API/dashboard can remain HTTP during trusted-LAN bring-up.

• API/binhost: http://127.0.0.1:18080
• Dashboard: http://127.0.0.1:18081 (admin / portage-demo)
• Grafana: http://127.0.0.1:23000 (admin / portage-grafana-local)
• Prometheus: http://127.0.0.1:29090

The Compose stack also publishes loopback-only development endpoints for Loki (23100), Tempo (23200), OTLP gRPC/HTTP (24317/24318), PostgreSQL (25432) and Redis (26379). Change any mapping in .env.compose when the preflight script reports a collision. The checked-in credentials are local defaults only; replace them before exposing any service beyond localhost.

Compose does not start package builders or mount the host Docker socket. It does start the control plane plus its local PostgreSQL, Redis and observability foundation. PostgreSQL/Redis metrics and current process logs are collected now; shared process log files rotate at 10 MiB with one backup and Docker JSON logs are capped separately. Schema v26 makes PostgreSQL the sole online job, infrastructure-cleanup, signing-task, external-subject and project-membership authority, including versioned project admission policy and active-attempt resource/phase/artifact/runtime reservations, phase execution context and durable Worker Gateway commands/uploads. It also owns OIDC session lifetime, idle expiry, revocation and per-subject token watermarks. The signer uses a least-privilege database role and a separate private GPG volume; server replicas see only its public key and queue status. Queue claims, attempts, leases/fencing, cancel/retry, redacted logs, cleanup leases, artifact/factory metadata and audited runtime settings survive replica failure. Publication is serialized across replicas and binpkg locations are immutable. JSON job snapshots are disabled whenever the database is enabled. Redis remains disposable. Project admission and phase dispatch use shared weighted virtual runtime with queue-age anti-starvation. Pull-aware worker score decisions and 24h/7d/30d target SLO/latency/cost history are visible in Monitor. Autoscaling can remain observe-only or write globally and per-provider budgeted single-slot actions for the separate, listener-free portage-capacity-actuator; provider calls never occur in a scheduler transaction.

Run a second local control-plane replica on port 18082:

docker compose --env-file .env.compose \
  -f docker-compose.yml -f docker-compose.ha.yml up -d

Create a logical backup and prove it restores into an isolated database:

scripts/postgres-backup.sh /mnt/portage-nas/postgres/portage-engine.dump
scripts/postgres-restore-check.sh /mnt/portage-nas/postgres/portage-engine.dump

Keep PostgreSQL PGDATA on reliable local block storage. Only backup/WAL repositories and restore evidence belong on the internal NAS. The optional PITR overlay uses an encrypted pgBackRest repository:

export PORTAGE_PGBACKREST_CIPHER_PASS='use-a-secret-provider'
export PORTAGE_PGBACKREST_REPO=/mnt/portage-nas/postgres/pgbackrest
docker compose -f docker-compose.yml -f docker-compose.pgbackrest.yml up -d postgres
scripts/pgbackrest-init.sh
scripts/pgbackrest-backup.sh full
scripts/pgbackrest-restore-drill.sh

Deployment and security

The PVE path clones a native Gentoo cloud-init template, runs emerge --buildpkg, collects the result and performs install verification. Native emerge mutates the guest root even with --oneshot, so builders are single-use: the server destroys the VM after publication instead of returning it to a warm pool. The former Docker/static-builder execution path has been removed. Profiles, repository commits, image generations and package sets are resolved from an operator-owned immutable catalog. Packer/Catalyst outputs remain candidate-only until their documented smoke, evidence, promotion and rollback gates pass.

The current trusted-alpha boundary includes strict request/config validation, server-owned catalog resolution, immutable repository revisions, per-attempt worker mTLS identity, VM-level egress default-deny, quarantine/promotion separation, an isolated outbound-pull signer, exact-issuer OIDC verification and project roles (viewer, developer, maintainer, owner). Job queries, mutations, overview statistics and SSE streams are project-scoped; mutable username/email/token group claims never grant authorization. Disposable builders open no inbound build API: after transient SSH bootstrap succeeds, the target PVE VM is switched to policy_in=DROP, all of that VM's inbound allow rules are removed, and both options and rule-list readbacks are verified. Cluster/Node policy and other VMs are not changed. Builders emit only unsigned GPKG files; neither builder nor server mounts the private release key. Project policy serializes submissions and scheduler claims in PostgreSQL: queued, active and UTC-day limits cannot be oversubscribed by multiple replicas, and suspension stops new submissions, retries and claims. Each claim reserves its catalog maximum runtime and estimated cloud cost, terminal settlement charges actual wall time, and repeated failed/expired attempts trigger a separately auditable time-bounded suspension. High-risk administrator writes require fresh OIDC authentication or an independent legacy step-up key; OIDC sessions can be listed, individually revoked, or revoked across all replicas. A public community service still needs production identity-provider callback validation, a Vault HA/unseal/backup runbook, and a reviewed persistent-executor PVE image for the live actuator Gate. Schema v24 derives provider/zone/architecture/profile/image capacity pools, reports their demand, and persists fenced action/instance ownership, heartbeat, drain and deletion state. Heterogeneous executors use the same exact pool/capability routing, and missing labels fail closed. The existing disposable job-builder template is not accepted as an autoscaled persistent executor.

The state platform supports multiple control-plane replicas through PostgreSQL FOR UPDATE SKIP LOCKED claims, short leases and fencing tokens. Non-secret cloud settings are versioned and audited in PostgreSQL; credential values are rejected by the shared settings API and must be injected through environment or a deployment secret provider. A pre-apply resource manifest plus provider-native absence check prevents an asynchronous PVE clone from being orphaned when Terraform is killed before writing state. Redis remains a disposable acceleration layer; build correctness and publication do not depend on it.

Never commit API tokens, PVE credentials, private signing keys or production logs containing secrets.

Documentation

• Using the binhost and requesting builds
• Federated identity and project RBAC
• Scheduler fairness and autoscaling
• Authentik, Google, GitHub, and generic OIDC providers
• Policy-validated Portage configuration
• PVE native Gentoo deployment and testing
• Profiles and immutable build catalog
• Offline Packer/Catalyst image factory
• Desktop E2E
• Roadmap, security architecture and release gates

Useful development checks:

go test ./...
go test -race ./internal/builder ./internal/server
go vet ./...

License

MIT