[Request for Feedback] Package Source Policy Design

[sozercan]

Package Source Policy Design

Request for Feedback

Feature Summary: This design proposes a "Package Source Policy" for Copa, allowing users to configure custom package repositories (mirrors) for apt, rpm, and apk. This is critical for air-gapped environments or organizations that require authenticated private mirrors. The policy is defined in a file and loaded via an environment variable, similar to BuildKit's source policies.

We are actively seeking feedback on the following design choices:

1. Authentication Methods

   • Current Plan: Support Basic Auth (username/password) and CA bundles initially.
   • Question: Does your environment require more complex authentication like Mutual TLS (client certificates) or custom GPG keys for your package mirrors?

2. Fallback Behavior

   • Current Plan: Fail the patch operation if the configured private mirror is unreachable.
   • Question: Would you prefer Copa to attempt falling back to the original public repositories if your private mirror fails, or is "fail-secure" (aborting) the preferred behavior?

3. Persistence of Changes

   • Current Plan: Repository configuration changes are temporary and only exist during the patching process. The final image retains its original repository configuration.
   • Question: Do you need these repository changes (e.g., sources.list pointing to your internal mirror) to persist in the final patched image for future use (e.g., for runtime updates)?

4. Secret Handling

   • Current Plan: Use BuildKit secrets (via --secret) to securely pass credentials.
   • Question: Is this workflow compatible with your CI/CD pipelines? Do you have constraints that would require passing credentials differently (e.g., inline in the policy, though less secure)?

5. Policy Complexity

   • Current Plan: The first matching rule in the policy is applied to a given package manager.
   • Question: Do you have scenarios where you need to merge multiple rules (e.g., "apply global company mirrors" AND "apply team-specific repos"), or is a single override sufficient?

6. Validation Strictness

   • Current Plan: Copa will log a warning if a policy rule is defined but never used (e.g., an apt rule when patching an rpm image).
   • Question: Should this be a hard error to prevent misconfiguration, or is a warning sufficient?

Overview

Users need Copa to operate in environments where public package mirrors are inaccessible or disallowed. Today, Copa assumes each target image already contains the desired repositories and only exposes BuildKit source policies for swapping tooling images. This design adds a first-class package source policy that mirrors BuildKit's policy experience, enabling Copa to rewrite apt/dnf/apk repository definitions and inject the credentials or trust material required to reach enterprise mirrors.

Goals

• Allow users to configure alternate package repositories for any supported OS package manager (apt/dpkg, rpm-based CLIs, apk) without modifying the base image.
• Reuse a familiar delivery mechanism by loading the policy from an environment variable (EXPERIMENTAL_COPA_PKG_SOURCE_POLICY) similar to EXPERIMENTAL_BUILDKIT_SOURCE_POLICY.
• Support mirrored, air-gapped, and authenticated repositories by providing hooks for credentials, client certificates, CA bundles, and signing keys.
• Ensure both standard and distroless tooling flows honor the same policy before running package manager commands.

Non-goals

• Automating mirror synchronization or validating mirror freshness.
• Managing language/library package manager repositories (Python, Node) in this iteration.
• Replacing BuildKit source policies for tooling images; the new policy focuses on OS package managers.

Constraints & Assumptions

• Copa continues to run inside BuildKit; any credential material must be mounted via BuildKit secrets or ephemeral files and must not persist in the final layer.
• Policy parsing must be deterministic and reject unsupported selectors early to avoid partially applied repo rewrites.
• Repo rewrites must preserve mandatory security configurations (e.g., signed-by for apt, gpgcheck=1 for rpm) unless the user explicitly overrides them.

High-level Architecture

1. Policy Loading: On startup, Copa reads EXPERIMENTAL_COPA_PKG_SOURCE_POLICY (path to JSON/YAML). The CLI layer (pkg/patch/cmd.go) parses the file into a typed struct and stores it in pkg/types.Options.
2. Propagation: pkg/patch/core.go threads the parsed policy into pkg/pkgmgr.GetPackageManager. Each concrete manager retains a reference for later use.
3. Selector Matching: Before running apt-get, dnf, or apk, the manager evaluates policy rules whose selectors match the current OS type/version and package manager name.
4. Repo Rewrite: Matching rules produce in-memory repo files (/etc/apt/sources.list, /etc/yum.repos.d/*.repo, /etc/apk/repositories). Managers inject these files into the image/tooling filesystem via LLB File ops and ensure relevant commands reference them (e.g., APT_CONFIG, --config, -c switches).
5. Credential & Trust Injection: When policy rules reference secrets, Copa mounts them as BuildKit secrets or temp files, wiring paths into repo definitions (sslclientcert, sslclientkey, ca_bundle, signed-by).
6. Execution: Package commands run with rewritten configs, using the same policy for both direct and distroless flows (including helper scripts like scripts/apt_get_download.sh).

Policy Schema Sketch

The schema intentionally resembles BuildKit's source policy structure. While a single policy file can contain rules for multiple operating systems (useful for a shared policy across an organization), here are examples for specific package managers.

APT Example (Debian/Ubuntu)

{
  "rules": [
    {
      "action": "REWRITE_REPO",
      "selector": {
        "os": {
          "type": "debian",
          "version": "12.*"
        },
        "pkgManager": "apt"
      },
      "updates": {
        "repositories": [
          {
            "name": "enterprise-mirror",
            "uri": "https://mirror.example.com/ubuntu/",
            "suite": "bookworm",
            "components": ["main", "universe"],
            "signedBySecret": "gpg-key-ent-mirror"
          }
        ],
        "caBundleSecret": "mirror-ca.pem",
        "auth": {
          "type": "basic",
          "secret": "mirror-basic-auth"
        }
      }
    }
  ]
}

RPM Example (RHEL/CentOS/Mariner)

{
  "rules": [
    {
      "action": "REWRITE_REPO",
      "selector": {
        "os": {
          "type": "rhel",
          "version": "9.*"
        },
        "pkgManager": "rpm"
      },
      "updates": {
        "repositories": [
          {
            "name": "enterprise-rpm-mirror",
            "baseurl": "https://mirror.example.com/rhel/9/baseos/x86_64/",
            "gpgcheck": true,
            "sslverify": true
          }
        ],
        "caBundleSecret": "mirror-ca.pem"
      }
    }
  ]
}

APK Example (Alpine)

{
  "rules": [
    {
      "action": "REWRITE_REPO",
      "selector": {
        "os": {
          "type": "alpine",
          "version": "3.18"
        },
        "pkgManager": "apk"
      },
      "updates": {
        "repositories": [
          {
            "uri": "https://mirror.example.com/alpine/v3.18/main",
            "mirrorLabel": "@enterprise"
          },
          {
            "uri": "https://mirror.example.com/alpine/v3.18/community"
          }
        ],
        "caBundleSecret": "mirror-ca.pem"
      }
    }
  ]
}

Schema Elements

• rules[]: ordered; first-match wins.
• action: initial value REWRITE_REPO. Future actions could cover DENY or APPEND.
• selector: matches on os.type, optional os.version glob, pkgManager (apt, rpm, apk), and optional image identifier.
• updates.repositories[]:
  • APT: uri, suite, components, optional options map (e.g., arch=amd64, signed-by).
  • RPM: name, baseurl, gpgcheck, sslverify, optional authRef, caBundleSecret.
  • APK: uri, optional pinning, mirrorLabel.
• auth: hints for HTTP auth (basic/bearer header). Value references BuildKit secret IDs, never inline credentials.
• caBundleSecret: BuildKit secret containing PEM bundle. Managers mount and point package managers to it.
• clientCertSecret / clientKeySecret: optional for mutual TLS.

Detailed Flow

1. Parsing & Validation

   • CLI loads file, unmarshals JSON/YAML into internal structs.
   • Validation ensures required keys per manager exist and selectors do not mix incompatible fields (e.g., apt-specific repo fields in rpm rule).
   • Provide meaningful errors describing rule index and offending field.

2. CLI Updates

   • Add --secret flag to copa patch command (e.g., --secret id=my-secret,src=./token.txt) to allow users to pass credentials referenced in the policy.
   • Configure the BuildKit session (pkg/buildkit/client.go) to use a secret provider populated by these flags.
   • Alternative: If/when Copa migrates to a pure BuildKit frontend implementation, secrets would be passed via standard buildctl --secret flags, making the Copa-specific flag redundant.

3. Manager Injection

   • pkg/pkgmgr/pkgmgr.go extends manager constructors to accept a *pkgpolicy.Policy (new package).
   • Each manager exposes helper applyPolicy(ctx, state, policyRule) that returns an llb.State with repo files + secret mounts applied.
   • Distroless flows reuse same helper before invoking helper scripts or tdnf installroot.

4. Repository Materialization

   • APT: generate /etc/apt/sources.list.d/copa-policy.list and optionally custom /etc/apt/auth.conf.d/*.conf. If policy sets replaceExisting=true, managers remove default list files.
   • RPM: create /etc/yum.repos.d/copa-policy.repo with [repo] stanzas. Support toggling enabled, gpgkey, sslcacert, sslclientcert, sslclientkey.
   • APK: rewrite /etc/apk/repositories. Add @mirror labels if provided.

5. Secret & CA Handling

   • Introduce helper pkg/pkgmgr/secretfs.go to map secret IDs to BuildKit llb.Secret. Provide mountSecret(state, destPath, secretID, mode) utility.
   • Support secret types:
     • Basic/Bearer tokens: mounted to /run/copa/policy/auth/<rule>/token and referenced via header = "Authorization: Bearer $token" or inserted into repo URL as https://user:pass@host for apt (only if user opts in).
     • TLS client cert/key: written to /run/copa/policy/tls/{cert,key} and referenced in repo config.
     • CA bundles: appended or replace /etc/ssl/certs/ca-certificates.crt depending on policy flag (caMode: append|replace).

6. Script Adjustments

   • Update pkg/pkgmgr/scripts/apt_get_download.sh to honor environment variables pointing to rewritten lists/auth files; leverage template substitution during script injection.
   • For rpm distroless path, ensure tdnf/dnf commands pass --disablerepo='*' --enablerepo='copa-policy-*' when replaceExisting is set.

Testing Strategy

• Unit Tests: new package for policy parsing/validation with table-driven tests; per-manager tests verifying generated repo files and command args when policy applied.
• Integration Tests: extend single-arch integration tests to simulate patching against a local mirror (e.g., using a lightweight HTTP server) with policy-specified repos.
• Regression Coverage: ensure existing behavior unaffected when env var unset (no policy loaded).

Documentation & Tooling

• Add website/docs/faq.md entry detailing EXPERIMENTAL_COPA_PKG_SOURCE_POLICY, schema, and security guidance.
• Provide sample policy files (deb, rpm, apk) under docs/examples/.
• Mention requirement for BuildKit secrets in README + website.

Rollout & Compatibility

• Feature hidden behind env var; default path remains unchanged.
• Emit warnings when policy references unsupported OS or pkg manager to aid troubleshooting.

Future Work

• Distroless/Tooling Support: Extending policy support to distroless images (which run package managers in a separate tooling container) requires complex config mounting and flag overrides.
• Architecture Selectors: Adding arch support to selectors for multi-arch policy definitions.
• Proxy Configuration: Adding per-repository proxy settings (currently relying on global HTTP_PROXY env vars).
• Repository Priorities: Exposing advanced settings like APT preferences or YUM priorities.
• Custom GPG Keys: Support for importing custom GPG keys for repository signing verification (currently relies on keys already present in the image).