Support custom/private OCI registry for systemd-nspawn rootfs image (runtimeImage)

[cello86]

Description

When deploying aks-flex-node in highly secure, air-gapped, or strictly regulated enterprise environments (such as banking DMZs), direct outbound connection to ghcr.io is forbidden by corporate firewalls.

While aks-flex-node allows configuration via a JSON file, the agent currently ignores any attempts to override the default OCI rootfs image location (ghcr.io/azure/agent-ubuntu2404) when specified inside the configuration file or via command-line flags.

Steps to Reproduce

Configure /etc/aks-flex-node/config.json

Run the bootstrap command:
aks-flex-node start --config /etc/aks-flex-node/config.json

Expected Behavior

The agent should parse the custom registry URL and pull the agent-ubuntu2404 image from the private mirror hosted on the internal network.

Actual Behavior

The agent completely ignores the custom registry settings and falls back to hardcoded defaults, attempting a direct connection to GitHub Packages:

level=INFO msg="no OCI image configured, using default" image=ghcr.io/azure/agent-ubuntu2404:v20260427
level=INFO msg="pulling OCI image" image=ghcr.io/azure/agent-ubuntu2404:v20260427
level=ERROR msg=failed task=oci-download-rootfs duration=7.37ms status=failed error="pull image ghcr.io/azure/agent-ubuntu2404:v20260427: failed to perform \"FetchReference\" on source: Get \"[https://ghcr.io/v2/azure/agent-ubuntu2404/manifests/v20260427](https://ghcr.io/v2/azure/agent-ubuntu2404/manifests/v20260427)\": Forbidden"

Environment

AKS Flex Node Version: v0.1.3 (Git Commit: 79c5487)

OS: Ubuntu 24.10 (Oracular Oriole)

Environment: On-premises with squid proxy and nexus proxy for docker images.

[bcho]

Thanks for reporting this, we are actively working on adding support for offline bootstraping in unbounded side. Tracking issue: Azure/unbounded#319

[cello86]

Hi @cello86. Thanks for the quick response. Do you plan to release these changes in these days?

Thanks,
Marcello

[bcho]

@cello86 my current estimation is next week (this Friday and weekend is public holiday in US), but I will post updates here for the progress.

[cello86]

@bcho I tried to download before the startup the requirements and the container image worked fine but the binaries not. I use this workaround:

Docker

On different node:

# docker pull docker.registry/azure/agent-ubuntu2404:v20260427
# docker create --name temp-agent docker.registry/azure/agent-ubuntu2404:v20260427
# docker export temp-agent | gzip > agent-rootfs.tar.gz
# docker rm temp-agent
# docker image prune -a

Copied the file to vm flex node and execute these commands:

# sudo mkdir -p /var/lib/machines/kube1
# sudo tar -C /var/lib/machines/kube1 -xzf /tmp/agent-rootfs.tar.gz
# sudo rm -f /tmp/agent-rootfs.tar.gz

Binaries

# sudo apt-get install -y nginx
# sudo apt-get update -y
# sudo vi /etc/nginx/sites-available/default
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;

    server_name github.com dl.k8s.io;

    ssl_certificate /etc/nginx/local-fake.pem;
    ssl_certificate_key /etc/nginx/local-fake.key;

    location / {
        proxy_pass https://nexus.local/nexus/repository/raw/;
        proxy_set_header Host nexus.local;
        proxy_ssl_verify off;
    }
}
# sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048   -keyout /etc/nginx/local-fake.key   -out /etc/nginx/local-fake.pem   -subj "/CN=test-local-rewrite"   -addext "subjectAltName = DNS:github.com, DNS:dl.k8s.io, DNS:localhost, IP:127.0.0.1"
# sudo cp /etc/nginx/local-fake.pem /usr/local/share/ca-certificates/local-fake.crt
# sudo update-ca-certificates
# sudo systemctl restart nginx
# sudo vi /etc/hosts
127.0.0.1   github.com
127.0.0.1   dl.k8s.io

Agent bootstrap:

# export NO_PROXY="127.0.0.1,localhost,10.0.0.0/8,172.16.0.0/12,aks-xxxx.privatelink.westeurope.azmk8s.io,ghcr.io,github.com,dl.k8s.io"
# exporg HTTPS_PROXY="http://proxy.local:3128"
# export HTTP_PROXY="http://proxy.local:3128"
# export no_proxy="127.0.0.1,localhost,10.0.0.0/8,172.16.0.0/12,aks-xxx.privatelink.westeurope.azmk8s.io,ghcr.io,github.com,dl.k8s.io"
# export https_proxy="http://proxy.local:3128"
# export http_proxy="http://proxy.local:3128"
# export GODEBUG=http2client=0
# aks-flex-node start --config /etc/aks-flex-node/config.json

[bcho]

@cello86 thanks for testing this out -- yes, in the offline mode we will include both the binary and oci image so you can host them locally

[cello86]

I added new config for registry images proxying after node startup:

# sudo vi /etc/hosts
127.0.0.1   github.com
127.0.0.1   dl.k8s.io
127.0.0.1   mcr.microsoft.com
127.0.0.1   ghcr.io
# sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048   -keyout /etc/nginx/local-fake.key   -out /etc/nginx/local-fake.pem   -subj "/CN=test-local-rewrite"   -addext "subjectAltName = DNS:github.com, DNS:dl.k8s.io, DNS:mcr.microsoft.com, DNS:ghcr.io, DNS:*.azurecr.io, DNS:localhost, IP:127.0.0.1"
# sudo cp /etc/nginx/local-fake.pem /usr/local/share/ca-certificates/local-fake.crt
# sudo cp /etc/nginx/local-fake.pem /var/lib/machines/kube1/usr/local/share/ca-certificates/local-fake.crt
# sudo update-ca-certificates
# sudo machinectl shell kube1 /usr/sbin/update-ca-certificates
# vim /etc/nginx/sites-available/default
server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;

    server_name github.com dl.k8s.io mcr.microsoft.com ghcr.io *.azurecr.io;

    ssl_certificate /etc/nginx/local-fake.pem;
    ssl_certificate_key /etc/nginx/local-fake.key;

    location /v2/ {
        proxy_pass https://cvs.fineco.it:8187/v2/;
        proxy_set_header Host cvs.fineco.it;
        proxy_ssl_verify off;

        proxy_connect_timeout 300;
        proxy_send_timeout 300;
        proxy_read_timeout 300;
    }

    location / {
        proxy_pass https://cvs.fineco.it/nexus/repository/fineco-raw-group/;
        proxy_set_header Host cvs.fineco.it;
        proxy_ssl_verify off;
    }
}
# sudo systemctl restart nginx
# sudo machinectl shell kube1 /bin/systemctl restart containerd