Skip to content

Pre-pivot init misresolves Ubuntu /var/run symlink #202

Description

@ethe

Affected component

Firecracker runtime

AgentENV version

  • Version or commit: 547c1a8a515382cfc2ba02cb8aeb1bd134b7327f (the same line is present on current main)
  • Local modifications: none

Host environment

  • OS: EKS node based on Ubuntu
  • Architecture: x86_64
  • Bare metal, VM, or cloud: AWS bare-metal node running Firecracker
  • Guest image: official ubuntu:24.04 OCI rootfs
  • Guest kernel: Linux 6.1.175

Operation and configuration

# Create a cold sandbox using the official Ubuntu 24.04 OCI rootfs.
POST /sandboxes-cold
{"image":"ubuntu:24.04@sha256:...","secure":true,"autoPause":false}

Steps to reproduce

  1. Build the AgentENV tools drive from the affected commit.
  2. Cold-start a sandbox from the official Ubuntu 24.04 rootfs.
  3. Inspect the sandbox's Firecracker serial log.
  4. Observe the pre-pivot mkdir error for /mnt/user/var/run.

Ubuntu supplies /var/run as an absolute symlink to /run. Before pivot_root, mkdir -p /mnt/user/var/run follows that symlink against the tools-drive root rather than the mounted user root.

Expected behavior

The pre-pivot bootstrap prepares only real mount targets and starts an official Ubuntu rootfs without emitting a filesystem error. /var/run can be handled after pivot_root, where its absolute /run target resolves inside the user rootfs.

Actual behavior

The pre-pivot directory preparation includes /mnt/user/var/run and emits:

mkdir: can't create directory '/mnt/user/var/run': No such file or directory

The init script intentionally does not use set -e, so the observed sandbox continued booting and envd became reachable. This report does not claim the warning is fatal. It is nevertheless a deterministic compatibility error, makes serial diagnostics misleading, and means that one member of the grouped directory preparation did not do what its comment says.

Logs and diagnostics

[    0.347784] Run /init as init process
[    0.351834] EXT4-fs (vdb): mounted filesystem without journal. Quota mode: none.
mkdir: can't create directory '/mnt/user/var/run': No such file or directory
/bin/sh: 0: can't access tty; job control turned off
#

Relevant source:

$BB mkdir -p \
    /mnt/user/proc \
    /mnt/user/dev \
    /mnt/user/sys \
    /mnt/user/run \
    /mnt/user/tmp \
    /mnt/user/var/run \
    /mnt/user/var/log/agentenv \
    /mnt/user/agentenv

A minimal fix is to omit /mnt/user/var/run from the pre-pivot list. pivot-init already runs mkdir -p /var/run after the root switch, where Ubuntu's /var/run -> /run resolves correctly.

Reproduction frequency

Always

Regression information

  • Last known good: unknown
  • First known bad: 547c1a8a515382cfc2ba02cb8aeb1bd134b7327f or earlier

Additional context

The same pre-pivot pattern may affect other distro rootfs images that use an absolute /var/run -> /run symlink.

Pre-submission checklist

  • I searched existing open and closed issues and did not find a duplicate.
  • I provided a minimal reproduction.
  • I removed credentials, tokens, private image references, and other sensitive information.
  • This is not a security vulnerability.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions