OverlayFS for rootfs

[snedosiekov]

Hi Tom @tdewey-rpi

Coul you suggest best approach to implement overlayfs root scheme in security boot?
In case of native provision I got a success with this approach but in case if fde mode I fake to boot.
Because I cannot modify kernel cmdline in fde (or dont know how to do this) I have renamed /sbin/init to /sbin/real_init and using custom script us /sbin/init
Also I commented fstab parsing and set rootDev and etc variables directly (rootDev to /dev/mapper/cryptfs)
trying to mount overlayfs witn ro option for rootfs as lower failed, probably because secure initramfs image have mounted it as rw, so I removed ro and init have started but process fail on starting systemd-remount-fs service

I would be grateful for your advice.

WBR
Serhii

[tdewey-rpi]

This is exactly that the bootfs-mounted and rootfs-mounted customisation hooks are designed for.

I asked Claude, and it suggested you used something like this for your FDE-only customisation hooks:

bootfs-mounted:

# File: /etc/rpi-sb-provisioner/scripts/fde-provisioner-bootfs-mounted.sh
# 
# This hook runs during FDE provisioning after bootfs is mounted
# Arguments:
# $1 - Path to mounted boot image
# $2 - Path to mounted rootfs image

set -e

BOOT_MOUNT="$1"
ROOTFS_MOUNT="$2"
INITRAMFS="${BOOT_MOUNT}/initramfs8"
WORK_DIR=$(mktemp -d)

echo "=== OverlayFS Customisation: Modifying initramfs for overlayfs support ==="

# Check if initramfs exists
if [ ! -f "${INITRAMFS}" ]; then
    echo "ERROR: initramfs not found at ${INITRAMFS}"
    exit 1
fi

# Extract initramfs
echo "Extracting initramfs..."
cd "${WORK_DIR}"
zstd -d "${INITRAMFS}" -o initramfs.cpio
mkdir initramfs
cd initramfs
cpio -id < ../initramfs.cpio
rm ../initramfs.cpio

# Add overlay kernel module from rootfs
echo "Adding overlay kernel module..."
cd "${ROOTFS_MOUNT}"
find usr/lib/modules -name 'overlay.*' -exec cp -r --parents "{}" "${WORK_DIR}/initramfs/" \;

# Update depmod
echo "Updating module dependencies..."
cd "${WORK_DIR}/initramfs"
find usr/lib/modules -mindepth 1 -maxdepth 1 -type d -exec basename {} \; | while read -r kernel; do
    depmod --basedir "${WORK_DIR}/initramfs" "${kernel}"
done

# Create overlayfs setup script
echo "Creating overlayfs setup script..."
cat > "${WORK_DIR}/initramfs/usr/bin/setup_overlayfs.sh" << 'EOF_OVERLAY'
#!/bin/sh
# OverlayFS setup for read-only encrypted root with writable overlay

CRYPTFS_MOUNT="$1"

echo "=== Setting up OverlayFS ==="

# Load overlay module
modprobe overlay
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to load overlay module"
    exit 1
fi

# Create directory structure
mkdir -p /overlay-lower /overlay-upper /overlay-work /newroot

# Mount decrypted root as read-only lower layer
echo "Mounting encrypted root as read-only lower layer..."
mount -o ro "${CRYPTFS_MOUNT}" /overlay-lower
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to mount cryptfs as lower layer"
    exit 1
fi

# Create tmpfs for upper layer (writable)
echo "Creating tmpfs for writable upper layer..."
mount -t tmpfs -o size=1G tmpfs /overlay-upper
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to create tmpfs"
    umount /overlay-lower
    exit 1
fi

# Create required directories in tmpfs
mkdir -p /overlay-upper/upper /overlay-upper/work

# Mount overlayfs
echo "Mounting overlayfs..."
mount -t overlay overlay \
    -o lowerdir=/overlay-lower,upperdir=/overlay-upper/upper,workdir=/overlay-upper/work \
    /newroot

if [ $? -ne 0 ]; then
    echo "ERROR: Failed to mount overlayfs"
    umount /overlay-upper
    umount /overlay-lower
    exit 1
fi

# Move overlay mounts into new root to persist after switch_root
mkdir -p /newroot/.overlay
mount --move /overlay-lower /newroot/.overlay/lower
mount --move /overlay-upper /newroot/.overlay/upper

echo "OverlayFS setup complete"
echo "/newroot"
exit 0
EOF_OVERLAY

chmod +x "${WORK_DIR}/initramfs/usr/bin/setup_overlayfs.sh"

# Modify init_cryptroot.sh to use overlayfs
echo "Modifying init_cryptroot.sh to integrate overlayfs..."

# The pi-gen-micro init_cryptroot.sh mounts /dev/mapper/cryptfs to /cryptroot
# then does: exec switch_root /cryptroot /sbin/init
# We need to intercept between mount and switch_root

cat >> "${WORK_DIR}/initramfs/usr/bin/init_cryptroot.sh" << 'EOF_INIT_MOD'

# ========== OverlayFS Integration ==========
# Setup overlayfs on top of encrypted root
echo "Attempting to setup overlayfs..."
if [ -x /usr/bin/setup_overlayfs.sh ]; then
    OVERLAY_ROOT=$(/usr/bin/setup_overlayfs.sh /dev/mapper/cryptfs)
    if [ $? -eq 0 ] && [ -n "$OVERLAY_ROOT" ] && [ -d "$OVERLAY_ROOT" ]; then
        echo "Using overlayfs root: $OVERLAY_ROOT"
        # Replace the switch_root target
        exec switch_root "$OVERLAY_ROOT" /sbin/init "$@"
    else
        echo "WARNING: OverlayFS setup failed, falling back to direct mount"
    fi
else
    echo "WARNING: OverlayFS setup script not found"
fi
# If we get here, overlayfs failed - use original behavior
# The original switch_root command should still execute below
# ========== End OverlayFS Integration ==========
EOF_INIT_MOD

# Find and modify the original switch_root line to be a fallback
# Replace "exec switch_root /cryptroot" with commented version as fallback
sed -i 's|^exec switch_root /cryptroot|# Fallback if overlayfs fails\nexec switch_root /cryptroot|' \
    "${WORK_DIR}/initramfs/usr/bin/init_cryptroot.sh"

# Repack initramfs
echo "Repacking initramfs..."
cd "${WORK_DIR}/initramfs"
find . -print0 | cpio --null -ov --format=newc > ../initramfs-new.cpio
cd "${WORK_DIR}"
zstd --rm -f -6 initramfs-new.cpio -o initramfs-new.cpio.zst

# Replace original initramfs
mv "${INITRAMFS}" "${INITRAMFS}.backup"
mv initramfs-new.cpio.zst "${INITRAMFS}"

# Cleanup
rm -rf "${WORK_DIR}"

echo "=== OverlayFS initramfs customisation complete ==="
exit 0

And then for rootfs-customisation:

#!/bin/bash
# File: /etc/rpi-sb-provisioner/scripts/fde-provisioner-rootfs-mounted.sh
#
# This hook runs during FDE provisioning after rootfs is mounted
# Arguments:
# $1 - Path to mounted boot image
# $2 - Path to mounted rootfs image

set -e

BOOT_MOUNT="$1"
ROOTFS_MOUNT="$2"

echo "=== OverlayFS Customisation: Preparing rootfs for overlayfs ==="

# 1. Disable systemd-remount-fs.service
# This service tries to remount root based on fstab, which conflicts with overlayfs
echo "Disabling systemd-remount-fs.service..."
mkdir -p "${ROOTFS_MOUNT}/etc/systemd/system"
ln -sf /dev/null "${ROOTFS_MOUNT}/etc/systemd/system/systemd-remount-fs.service"

# 2. Update fstab to reflect overlayfs
echo "Updating fstab for overlayfs..."
# Comment out the original root entry
sed -i 's|^\([^#].*[[:space:]]/[[:space:]].*\)$|# \1  # Disabled for overlayfs|' \
    "${ROOTFS_MOUNT}/etc/fstab"

# Add overlayfs entry
if ! grep -q "overlay / overlay" "${ROOTFS_MOUNT}/etc/fstab"; then
    echo "# Root filesystem using overlayfs (read-only lower + tmpfs upper)" >> "${ROOTFS_MOUNT}/etc/fstab"
    echo "overlay / overlay defaults 0 0" >> "${ROOTFS_MOUNT}/etc/fstab"
fi

# 3. Ensure overlay mount point directory exists (for persistence)
mkdir -p "${ROOTFS_MOUNT}/.overlay"

# 4. Optional: Add a marker file to indicate overlayfs is enabled
cat > "${ROOTFS_MOUNT}/etc/overlayfs-enabled" << 'EOF'
This system uses OverlayFS for the root filesystem.
- Lower layer (read-only): Encrypted root partition
- Upper layer (writable): tmpfs in RAM
- All changes are lost on reboot

To persist changes, you must:
1. Disable overlayfs temporarily
2. Make changes
3. Re-enable overlayfs

Or use a persistent upper layer on a separate partition.
EOF

# 5. Optional: Create a helper script to show overlay status
cat > "${ROOTFS_MOUNT}/usr/local/bin/overlayfs-status" << 'EOF_STATUS'
#!/bin/bash
# Show OverlayFS status

echo "=== OverlayFS Status ==="
echo

if mount | grep -q "overlay on / type overlay"; then
    echo "✓ Root filesystem is mounted as overlay"
    echo
    
    # Show overlay details
    MOUNT_INFO=$(mount | grep "overlay on / type overlay")
    echo "Mount info:"
    echo "  $MOUNT_INFO"
    echo
    
    # Show layer usage
    if [ -d /.overlay/upper/upper ]; then
        UPPER_SIZE=$(du -sh /.overlay/upper/upper 2>/dev/null | cut -f1)
        echo "Upper layer (tmpfs) usage: ${UPPER_SIZE}"
        
        # Show what's been modified
        echo
        echo "Modified files/directories:"
        find /.overlay/upper/upper -mindepth 1 -maxdepth 2 2>/dev/null | head -20
    fi
else
    echo "✗ Root filesystem is NOT using overlay"
    mount | grep " on / "
fi
EOF_STATUS

chmod +x "${ROOTFS_MOUNT}/usr/local/bin/overlayfs-status"

echo "=== OverlayFS rootfs customisation complete ==="
echo "Changes made:"
echo "  - Disabled systemd-remount-fs.service"
echo "  - Updated /etc/fstab"
echo "  - Created /etc/overlayfs-enabled marker"
echo "  - Created /usr/local/bin/overlayfs-status helper"

exit 0

Of course, check those over before using them - they're AI generated. But that's the general approach I'd take.

[snedosiekov]

Thanks a lot, I will try. Also will investigate how to add third partition for owerlay upper layer and encript/decrypt it

[snedosiekov]

In case of sb mode this approache will work to? hope security bootloader did not need to be modify to support overalyfs

[tdewey-rpi]

It will, as the modifications get included in the signed boot image.

[snedosiekov]

Hello @tdewey-rpi

I have configure thous scripts and did a provision but did not observe in logs this scripts output and, also, after provisioned PRI did not observe in mount appropriate output

Looks like i miss some part of process?

[tdewey-rpi]

@roliver-rpi Can you take a look?

[snedosiekov]

hello @tdewey-rpi
Now all two scripts are running, but looks like something going wrong with initing overlay in initramfs

current version of bootfs customization

#!/bin/sh

# File: /etc/rpi-sb-provisioner/scripts/fde-provisioner-bootfs-mounted.sh
# 
# This hook runs during FDE provisioning after bootfs is mounted
# Arguments:
# $1 - Path to mounted boot image
# $2 - Path to mounted rootfs image

set -e

BOOT_MOUNT="$1"
ROOTFS_MOUNT="$2"
INITRAMFS="${BOOT_MOUNT}/initramfs8"
WORK_DIR=$(mktemp -d)

echo "=== OverlayFS Customisation: Modifying initramfs for overlayfs support ==="

# Check if initramfs exists
if [ ! -f "${INITRAMFS}" ]; then
    echo "ERROR: initramfs not found at ${INITRAMFS}"
    exit 1
fi

# Extract initramfs
echo "Extracting initramfs..."
cd "${WORK_DIR}"
zstd -d "${INITRAMFS}" -o initramfs.cpio
mkdir initramfs
cd initramfs
cpio -id < ../initramfs.cpio
rm ../initramfs.cpio

# Add overlay kernel module from rootfs
echo "Adding overlay kernel module..."
cd "${ROOTFS_MOUNT}"
find usr/lib/modules -name 'overlay.*' -exec cp -r --parents "{}" "${WORK_DIR}/initramfs/" \;

# Update depmod
echo "Updating module dependencies..."
cd "${WORK_DIR}/initramfs"
find usr/lib/modules -mindepth 1 -maxdepth 1 -type d -exec basename {} \; | while read -r kernel; do
    depmod --basedir "${WORK_DIR}/initramfs" "${kernel}"
done

# Modify init_cryptroot.sh to use overlayfs
echo "Modifying init_cryptroot.sh to integrate overlayfs..."

# The pi-gen-micro init_cryptroot.sh mounts /dev/mapper/cryptfs to /cryptroot
# then does: exec switch_root /cryptroot /sbin/init
# We need to intercept between mount and switch_root

cat >> "${WORK_DIR}/initramfs/usr/bin/init_cryptroot.sh" << 'EOF_INIT_MOD'

# ========== OverlayFS Integration ==========
# Setup overlayfs on top of encrypted root
set +e
set +x
echo "Attempting to setup overlayfs..."

CRYPTFS_MOUNT="/dev/mapper/cryptroot"
OVERLAY_ROOT="/newroot"

echo "=== Setting up OverlayFS ===" 
      
# Load overlay module
modprobe overlay
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to load overlay module"
    exit 1
fi

# Create directory structure
mkdir -p /overlay-lower /overlay-upper /newroot

# Mount decrypted root as read-only lower layer
echo "Mounting encrypted root as read-only lower layer..."
/usr/bin/busybox mount -o ro "${CRYPTFS_MOUNT}" /overlay-lower
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to mount cryptfs as lower layer"
    exit 1
fi

# Create tmpfs for upper layer (writable)
echo "Creating tmpfs for writable upper layer..."
/usr/bin/busybox mount -t tmpfs -o size=1G tmpfs /overlay-upper
if [ $? -ne 0 ]; then
    echo "ERROR: Failed to create tmpfs"
    umount /overlay-lower
    exit 1
fi


# Create required directories in tmpfs
mkdir -p /overlay-upper/upper /overlay-upper/work

# Mount overlayfs
echo "Mounting overlayfs..."
/usr/bin/busybox mount -t overlay overlay -o lowerdir=/overlay-lower,upperdir=/overlay-upper/upper,workdir=/overlay-upper/work /newroot

if [ $? -ne 0 ]; then
    echo "ERROR: Failed to mount overlayfs"
    umount /overlay-upper
    umount /overlay-lower
    exit 1
fi

# Move overlay mounts into new root to persist after switch_root
mkdir -p /newroot/.overlay
mount --move /overlay-lower /newroot/.overlay/lower
mount --move /overlay-upper /newroot/.overlay/upper

echo "OverlayFS setup complete"

echo "Using overlayfs root: $OVERLAY_ROOT" &>2
# Replace the switch-root target  

#exec switch_root "$OVERLAY_ROOT" /usr/sbin/init 
systemctl switch-root  "$OVERLAY_ROOT" /usr/sbin/init 

# If we get here, overlayfs failed - use original behavior
# The original switch-root command should still execute below

set -e
set -x
# ========== End OverlayFS Integration ==========
EOF_INIT_MOD

# Find and modify the original switch_root line to be a fallback
# Replace "exec switch_root /cryptroot" with commented version as fallback

sed -i 's|/usr/bin/busybox mount /dev/mapper/cryptroot /mnt||g' "${WORK_DIR}/initramfs/usr/bin/init_cryptroot.sh"
sed -i 's|systemctl switch-root /mnt /usr/sbin/init||g' "${WORK_DIR}/initramfs/usr/bin/init_cryptroot.sh"
sed -i '$a\systemctl switch-root /mnt /usr/sbin/init' "${WORK_DIR}/initramfs/usr/bin/init_cryptroot.sh"

# Repack initramfs
echo "Repacking initramfs..."
cd "${WORK_DIR}/initramfs"
find . -print0 | cpio --null -ov --format=newc > ../initramfs-new.cpio
cd "${WORK_DIR}"
zstd --rm -f -6 initramfs-new.cpio -o initramfs-new.cpio.zst

# Replace original initramfs
mv "${INITRAMFS}" "${INITRAMFS}.backup"
mv initramfs-new.cpio.zst "${INITRAMFS}"

# Cleanup
rm -rf "${WORK_DIR}"

echo "=== OverlayFS initramfs customisation complete ==="
exit 0

But, unfortunately, I got initramfs login prompt with no ability to login to debug.
Also I did not find how to got log messages, looks like before init_cryptroot.sh is starting screen is reseting and echo did not print nothing

Could you suggest how to login/get command prompt in initramfs stage? root did not accepted by login(