arm+mmap+prot_exec crash/workaround writeup

[emetsger]

Howdy team, today I went through a troublehooting process whereby I was using an agentfs FUSE filesystem as a chroot jail.

The interesting thing (heh) is that I ran into a few issues, which (happily) resulted in a workaround. The issue is not agentfs specific, but users of agentfs may encounter this behavior and want a workaround.

Claude helped me with this writeup, and I wonder if you would consider hosting it in your project somewhere. If yes, I can open a PR, just LMK what your preference is for a file name and location!

---

ARM64 Executable Memory Limitation with FUSE

Summary

When running applications that require executable memory mappings (mmap with PROT_EXEC) on ARM64 systems, files stored on an agentfs FUSE filesystem may cause SIGILL (illegal instruction) crashes. This affects any application that memory-maps executable
content, including:

• Native addons (.node, .so files)
• JIT compilers (Node.js/V8, Python, Java, etc.)
• Any dynamically loaded executable code

Scope and Affected Configurations

• Affected: ARM64 systems (Apple Silicon, AWS Graviton, Ampere, etc.)

• Not affected: x86_64 systems (unified cache architecture)

• Applies to: All FUSE filesystems, not just agentfs

• Architecture: ARM64 / Apple Silicon (M1/M2/M3)

• Environment: Linux containers (Docker, Rancher, etc.) on macOS hosts

• Use case: Running applications inside a chroot jail on an agentfs mount

Symptoms

• SIGILL (signal 4) when an application attempts to execute memory-mapped code
• Crashes occur when executables or shared libraries reside on the FUSE filesystem
• The same application works correctly on native filesystems (ext4, tmpfs, etc.)

Example crash sequence:
openat("/tmp/.7fd3d75f1ff3f8ef-00000000.node", ...)
mmap(..., PROT_READ|PROT_EXEC, ...) # Map file as executable
Crash: SIGILL when jumping into mapped code

Verification

To determine if you're affected by this issue:

1. Check architecture

uname -m                                                                                                                                                                                                                                                               
# If output is "aarch64" or "arm64", you may be affected                                                                                                                                                                                                               
# If output is "x86_64", you are not affected    

2. Trace the crash

  # Run your application under strace to capture the crash context                                                                                                                                                                                                       
  strace -f your-app 2>&1 | grep -B5 SIGILL                                                                                                                                                                                                                              

Look for a pattern like:

  openat(AT_FDCWD, "/tmp/something.node", O_RDONLY) = 3                                                                                                                                                                                                                  
  mmap(NULL, 12345, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x7f...                                                                                                                                                                                                    
  --- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, ...} ---                                                                                                                                                                                                              

The PROT_EXEC flag combined with a file path on your FUSE mount indicates this issue.

3. Confirm the file is on FUSE

  # Check the filesystem type for the crashed file's location                                                                                                                                                                                                            
  df -T /path/to/file                                                                                                                                                                                                                                                    
  # or                                                                                                                                                                                                                                                                   
  findmnt /path/to/file                                                                                                                                                                                                                                                  

If the filesystem type is fuse or fuse.agentfs, the file is on a FUSE mount.

4. Test with tmpfs

Move the problematic file (or mount tmpfs over its directory) and retry:

  # Mount tmpfs over /tmp                                                                                                                                                                                                                                                
  mount -t tmpfs tmpfs /path/to/chroot/tmp                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                         
  # Retry the application                                                                                                                                                                                                                                                
  your-app                                                                                                                                                                                                                                                               

If the crash disappears, this confirms the FUSE + ARM64 + PROT_EXEC issue.

Root Cause

ARM64 processors have separate instruction and data caches (I-cache and D-cache). When executable pages are loaded from a FUSE filesystem—whether from a pre-compiled binary or JIT-generated code—the kernel's FUSE page cache may not issue the necessary ARM64 cache
maintenance instructions (ic ivau, dsb ish, isb) before the CPU attempts to execute from those pages.

When a page fault occurs on an executable mapping backed by FUSE:

1. The kernel requests the page data from the FUSE daemon
2. Data is read into the page cache via the D-cache
3. The page is mapped into the process's address space as executable
4. The I-cache is not invalidated for this address range
5. The CPU attempts to execute, but the I-cache contains stale/invalid data
6. Result: SIGILL

This is a kernel-level limitation affecting all FUSE filesystems on ARM64, not specific to agentfs.

Workaround

Mount a tmpfs filesystem over directories that contain executable content. Common directories that need this treatment:

• /tmp - Used by runtimes for temporary native modules and JIT output
• /var/tmp - Alternative temp directory
• Application-specific cache directories

Why tmpfs Works

tmpfs is implemented entirely within the kernel and properly handles ARM64 cache coherency for executable mappings. When pages are faulted in from tmpfs, the kernel issues the appropriate cache maintenance instructions before allowing execution.

Performance Considerations

Using tmpfs for /tmp is often beneficial regardless of this issue:

• tmpfs is RAM-backed, providing very fast I/O for temporary files
• Reduces I/O overhead for frequently-accessed temp files
• Contents are automatically cleaned on unmount

The trade-off is that tmpfs uses system memory. Ensure adequate RAM is available, or set size limits:

  mount -t tmpfs -o size=512M tmpfs /mnt/chroot/tmp

Example: Chroot Setup

  # Create the chroot environment on agentfs                                                                                                                                                                                                                             
  agentfs init my-chroot                                                                                                                                                                                                                                                 
  agentfs mount my-chroot /mnt/chroot                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                         
  # Set up minimal root filesystem                                                                                                                                                                                                                                       
  debootstrap --variant=minbase bookworm /mnt/chroot http://deb.debian.org/debian                                                                                                                                                                                        
                                                                                                                                                                                                                                                                         
  # Mount tmpfs over /tmp before entering chroot                                                                                                                                                                                                                         
  mount -t tmpfs tmpfs /mnt/chroot/tmp                                                                                                                                                                                                                                   
  mount -t tmpfs tmpfs /mnt/chroot/var/tmp                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                         
  # Enter the chroot                                                                                                                                                                                                                                                     
  chroot /mnt/chroot /bin/bash                                                                                                                                                                                                                                           

Example: Docker/Container Setup

If running inside a container that mounts an agentfs filesystem:

  # Inside the container, after agentfs is mounted                                                                                                                                                                                                                       
  mount -t tmpfs tmpfs /path/to/agentfs-mount/tmp