Write the compiled kernel cache atomically to prevent crashes when starting renderers concurrently.

[duburcqa]

Problem

Starting several batched renderers concurrently on the same host (e.g. pytest -n <N> GPU test runs, or multiple training workers) intermittently aborts with SIGABRT (exit 134) inside cuModuleLoadData, once the compiled kernel cache exists on disk. The failure is non-deterministic and disappears when the run is serialized.

Root cause

The megakernel and BVH kernel caches are written in place with a plain std::ofstream at a shared path ($MADRONA_ROOT_CACHE_DIR/kernel_cache, $HOME/.cache/madrona by default). When one process is part-way through writing the cubin, a second process that finds the cache present (kernelCacheNeedsRecompile only checks existence + last_write_time) opens it and feeds a truncated, half-written buffer to cuModuleLoadData, which aborts.

This was confirmed deterministically: truncating the cache file to a partial cubin reproduces the exact SIGABRT exit 134.

Fix

Write each cache to a per-process temporary file (<cache_path>.tmp.<pid>) and std::filesystem::rename it into place. POSIX rename is atomic, so a reader observes either the previous complete file or the new complete file, never a partial one. Concurrent compilers each write their own temp and the final rename wins with a valid, self-consistent cubin.

Applied to both the megakernel cache write and the BVH kernel cache write.

Validation

• Root cause and failure mode reproduced deterministically (truncated cache -> exit 134).
• Atomic-rename write makes torn reads unobservable by construction.