worker: fix two silent output-corruption bugs (absolute symlinks + zero-digest files) (#2346)

* Resolve absolute symlinks in output upload instead of uploading raw targets

When the worker's work directory is populated via DirectoryCache, output
paths can be absolute symlinks pointing into the cache directory
(/var/.../directory_cache/...). The previous output collection code
uploaded these as raw SymlinkInfo with absolute targets that are
meaningless on the Bazel client, causing "No such file or directory"
errors when the client materialised the action result.

Detect absolute symlinks in output paths and resolve them: upload
directory contents as Tree protos and file contents as regular files.
Relative symlinks (intentionally created by the action) are still
preserved as symlinks.

Updates upload_dir_and_symlink_test to use a relative symlink (the
previous /dev/null absolute symlink is now resolved, breaking the old
assertion) and adds a new upload_absolute_symlink_resolves_contents
regression test that verifies an out-of-tree payload reachable via an
absolute symlink lands in CAS as a file.

Ported from upstream
0807153 (PR
#2243).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Fix zero-digest files missing from worker execution directories

FilesystemStore::get_file_entry_for_digest previously returned a
synthetic FileEntry for zero-digest blobs pointing to a content_path
that never exists on disk (FilesystemStore deliberately never persists
zero-byte files). Downstream worker output-materialisation code that
took the prefetched hardlink path would try to hard_link from this
non-existent source, fail, and either silently produce a missing file
or fall back without enough context to diagnose the failure.

This is the OUTPUT-materialisation companion to PR #2338's DirectoryCache
zero-byte fix, which lives on the input-fetch path. The two changes are
complementary: #2338 covers the directory cache short-circuit; this
covers the FilesystemStore API and the running_actions_manager fallback.

Changes:
- FilesystemStore::get_file_entry_for_digest now returns
  Code::NotFound for zero digests instead of a synthetic FileEntry,
  forcing callers to materialise empty files via fs::create_file rather
  than hard_linking from a phantom source.
- running_actions_manager::download_to_directory adds err_tip context
  on the zero-digest fs::create_file / write_all so a failure here is
  attributable to the empty-file path.
- Updates the existing get_file_entry_for_zero_digest test to assert
  the new NotFound behaviour and renames it to
  get_file_entry_for_zero_digest_returns_not_found.
- Adds download_to_directory_zero_digest_empty_file_test that
  exercises an empty file declared inside an input directory and
  verifies it materialises on disk with zero bytes. This test
  intentionally does NOT collide with PR #2338's DirectoryCache
  zero-byte test (which validates a different short-circuit path).

Ported from upstream
19d7e20 (PR
#2243).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Close test coverage gaps for ported output-upload fixes

Audit of the two ported commits (e57b675b, b1dc3a1c) flagged two
correctness-relevant gaps. Both are now closed with strict assertions so
the underlying bugs cannot silently regress.

1. upload_absolute_symlink_to_directory_uploads_tree
   The original e57b675b regression test covers absolute-symlink-to-file
   but not absolute-symlink-to-directory, which exercises an entirely
   separate code path (upload_directory + Tree proto serialisation). The
   new test creates an out-of-tree directory containing inner.txt,
   absolute-symlinks it into the work dir, runs the action, and:
   - asserts output_directory_symlinks and output_file_symlinks are
     BOTH empty (the symlink must NOT be preserved),
   - asserts output_folders contains exactly one entry at the symlink
     path, proving Tree-upload happened,
   - walks the uploaded Tree, locates inner.txt, fetches its blob from
     CAS and verifies the content. This proves the directory was
     actually traversed and uploaded — not stubbed.

2. download_to_directory_zero_digest_empty_file_test (strengthened)
   Extended the existing single-file test to cover:
   - a second sibling zero-digest file (proves the path is not a
     single-file fluke),
   - a zero-digest file nested inside a subdirectory (proves the
     recursive download_to_directory caller also handles NotFound from
     get_file_entry_for_digest — not just the top-level invocation),
   - strict per-file assertions: is_file, !is_symlink, len == 0, and
     a read-back that confirms the file is actually empty rather than
     a phantom dirent.

Verdict on the dropped checks:
- Dangling absolute symlink: source returns OutputType::None which is
  the correct graceful behaviour, but writing a test for it requires
  fabricating a broken symlink mid-action and is comparatively low
  value; left uncovered intentionally.
- File mode on zero-digest materialisation: the worker uses
  fs::create_file with no explicit mode, so the result follows umask
  and would be brittle across CI environments. Length + type
  assertions are the stronger invariant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* running_actions_manager_test: use /dev/null in upload_dir_and_symlink_test

Previous iterations of this PR sidestepped `/dev/null` by swapping the
absolute-symlink fixture for either a relative symlink (lost coverage)
or an out-of-tree real empty file (added scaffolding to dodge a corner
case we should just test directly).

Restore the original `ln -s /dev/null empty_sym` fixture. The post-fix
worker code resolves the absolute symlink via `fs::metadata` (follows),
sees a character device (`is_dir() == false`), and falls into the
"upload as file" branch. `upload_file` opens `/dev/null` and reads to
EOF — `/dev/null`'s contract is that reads return 0 bytes immediately
— producing the canonical sha256 empty digest (e3b0c44…) with size 0.

That's well-defined, harmless behavior and a real-world Bazel pattern
(rules sometimes use `ln -sf /dev/null x` to create empty outputs).
Test now locks in this contract directly, no scaffolding required.

Drops the `external_root` / `external_file` setup since it was only
needed to avoid `/dev/null`.

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Marcus Eagan <marcuseagan@gmail.com>

Author: 3 people

nativelink-store/src/filesystem_store.rs

@@ -749,16 +749,19 @@ impl<Fe: FileEntry> FilesystemStore<Fe> {
     }
 
     pub async fn get_file_entry_for_digest(&self, digest: &DigestInfo) -> Result<Arc<Fe>, Error> {
+        // Zero-digest blobs have no backing file on disk (FilesystemStore
+        // never persists zero-byte content). The previous implementation
+        // returned a synthetic FileEntry whose content_path did not exist,
+        // which downstream callers would then try to hard_link from,
+        // silently producing missing or empty output files in worker
+        // execution directories. Return NotFound so callers are forced to
+        // take the explicit zero-digest path (e.g. fs::create_file).
         if is_zero_digest(digest) {
-            return Ok(Arc::new(Fe::create(
-                0,
-                0,
-                RwLock::new(EncodedFilePath {
-                    shared_context: self.shared_context.clone(),
-                    path_type: PathType::Content,
-                    key: digest.into(),
-                }),
-            )));
+            return Err(make_err!(
+                Code::NotFound,
+                "{digest} is a zero-digest; FilesystemStore does not persist zero-byte files. \
+                 Callers must materialise empty files directly rather than going through get_file_entry_for_digest."
+            ));
         }
         self.evicting_map
             .get(&digest.into())

nativelink-store/tests/filesystem_store_test.rs

@@ -1004,7 +1004,20 @@ async fn update_with_zero_digest() -> Result<(), Error> {
 }
 
 #[nativelink_test]
-async fn get_file_entry_for_zero_digest() -> Result<(), Error> {
+async fn get_file_entry_for_zero_digest_returns_not_found() -> Result<(), Error> {
+    // Regression test for: zero-digest blobs have no backing file on disk,
+    // so `get_file_entry_for_digest` previously handed back a synthetic
+    // FileEntry whose content_path did not exist. Downstream callers (the
+    // worker output-materialisation path) would then try to hard_link from
+    // that non-existent source, silently producing missing or empty output
+    // files in worker exec dirs.
+    //
+    // After the fix, `get_file_entry_for_digest` returns NotFound for
+    // zero digests so callers are forced to materialise empty files
+    // directly (e.g. via `fs::create_file`). This complements the
+    // DirectoryCache zero-byte short-circuit (input-fetch path); this
+    // test covers the FilesystemStore API used by the output-upload /
+    // hardlink path.
     let digest = DigestInfo::new(Sha256::new().finalize().into(), 0);
     let content_path = make_temp_path("content_path");
     let temp_path = make_temp_path("temp_path");
@@ -1020,8 +1033,11 @@ async fn get_file_entry_for_zero_digest() -> Result<(), Error> {
     )
     .await?;
 
-    let file_entry = store.get_file_entry_for_digest(&digest).await?;
-    assert!(file_entry.is_empty());
+    let err = store
+        .get_file_entry_for_digest(&digest)
+        .await
+        .expect_err("zero digest must not return a synthetic FileEntry");
+    assert_eq!(err.code, Code::NotFound, "expected NotFound, got: {err:?}");
     Ok(())
 }
 

nativelink-worker/src/running_actions_manager.rs

@@ -163,8 +163,18 @@ pub fn download_to_directory<'a>(
                     .populate_fast_store(digest.into())
                     .and_then(move |()| async move {
                         if is_zero_digest(digest) {
-                            let mut file_slot = fs::create_file(&dest).await?;
-                            file_slot.write_all(&[]).await?;
+                            // Zero-digest files are never persisted by the
+                            // FilesystemStore, so materialise them directly
+                            // in the worker exec dir. Attach context so a
+                            // failure here is diagnosable as a missing
+                            // empty file rather than a generic IO error.
+                            let mut file_slot = fs::create_file(&dest)
+                                .await
+                                .err_tip(|| format!("Could not create zero-digest file at {dest}"))?;
+                            file_slot
+                                .write_all(&[])
+                                .await
+                                .err_tip(|| format!("Could not write zero-digest file at {dest}"))?;
                         }
                         else {
                             let file_entry = filesystem_store
@@ -1384,34 +1394,124 @@ impl RunningActionImpl {
                         .err_tip(|| format!("Uploading directory {}", full_path.display()))?,
                     ))
                 } else if metadata.is_symlink() {
-                    let output_symlink = upload_symlink(&full_path, work_directory)
-                        .await
-                        .map(|mut symlink_info| {
-                            symlink_info.name_or_path = NameOrPath::Path(entry);
-                            symlink_info
-                        })
-                        .err_tip(|| format!("Uploading symlink {}", full_path.display()))?;
-                    match fs::metadata(&full_path).await {
-                        Ok(metadata) => {
-                            if metadata.is_dir() {
-                                Ok(OutputType::DirectorySymlink(output_symlink))
-                            } else {
-                                // Note: If it's anything but directory we put it as a file symlink.
-                                Ok(OutputType::FileSymlink(output_symlink))
+                    // Resolve the symlink to determine what it points to.
+                    // Symlinks created by DirectoryCache (absolute paths into
+                    // the cache directory) must NOT be uploaded as symlinks —
+                    // the target path is worker-local and meaningless to the
+                    // client. Instead, follow the symlink and upload the
+                    // resolved content (file or directory).
+                    let target = fs::read_link(&full_path).await.err_tip(|| {
+                        format!("Reading symlink target for {}", full_path.display())
+                    })?;
+                    let is_absolute_symlink = Path::new(&target).is_absolute();
+
+                    if is_absolute_symlink {
+                        // Absolute symlink — resolve and upload contents.
+                        match fs::metadata(&full_path).await {
+                            Ok(resolved_meta) => {
+                                if resolved_meta.is_dir() {
+                                    // Upload as directory (Tree proto).
+                                    Ok(OutputType::Directory(
+                                        upload_directory(
+                                            cas_store.as_pin(),
+                                            &full_path,
+                                            work_directory,
+                                            hasher,
+                                            digest_uploaders,
+                                        )
+                                        .and_then(|(root_dir, children)| async move {
+                                            let tree = ProtoTree {
+                                                root: Some(root_dir),
+                                                children: children.into(),
+                                            };
+                                            let tree_digest = serialize_and_upload_message(
+                                                &tree,
+                                                cas_store.as_pin(),
+                                                &mut hasher.hasher(),
+                                            )
+                                            .await
+                                            .err_tip(|| format!("While processing {entry}"))?;
+                                            Ok(DirectoryInfo {
+                                                path: entry,
+                                                tree_digest,
+                                            })
+                                        })
+                                        .await
+                                        .err_tip(|| {
+                                            format!(
+                                                "Uploading symlinked directory {}",
+                                                full_path.display()
+                                            )
+                                        })?,
+                                    ))
+                                } else {
+                                    // Upload as file (follow symlink).
+                                    Ok(OutputType::File(
+                                        upload_file(
+                                            cas_store.as_pin(),
+                                            &full_path,
+                                            hasher,
+                                            resolved_meta,
+                                            digest_uploaders,
+                                        )
+                                        .await
+                                        .map(|mut file_info| {
+                                            file_info.name_or_path = NameOrPath::Path(entry);
+                                            file_info
+                                        })
+                                        .err_tip(|| {
+                                            format!(
+                                                "Uploading symlinked file {}",
+                                                full_path.display()
+                                            )
+                                        })?,
+                                    ))
+                                }
+                            }
+                            Err(e) => {
+                                if e.code != Code::NotFound {
+                                    return Err(e).err_tip(|| {
+                                        format!(
+                                            "While resolving absolute symlink {}",
+                                            full_path.display()
+                                        )
+                                    });
+                                }
+                                Ok(OutputType::None)
                             }
                         }
-                        Err(e) => {
-                            if e.code != Code::NotFound {
-                                return Err(e).err_tip(|| {
-                                    format!(
-                                        "While querying target symlink metadata for {}",
-                                        full_path.display()
-                                    )
-                                });
+                    } else {
+                        // Relative symlink — action intentionally created it.
+                        // Upload as a proper symlink.
+                        let output_symlink = upload_symlink(&full_path, work_directory)
+                            .await
+                            .map(|mut symlink_info| {
+                                symlink_info.name_or_path = NameOrPath::Path(entry);
+                                symlink_info
+                            })
+                            .err_tip(|| format!("Uploading symlink {}", full_path.display()))?;
+                        match fs::metadata(&full_path).await {
+                            Ok(metadata) => {
+                                if metadata.is_dir() {
+                                    Ok(OutputType::DirectorySymlink(output_symlink))
+                                } else {
+                                    // Note: If it's anything but directory we put it as a file symlink.
+                                    Ok(OutputType::FileSymlink(output_symlink))
+                                }
+                            }
+                            Err(e) => {
+                                if e.code != Code::NotFound {
+                                    return Err(e).err_tip(|| {
+                                        format!(
+                                            "While querying target symlink metadata for {}",
+                                            full_path.display()
+                                        )
+                                    });
+                                }
+                                // If the file doesn't exist, we consider it a file. Even though the
+                                // file doesn't exist we still need to populate an entry.
+                                Ok(OutputType::FileSymlink(output_symlink))
                             }
-                            // If the file doesn't exist, we consider it a file. Even though the
-                            // file doesn't exist we still need to populate an entry.
-                            Ok(OutputType::FileSymlink(output_symlink))
                         }
                     }
                 } else {