Retry build when `RemoteActionFileSystem` encounters a missing digest #25358

fmeum · 2025-02-21T19:51:32Z

Cache evictions encountered during reads of remote files in RemoteActionFileSystem now result in the build being retried when --experimental_remote_cache_eviction_retries is set to a positive value (the default).

This is enabled by aligning the logic behind "build rewinding" closer with that of "action rewinding". By throwing LostInputExecExceptions in the right locations and implementing the checkForLostInputs method on the RemoteOutputService, build rewinding can be realized as a coarse-grained fallback for the more fine-grained action rewinding. This approach avoids further divergence between Bazel and Blaze logic and ensures that further improvements to build rewinding also simplify future efforts to add action rewinding to Bazel.

This change also adds a test case that demonstrates how top-level artifacts that have been cache evicted result in a build failure that isn't retried, which needs to be fixed by follow-up work.

This reverts commit 5cc0347.

fmeum · 2025-02-25T11:57:56Z

@justinhorvitz Could you review the interaction with the action rewinding machinery, including the changes I had to make to RewindingTest?
@coeuvre Could you review the rest, i.e., the integration with build rewinding and the remote module?

fmeum · 2025-02-25T11:59:03Z

src/main/java/com/google/devtools/build/lib/remote/RemoteSpawnRunner.java

@@ -639,9 +674,13 @@ private SpawnResult handleError(
      status = Status.EXECUTION_FAILED_CATASTROPHICALLY;
      detailedCode = FailureDetails.Spawn.Code.EXECUTION_FAILED;
      catastrophe = true;
-    } else if (remoteCacheFailed) {
+    } else if (BulkTransferException.allCausedByCacheNotFoundException(exception)) {
+      // At this point, cache evictions that affect uploaded inputs have already been handled.


@coeuvre This is a change in behavior, but I think it's for the better as it avoids retries that are very unlikely to succeed.

fmeum · 2025-02-25T12:09:10Z

src/test/shell/bazel/remote/build_without_the_bytes_test.sh

+
+  # Incremental build in toplevel build triggers remote cache eviction error
+  # but Bazel doesn't automatically retry the build yet.
+  # TODO: This documents the current behavior, but it's not intended.


@justinhorvitz To fix this, I would need to thread information about lost inputs obtained in

bazel/src/main/java/com/google/devtools/build/lib/skyframe/CompletionFunction.java

Line 375 in 998e762

ensureToplevelArtifacts(env, importantArtifacts, inputMap);

through to the ImportantOutputHandler. I think I roughly understand what that would require, but the two calls to informImportantOutputHandler further above and the mentioning of error bubbling tell me that this is probably pretty difficult to get right. Do you have any advice for me?

coeuvre

Thanks for the PR! I like the direction that you make build rewinding more similar to action rewinding.

If it is not too difficult to do, I would like you to split this PR into 3 PRs for easier reviews:

A PR that overhauls the build rewinding mechanism.
A PR that fixes build rewinding for jdeps.
A PR that contains remaining changes in this PR that don't belong to above 2.

coeuvre · 2025-02-26T10:02:10Z

src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java

+      ImmutableMap<String, ActionInput> newlyLostInputs =
+          e.getLostInputs(inputArtifactData::getInput);
+      if (!newlyLostInputs.isEmpty()) {
+        if (lostInputs == null) {


Access to lostInputs should be synchronized since an action may read its inputs concurrently.

coeuvre · 2025-02-26T10:14:00Z

src/main/java/com/google/devtools/build/lib/remote/RemoteExecutionCache.java

-              throw new CacheNotFoundException(digest, path.getPathString());
+              throw new CacheNotFoundException(
+                  digest,
+                  context.getSpawnExecutionContext().getPathResolver().relativeToExecRoot(path));


I prefer not to add new method relativeToExecRoot to ArtifactPathResolver for only this purpose.

An alternative is to pass in RemovePathResolver and use that to convert local path to exec path, which is consistent with what you've done in RemoteExecutionService.

coeuvre · 2025-02-26T10:18:45Z

src/main/java/com/google/devtools/build/lib/remote/util/DigestUtil.java

-      throws IOException {
-    // Stream the virtual action input as parameter files, which can be very large, are lazily
-    // computed from the in-memory CommandLine object. This avoids allocating large byte arrays.
+  public static Digest compute(StreamWriter input, HashFunction hashFunction) throws IOException {


It seems like changes in this file are not relevant to the feature the PR aims for, can you split it out?

coeuvre · 2025-02-26T10:20:36Z

src/main/java/com/google/devtools/build/lib/rules/java/JavaCompileActionContext.java

-      }
-    } catch (UncheckedIOException e) {
-      throw e.getCause();
+    // The cache value computation is potentially expensive, e.g. when we have to download an input


It seems like changes in this file are not relevant to the feature the PR aims for, can you split it out?

coeuvre · 2025-02-26T10:32:22Z

src/main/java/com/google/devtools/build/lib/remote/RemoteActionFileSystem.java

@@ -921,6 +940,18 @@ protected void createHardLink(PathFragment linkPath, PathFragment originalPath)
    localFs.getPath(linkPath).createHardLink(getPath(originalPath));
  }

+  public void checkForLostInputs(Action action) throws LostInputsActionExecutionException {


I might be missing some dots, but can you explain, before this PR, why Bazel didn't rewind the build? (the CacheNotFoundError was ignored by the call sites?)

fmeum force-pushed the build-rewinding-jdeps branch 3 times, most recently from aae13ad to 7d8cbef Compare February 22, 2025 16:26

fmeum added 4 commits February 24, 2025 10:06

Add test

0645b69

Drop unused argument

628d026

Set exec path string

873fb27

Improve exec path handling

9cb500c

fmeum force-pushed the build-rewinding-jdeps branch from 7d8cbef to 9cb500c Compare February 24, 2025 12:28

fmeum added 10 commits February 24, 2025 13:33

Fix more tests

382bc4c

Fix more tests

a485e09

Undo RemoteSpawnRunner changes

0facf64

Fix spawn runner

5f7c524

Avoid unnecessary retries

417e5d0

Adapt remote test

3fa1435

Adapt another test

24b4b78

Use InputMetadataProvider

5cc0347

Revert "Use InputMetadataProvider"

6c1e15c

This reverts commit 5cc0347.

Add comment

a12ef71

fmeum changed the title ~~Build rewinding jdeps~~ Rewind builds when the RemoteActionFileSystem encounters a missing digest Feb 25, 2025

fmeum changed the title ~~Rewind builds when the RemoteActionFileSystem encounters a missing digest~~ Retry build when RemoteActionFileSystem encounters a missing digest Feb 25, 2025

Actually run the test

424380c

fmeum requested review from coeuvre and justinhorvitz February 25, 2025 11:56

fmeum marked this pull request as ready for review February 25, 2025 11:56

fmeum requested review from a team and lberki as code owners February 25, 2025 11:56

github-actions bot added team-Performance Issues for Performance teams team-Rules-Java Issues for Java rules team-Remote-Exec Issues and PRs for the Execution (Remote) team labels Feb 25, 2025

fmeum removed the request for review from a team February 25, 2025 11:57

github-actions bot added the awaiting-review PR is awaiting review from an assigned reviewer label Feb 25, 2025

fmeum removed the request for review from lberki February 25, 2025 11:57

fmeum commented Feb 25, 2025

View reviewed changes

coeuvre requested changes Feb 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retry build when `RemoteActionFileSystem` encounters a missing digest #25358

Retry build when `RemoteActionFileSystem` encounters a missing digest #25358

fmeum commented Feb 21, 2025 •

edited

Loading

fmeum commented Feb 25, 2025

fmeum Feb 25, 2025

fmeum Feb 25, 2025

coeuvre left a comment

coeuvre Feb 26, 2025

coeuvre Feb 26, 2025

coeuvre Feb 26, 2025

coeuvre Feb 26, 2025

coeuvre Feb 26, 2025

Retry build when RemoteActionFileSystem encounters a missing digest #25358

Are you sure you want to change the base?

Retry build when RemoteActionFileSystem encounters a missing digest #25358

Conversation

fmeum commented Feb 21, 2025 • edited Loading

fmeum commented Feb 25, 2025

fmeum Feb 25, 2025

Choose a reason for hiding this comment

fmeum Feb 25, 2025

Choose a reason for hiding this comment

coeuvre left a comment

Choose a reason for hiding this comment

coeuvre Feb 26, 2025

Choose a reason for hiding this comment

coeuvre Feb 26, 2025

Choose a reason for hiding this comment

coeuvre Feb 26, 2025

Choose a reason for hiding this comment

coeuvre Feb 26, 2025

Choose a reason for hiding this comment

coeuvre Feb 26, 2025

Choose a reason for hiding this comment

Retry build when `RemoteActionFileSystem` encounters a missing digest #25358

Retry build when `RemoteActionFileSystem` encounters a missing digest #25358

fmeum commented Feb 21, 2025 •

edited

Loading