action rewinding: recover lost CAS blobs

Remote CAS loss could leave deferred builds unable to recover generated
inputs or final outputs without a daemon restart. This matters because
the build graph still contains producer actions that can recreate those
blobs, but Buck was not always turning missing-CAS errors into graph
rewinds. The existing retry covered some upload and forced
materialization failures, but missed default final materialization,
local executor input materialization, and upload passes with several
generated inputs missing at once.

Track lost inputs as a typed batch so the build command can dirty
all producer BuildKey nodes and the consumer in one rewind. Canonicalize
rewind keys through the registered action lookup before dirtying DICE.
This makes dynamic_output redirects invalidate the producer action that
can recreate the missing blob.

When a rewound action is replayed, bypass both Buck action-cache lookups
and the remote executor cache lookup. Remote execution can otherwise
return the same cached ActionResult, leaving the missing CAS blob absent
and causing the consumer to hit the rewind cap.

When local materialization discovers an expired CAS entry, convert the
materializer not-found error into the same typed context. Also treat
default final materialization not-found errors as rewindable, since
materializations = deferred still materializes requested outputs unless
the stricter skip-final mode is selected.

When final materialization and final upload run together, upload can
report a missing CAS blob before materialization cleans the shared
queue. Because those branches run under try_compute2, the upload error
can drop the materialization side before it removes queue_tracker
entries. Clear those entries on the upload-side rewind path as well.
Also clear the per-transaction materialization queue after committing a
rewind, so the retry does not skip outputs that were queued before the
DICE transaction was invalidated.

The tests use Buck remote-execution test hooks and a hybrid execution
platform instead of external RE configuration. They cover remote
generated inputs, directory leaves, worker-side missing input reports,
local-only consumers, default final materialization, final upload with
materialization, and a missing-input count above the repeated-rewind
cap.

Author: sluongng

app/buck2_action_impl/src/actions/impls/run.rs

@@ -1076,9 +1076,14 @@ impl RunAction {
         // First, check in the local dep file cache if an identical action can be found there.
         // Do this before checking the action cache as we can avoid a potentially large download.
         // Once the action cache lookup misses, we will do the full dep file cache look up.
-        let (outputs, should_fully_check_dep_file_cache) = dep_file_bundle
-            .check_local_dep_file_cache_for_identical_action(ctx, self.outputs.as_slice())
-            .await?;
+        let should_bypass_action_cache = ctx.should_bypass_action_cache();
+        let (outputs, should_fully_check_dep_file_cache) = if should_bypass_action_cache {
+            (None, false)
+        } else {
+            dep_file_bundle
+                .check_local_dep_file_cache_for_identical_action(ctx, self.outputs.as_slice())
+                .await?
+        };
         if let Some((outputs, metadata)) = outputs {
             return Ok(ExecuteResult::LocalDepFileHit(outputs, metadata));
         }
@@ -1541,7 +1546,8 @@ impl Action for RunAction {
         waiting_data: WaitingData,
     ) -> Result<(ActionOutputs, ActionExecutionMetadata), ExecuteError> {
         // Check offline cache first if parameter enabled
-        if self.inner.allow_offline_output_cache
+        if !ctx.should_bypass_action_cache()
+            && self.inner.allow_offline_output_cache
             && ctx.run_action_knobs().use_network_action_output_cache
         {
             if let Some((outputs, metadata)) = self.execute_for_offline(ctx).await? {

app/buck2_build_api/src/actions.rs

@@ -97,6 +97,7 @@ pub mod execute;
 pub mod impls;
 pub mod query;
 pub mod registry;
+pub mod rewind;
 
 /// Represents an unregistered 'Action' that will be registered into the 'Actions' module.
 /// The 'UnregisteredAction' is not executable until it is registered, upon which it becomes an
@@ -286,6 +287,10 @@ pub trait ActionExecutionCtx: Send + Sync {
         prepared_action: &PreparedAction,
     ) -> ControlFlow<CommandExecutionResult, CommandExecutionManager>;
 
+    fn should_bypass_action_cache(&self) -> bool {
+        false
+    }
+
     async fn cache_upload(
         &mut self,
         action: &ActionDigestAndBlobs,