Implement GetTree()

[allada]

To optimize clients that support it, we should implement get_tree()

see:

nativelink/nativelink-service/src/cas_server.rs

Line 244 in 2ae7cab

Err(make_err!(

https://github.com/TraceMachina/turbo-cache/blob/5e2e81af8999482fef202b50ee880509e8811e6f/proto/build/bazel/remote/execution/v2/remote_execution.proto#L430

[blizzardc0der]

I'd like to work on this issue.

[allada]

Given the following documentation in the proto file:

  // Fetch the entire directory tree rooted at a node.
  //
  // This request must be targeted at a
  // [Directory][build.bazel.remote.execution.v2.Directory] stored in the
  // [ContentAddressableStorage][build.bazel.remote.execution.v2.ContentAddressableStorage]
  // (CAS). The server will enumerate the `Directory` tree recursively and
  // return every node descended from the root.
  //
  // The GetTreeRequest.page_token parameter can be used to skip ahead in
  // the stream (e.g. when retrying a partially completed and aborted request),
  // by setting it to a value taken from GetTreeResponse.next_page_token of the
  // last successfully processed GetTreeResponse).
  //
  // The exact traversal order is unspecified and, unless retrieving subsequent
  // pages from an earlier request, is not guaranteed to be stable across
  // multiple invocations of `GetTree`.
  //
  // If part of the tree is missing from the CAS, the server will return the
  // portion present and omit the rest.
  //
  // Errors:
  //
  // * `NOT_FOUND`: The requested tree root is not present in the CAS.

referenced here:

nativelink/proto/build/bazel/remote/execution/v2/remote_execution.proto

Line 430 in 5e2e81a

rpc GetTree(GetTreeRequest) returns (stream GetTreeResponse) {

We want to implement this API.

In short this API is an optimization to reading the CAS objects and walking recursively to collect all sub-directories of the provided digests.

This is a streaming API, so as we are collecting data we can send the data to the client so the client can perform work while we are still doing work.

[allada]

Some code already in NativeLink that already does some similar things:

nativelink/nativelink-worker/src/running_actions_manager.rs

Line 122 in e890c01

pub fn download_to_directory<'a>(

and:

nativelink/nativelink-store/src/completeness_checking_store.rs

Line 75 in e890c01

async fn check_output_directories(

[aleksdmladenovic]

I'd like to work on this issue.
cc: @allada