Skip to content

Commit d396d12

Browse files
KapJIfacebook-github-bot
KapJI
authored and
facebook-github-bot
committed
Add test for persistent remote workers on BuildBuddy
Summary: Part of #787 Includes an example setup that works with - local builds without persistent worker - local builds with persistent worker (Buck2 protocol) - remote builds without persistent worker The demo worker included in the example in this PR distinguishes between Buck2 worker, Bazel remote worker, and one-shot modes depending on whether Buck2's WORKER_SOCKET, Bazel's --persistent_worker flag, or neither is set. The example includes a README with detailed instructions how to test this feature. - remote builds with persistent worker (Bazel protocol) Reviewed By: scottcao Differential Revision: D68157749 fbshipit-source-id: 51e2e247c75e0ca9736ddc0a5f383e662edee298
1 parent df48a53 commit d396d12

29 files changed

+1150
-1
lines changed

‎.github/actions/build_example_persistent_worker/action.yml

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
name: build_example_persistent_worker
2+
inputs:
3+
buildbuddyApiKey:
4+
description: "The API key for BuildBuddy remote cache and execution."
5+
required: true
6+
runs:
7+
using: composite
8+
steps:
9+
- name: Build examples/persistent_worker directory
10+
env:
11+
BUILDBUDDY_API_KEY: ${{ inputs.buildbuddyApiKey }}
12+
run: |-
13+
cd examples/persistent_worker
14+
export PATH="$RUNNER_TEMP/artifacts:$PATH"
15+
./test.sh
16+
shell: bash

‎.github/workflows/build-and-test.yml

+5-1
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ name: Build and test
22
on:
33
push:
44
pull_request:
5+
workflow_dispatch: # allows manual triggering
56
jobs:
67
linux-build-and-test:
78
runs-on: 4-core-ubuntu
@@ -51,7 +52,7 @@ jobs:
5152
- uses: ./.github/actions/setup_reindeer
5253
- uses: ./.github/actions/build_bootstrap
5354
linux-build-examples:
54-
runs-on: ubuntu-latest
55+
runs-on: ubuntu-22.04
5556
steps:
5657
- uses: actions/[email protected]
5758
- uses: ./.github/actions/setup_linux_env
@@ -69,6 +70,9 @@ jobs:
6970
$RUNNER_TEMP/artifacts/buck2 test //... -v 2
7071
- uses: ./.github/actions/build_example_conan
7172
- uses: ./.github/actions/build_example_no_prelude
73+
- uses: ./.github/actions/build_example_persistent_worker
74+
with:
75+
buildbuddyApiKey: ${{ secrets.BUILDBUDDY_API_KEY }}
7276
- uses: ./.github/actions/setup_reindeer
7377
- uses: ./.github/actions/build_bootstrap
7478
windows-build-examples:

‎examples/persistent_worker/.buckconfig

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
[cells]
2+
root = .
3+
prelude = prelude
4+
toolchains = toolchains
5+
none = none
6+
7+
[cell_aliases]
8+
config = prelude
9+
fbcode = none
10+
fbsource = none
11+
buck = none
12+
13+
[external_cells]
14+
prelude = bundled
15+
16+
[parser]
17+
target_platform_detector_spec = target:root//...->prelude//platforms:default

‎examples/persistent_worker/.buckconfig.buildbuddy

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[buck2]
2+
digest_algorithms = SHA256
3+
4+
[buck2_re_client]
5+
engine_address = grpc://remote.buildbuddy.io
6+
action_cache_address = grpc://remote.buildbuddy.io
7+
cas_address = grpc://remote.buildbuddy.io
8+
tls = true
9+
http_headers = \
10+
x-buildbuddy-api-key:$BUILDBUDDY_API_KEY
11+
12+
[build]
13+
execution_platforms = root//platforms:buildbuddy

‎examples/persistent_worker/.buckconfig.buildbuddy-persistent-workers

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[buck2]
2+
digest_algorithms = SHA256
3+
4+
[buck2_re_client]
5+
engine_address = grpc://remote.buildbuddy.io
6+
action_cache_address = grpc://remote.buildbuddy.io
7+
cas_address = grpc://remote.buildbuddy.io
8+
tls = true
9+
http_headers = \
10+
x-buildbuddy-api-key:$BUILDBUDDY_API_KEY
11+
12+
[build]
13+
execution_platforms = root//platforms:buildbuddy-persistent-workers

‎examples/persistent_worker/.buckconfig.local-persistent-workers

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[build]
2+
execution_platforms = root//platforms:local-persistent-workers

‎examples/persistent_worker/.buckconfig.no-workers

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
[build]
2+
execution_platforms = root//platforms:local

‎examples/persistent_worker/.buckroot

Whitespace-only changes.

‎examples/persistent_worker/.envrc

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# specify the following:
2+
# - BUILDBUDDY_API_KEY
3+
source_env_if_exists .envrc.private

‎examples/persistent_worker/.gitignore

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
.buckconfig.local
2+
.direnv
3+
.envrc.private
4+
prelude

‎examples/persistent_worker/BUCK

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
load("defs.bzl", "demo", "worker")
2+
3+
oncall("build_infra")
4+
5+
python_binary(
6+
name = "one_shot",
7+
main = "one_shot.py",
8+
)
9+
10+
python_binary(
11+
name = "worker_py",
12+
main = "persistent_worker.py",
13+
deps = [
14+
"//proto/bazel:worker_protocol_pb2",
15+
"//proto/buck2:worker_pb2",
16+
],
17+
)
18+
19+
worker(
20+
name = "worker",
21+
visibility = ["PUBLIC"],
22+
worker = ":worker_py",
23+
)
24+
25+
[
26+
demo(name = "demo-" + str(i))
27+
for i in range(4)
28+
]

‎examples/persistent_worker/README.md

+145
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
# Persistent Worker Demo
2+
3+
At the time of writing (2024-09-25) Buck2 supports persistent workers for local
4+
builds through a dedicated Buck2 persistent worker gRPC protocol. However, Buck2
5+
does not support persistent workers for builds that use remote execution. This
6+
demo is part of a patch-set that adds support for remote persistent workers to
7+
Buck2, see [#776].
8+
9+
[#776]: https://github.com/facebook/buck2/issues/776
10+
11+
## Requirements
12+
13+
This demo uses BuildBuddy remote execution to demonstrate remote persistent
14+
workers. You will need an API token for at least a free open source account. You
15+
can use [direnv] to set up the environment:
16+
17+
Credentials for [BuildBuddy] stored in `.envrc.private`:
18+
19+
```
20+
export BUILDBUDDY_API_KEY=...
21+
```
22+
23+
On CI the API key is not available for pipelines initiated from forks of the
24+
main Buck2 repository. The corresponding tests will be skipped in that case. A
25+
Meta engineer can manually initiate a pipeline run with the token set.
26+
27+
[direnv]: https://direnv.net/
28+
[BuildBuddy]: https://www.buildbuddy.io/
29+
30+
## Local Build
31+
32+
Configure a local build without persistent workers:
33+
34+
```
35+
$ cd examples/persistent_worker
36+
$ echo '<file:.buckconfig.no-workers>' > .buckconfig.local
37+
```
38+
39+
Run a clean build:
40+
41+
```
42+
$ buck2 clean; buck2 build : -vstderr
43+
...
44+
stderr for root//:demo-7 (demo):
45+
...
46+
ONE-SHOT START
47+
...
48+
```
49+
50+
## Local Persistent Worker
51+
52+
Configure a local build with persistent workers:
53+
54+
```
55+
$ cd examples/persistent_worker
56+
$ echo '<file:.buckconfig.local-persistent-workers>' > .buckconfig.local
57+
```
58+
59+
Run a clean build:
60+
61+
```
62+
$ buck2 clean; buck2 build : -vstderr
63+
...
64+
stderr for root//:demo-7 (demo):
65+
...
66+
Buck2 persistent worker ...
67+
...
68+
```
69+
70+
## Remote Execution
71+
72+
Configure a remote build without persistent workers:
73+
74+
```
75+
$ cd examples/persistent_worker
76+
$ echo '<file:.buckconfig.buildbuddy>' > .buckconfig.local
77+
```
78+
79+
Run a clean build:
80+
81+
```
82+
$ buck2 clean; buck2 build : -vstderr
83+
...
84+
stderr for root//:demo-7 (demo):
85+
...
86+
ONE-SHOT START
87+
...
88+
```
89+
90+
## Remote Persistent Worker
91+
92+
Configure a remote build with persistent workers:
93+
94+
```
95+
$ cd examples/persistent_worker
96+
$ echo '<file:.buckconfig.buildbuddy-persistent-workers>' > .buckconfig.local
97+
```
98+
99+
Run a clean build:
100+
101+
```
102+
$ buck2 clean; buck2 build : -vstderr
103+
...
104+
stderr for root//:demo-7 (demo):
105+
...
106+
Bazel persistent worker ...
107+
...
108+
```
109+
110+
## Protocol
111+
112+
### Starlark
113+
114+
A Buck2 persistent worker is created by a rule that emits the `WorkerInfo`
115+
provider. Setting `remote = True` on this provider indicates that this worker is
116+
remote execution capable.
117+
118+
Buck2 actions indicate that they can utilize a persistent worker by setting the
119+
`exe` parameter to `ctx.actions.run` to `WorkerRunInfo(worker, exe)`, where
120+
`worker` is a `WorkerInfo` provider, and `exe` defines the fallback executable
121+
for non persistent-worker execution.
122+
123+
Buck2 actions that want to utilize a remote persistent worker must pass
124+
command-line arguments in an argument file specified as `@argfile`,
125+
`-flagfile=argfile`, or `--flagfile=argfile` on the command-line.
126+
127+
### Local Persistent Worker
128+
129+
A locally executed Buck2 persistent worker falls under the
130+
[Buck2 persistent worker protocol](./proto/buck2/worker.proto): It is started
131+
and managed by Buck2 and passed a file path in the `WORKER_SOCKET` environment
132+
variable where it should create a gRPC Unix domain socket to serve worker
133+
requests over. Multiple requests may be sent in parallel and expected to be
134+
served at the same time depending on the `concurrency` attribute of the
135+
`WorkerInfo` provider.
136+
137+
### Remote Persistent Worker
138+
139+
A remotely executed Buck2 persistent worker falls under the
140+
[Bazel persistent worker protocol](./proto/bazel/worker_protocol.proto): It is
141+
started and managed by the remote execution system. Work requests are sent as
142+
length prefixed protobuf objects to the standard input of the worker process.
143+
Work responses are expected as length prefixed protobuf objects on the standard
144+
output of the worker process. The worker process may not use standard output for
145+
anything else.

‎examples/persistent_worker/defs.bzl

+63
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Copyright (c) Meta Platforms, Inc. and affiliates.
2+
#
3+
# This source code is licensed under both the MIT license found in the
4+
# LICENSE-MIT file in the root directory of this source tree and the Apache
5+
# License, Version 2.0 found in the LICENSE-APACHE file in the root directory
6+
# of this source tree.
7+
8+
load("@prelude//utils:argfile.bzl", "at_argfile")
9+
10+
def _worker_impl(ctx: AnalysisContext) -> list[Provider]:
11+
return [
12+
DefaultInfo(),
13+
WorkerInfo(
14+
exe = ctx.attrs.worker[RunInfo].args,
15+
concurrency = None,
16+
supports_bazel_remote_persistent_worker_protocol = True,
17+
),
18+
]
19+
20+
worker = rule(
21+
impl = _worker_impl,
22+
attrs = {
23+
"worker": attrs.dep(providers = [RunInfo]),
24+
},
25+
)
26+
27+
def _demo_impl(ctx: AnalysisContext) -> list[Provider]:
28+
output = ctx.actions.declare_output(ctx.label.name)
29+
argfile = at_argfile(
30+
actions = ctx.actions,
31+
name = "demo." + ctx.label.name + ".args",
32+
args = cmd_args(output.as_output()),
33+
)
34+
ctx.actions.run(
35+
cmd_args(argfile),
36+
category = "demo",
37+
env = {
38+
# modify this value to force an action rerun even if caching is enabled.
39+
# `--no-remote-cache` does not have the desired effect, because it also causes
40+
# the action to be omitted from `buck2 log what-ran`, which interferes with the
41+
# test setup.
42+
"CACHE_SILO_KEY": read_root_config("build", "cache_silo_key", "0"),
43+
},
44+
exe = WorkerRunInfo(
45+
worker = ctx.attrs._worker[WorkerInfo],
46+
exe = ctx.attrs._one_shot[RunInfo].args,
47+
),
48+
)
49+
return [DefaultInfo(default_output = output)]
50+
51+
demo = rule(
52+
impl = _demo_impl,
53+
attrs = {
54+
"_one_shot": attrs.exec_dep(
55+
default = "//:one_shot",
56+
providers = [RunInfo],
57+
),
58+
"_worker": attrs.exec_dep(
59+
default = "//:worker",
60+
providers = [WorkerInfo],
61+
),
62+
},
63+
)

‎examples/persistent_worker/one_shot.py

+31
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/usr/bin/env python3
2+
# Copyright (c) Meta Platforms, Inc. and affiliates.
3+
#
4+
# This source code is licensed under both the MIT license found in the
5+
# LICENSE-MIT file in the root directory of this source tree and the Apache
6+
# License, Version 2.0 found in the LICENSE-APACHE file in the root directory
7+
# of this source tree.
8+
9+
import argparse
10+
import os
11+
import sys
12+
13+
14+
def main():
15+
parser = argparse.ArgumentParser(
16+
fromfile_prefix_chars="@", prog="one_shot", description="One-shot command"
17+
)
18+
parser.add_argument("outfile", type=argparse.FileType("w"), help="Output file.")
19+
20+
args = parser.parse_args()
21+
22+
print("one-shot.py", file=sys.stderr)
23+
print("ONE-SHOT START", file=sys.stderr)
24+
name = os.path.basename(args.outfile.name)
25+
args.outfile.write(name + "\n")
26+
args.outfile.close()
27+
print("ONE-SHOT END", file=sys.stderr)
28+
29+
30+
if __name__ == "__main__":
31+
main()

0 commit comments

Comments
 (0)