Render generate-github-actions output via Jinja2

Summary:
`GenerateGitHubActionsCmd.write_job_for_platform` previously emitted GitHub Actions workflow YAML through ~120 inline `out.write()` calls interleaved with platform conditionals — a wall of string-building that made every CI-shape change a delicate edit. This diff replaces that with a single Jinja2 template plus a context-builder method, while preserving byte-identical output for every existing invocation.

The template lives at `getdeps/templates/workflow.yml.j2`. To keep GitHub Actions `${{ ... }}` expressions readable inside the template, the Jinja environment uses non-default delimiters: `<< >>` for variables, `<% %>` for blocks, `<# #>` for comments. `keep_trailing_newline=True` is set so the template's final newline is preserved verbatim. The literal string `generated` in the file header is constructed at render time as `<<"@" + "generated">>` so the template itself isn't flagged as a generated file. `jinja2` is imported lazily inside `_render_workflow` so OSS GHA runners executing `getdeps.py build`/`test`/`install-system-deps` are not forced to install it; only callers of `generate-github-actions` need it on their system.

The refactor also splits `getdeps.py` into a thin OSS-facing shim and a `getdeps/cli.py` module that holds all the command classes, `parse_args`, and `main`. Supporting pieces moved with it: `ProjectCmdBase`, `UsageError`, and `BUILD_TYPE_ARG` to `getdeps/cmd_base.py`; `GenerateGitHubActionsCmd` plus its rendering helpers (`_render_workflow`, `_build_render_context`, `_parse_per_package_defines`) to `getdeps/workflow_generator.py`. The shim's only job is `sys.path.insert` for OSS users running `python3 build/fbcode_builder/getdeps.py …` directly. Inside Buck, a new `python_binary` target `//opensource/fbcode_builder:getdeps` invokes `getdeps.cli:main`, brings the third-party deps along, and resolves the long-standing namespace collision between `getdeps.py` (script) and `getdeps/` (package). `update-all-github-actions.sh` now invokes via `buck run` so Meta engineers no longer need jinja2 installed in their system python.

The new `_build_render_context` method does all data preparation (manifest walk, dep ordering, command-string composition, env-var assembly) and returns a flat `dict` consumed by the template. CLI-string assembly (`build_type_arg`, `cmake_arg_for(...)`, `getdepscmd ...`) stays in Python — composing these in Jinja would be worse than what we have today.

Several small simplifications fall out of the move: the per-OS `(artifacts, runs_on, py3)` derivation moves into a `_resolve_platform(args, build_opts)` helper at module scope; the `_PLATFORMS` list becomes a module-level constant; the five repeated `manifest.get("github.actions", ..., ctx=manifest_ctx)` lookups go through a local `gh(key)` closure; and the `tests_arg`/`job_file_prefix`/`job_name` "default-then-override" blocks collapse to single `or`/ternary expressions. With those out, `write_job_for_platform` no longer needs its `noqa: C901`.

Output equivalence is verified two ways. (1) `getdeps/test/workflow_generator_test.py` is a golden test that drives the generator in-process for four representative scenarios (xxhash all-OSes, folly shared-libs linux, openr no-system-packages run-on-all-branches, rebalancer with cmake overrides) and asserts each emitted YAML file matches a checked-in fixture. When the fixture and actual output drift, the assertion failure prints how to regenerate. The test accepts a `--update-fixtures` flag (`buck run //opensource/fbcode_builder/getdeps/test:test -- --update-fixtures`) which rewrites the per-scenario fixture files from current output and skips assertions; the resulting `sl status` diff is the change record. (2) Running every invocation in `getdeps/facebook/update-all-github-actions.sh` against pre- and post-refactor builds and diffing produces zero differences across all 44 generated `.yml` files.

This change does not touch the CLI flag surface, schema, or output files of `update-all-github-actions.sh`. It is a pure refactor.

Reviewed By: bigfootjon

Differential Revision: D104495377

fbshipit-source-id: c7eb6a118c94d41fdc4525235f2319772f259812

Author: r-barnes

build/fbcode_builder/getdeps.py

@@ -0,0 +1,188 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import os
+from typing import Any
+
+from .buildopts import setup_build_options
+from .load import ManifestLoader
+from .subcmd import SubCmd
+
+
+class UsageError(Exception):
+    pass
+
+
+# Shared argument definition for --build-type used by multiple commands
+BUILD_TYPE_ARG: dict[str, Any] = {
+    "help": "Set the build type explicitly: Debug (unoptimized, debug symbols), RelWithDebInfo (optimized with debug symbols, default), MinSizeRel (size-optimized, no debug), or Release (optimized, no debug).",
+    "choices": ["Debug", "Release", "RelWithDebInfo", "MinSizeRel"],
+    "action": "store",
+    "default": "RelWithDebInfo",
+}
+
+
+class ProjectCmdBase(SubCmd):
+    def run(self, args):
+        opts = setup_build_options(args)
+
+        if args.current_project is not None:
+            opts.repo_project = args.current_project
+        if args.project is None:
+            if opts.repo_project is None:
+                raise UsageError(
+                    "no project name specified, and no .projectid file found"
+                )
+            if opts.repo_project == "fbsource":
+                # The fbsource repository is a little special.  There is no project
+                # manifest file for it.  A specific project must always be explicitly
+                # specified when building from fbsource.
+                raise UsageError(
+                    "no project name specified (required when building in fbsource)"
+                )
+            args.project = opts.repo_project
+
+        ctx_gen = opts.get_context_generator()
+        if args.test_dependencies:
+            ctx_gen.set_value_for_all_projects("test", "on")
+        if args.enable_tests:
+            ctx_gen.set_value_for_project(args.project, "test", "on")
+        else:
+            ctx_gen.set_value_for_project(args.project, "test", "off")
+
+        if opts.shared_libs:
+            ctx_gen.set_value_for_all_projects("shared_libs", "on")
+
+        loader = ManifestLoader(opts, ctx_gen)
+        self.process_project_dir_arguments(args, loader)
+
+        manifest = loader.load_manifest(args.project)
+
+        return self.run_project_cmd(args, loader, manifest)
+
+    def process_project_dir_arguments(self, args, loader):
+        def parse_project_arg(arg, arg_type):
+            parts = arg.split(":")
+            if len(parts) == 2:
+                project, path = parts
+            elif len(parts) == 1:
+                project = args.project
+                path = parts[0]
+            # On Windows path contains colon, e.g. C:\open
+            elif os.name == "nt" and len(parts) == 3:
+                project = parts[0]
+                path = parts[1] + ":" + parts[2]
+            else:
+                raise UsageError(
+                    "invalid %s argument; too many ':' characters: %s" % (arg_type, arg)
+                )
+
+            return project, os.path.abspath(path)
+
+        # If we are currently running from a project repository,
+        # use the current repository for the project sources.
+        build_opts = loader.build_opts
+        if build_opts.repo_project is not None and build_opts.repo_root is not None:
+            loader.set_project_src_dir(build_opts.repo_project, build_opts.repo_root)
+
+        for arg in args.src_dir:
+            project, path = parse_project_arg(arg, "--src-dir")
+            loader.set_project_src_dir(project, path)
+
+        for arg in args.build_dir:
+            project, path = parse_project_arg(arg, "--build-dir")
+            loader.set_project_build_dir(project, path)
+
+        for arg in args.install_dir:
+            project, path = parse_project_arg(arg, "--install-dir")
+            loader.set_project_install_dir(project, path)
+
+        for arg in args.project_install_prefix:
+            project, path = parse_project_arg(arg, "--install-prefix")
+            loader.set_project_install_prefix(project, path)
+
+    def setup_parser(self, parser):
+        parser.add_argument(
+            "project",
+            nargs="?",
+            help=(
+                "name of the project or path to a manifest "
+                "file describing the project"
+            ),
+        )
+        parser.add_argument(
+            "--no-tests",
+            action="store_false",
+            dest="enable_tests",
+            default=True,
+            help="Disable building tests for this project.",
+        )
+        parser.add_argument(
+            "--test-dependencies",
+            action="store_true",
+            help="Enable building tests for dependencies as well.",
+        )
+        parser.add_argument(
+            "--current-project",
+            help="Specify the name of the fbcode_builder manifest file for the "
+            "current repository.  If not specified, the code will attempt to find "
+            "this in a .projectid file in the repository root.",
+        )
+        parser.add_argument(
+            "--src-dir",
+            default=[],
+            action="append",
+            help="Specify a local directory to use for the project source, "
+            "rather than fetching it.",
+        )
+        parser.add_argument(
+            "--build-dir",
+            default=[],
+            action="append",
+            help="Explicitly specify the build directory to use for the "
+            "project, instead of the default location in the scratch path. "
+            "This only affects the project specified, and not its dependencies.",
+        )
+        parser.add_argument(
+            "--install-dir",
+            default=[],
+            action="append",
+            help="Explicitly specify the install directory to use for the "
+            "project, instead of the default location in the scratch path. "
+            "This only affects the project specified, and not its dependencies.",
+        )
+        parser.add_argument(
+            "--project-install-prefix",
+            default=[],
+            action="append",
+            help="Specify the final deployment installation path for a project",
+        )
+
+        self.setup_project_cmd_parser(parser)
+
+    def setup_project_cmd_parser(self, parser):
+        pass
+
+    def create_builder(self, loader, manifest):
+        fetcher = loader.create_fetcher(manifest)
+        src_dir = fetcher.get_src_dir()
+        ctx = loader.ctx_gen.get_context(manifest.name)
+        build_dir = loader.get_project_build_dir(manifest)
+        inst_dir = loader.get_project_install_dir(manifest)
+        return manifest.create_builder(
+            loader.build_opts,
+            src_dir,
+            build_dir,
+            inst_dir,
+            ctx,
+            loader,
+            loader.dependencies_of(manifest),
+        )
+
+    def check_built(self, loader, manifest):
+        built_marker = os.path.join(
+            loader.get_project_install_dir(manifest), ".built-by-getdeps"
+        )
+        return os.path.exists(built_marker)

build/fbcode_builder/getdeps/cli.py

@@ -476,7 +476,7 @@ def filter_strip_marker(dest_name: str, marker: str) -> None:
 
 
 def list_files_under_dir_newer_than_timestamp(
-    dir_to_scan: str, ts: int
+    dir_to_scan: str, ts: float
 ) -> Iterator[str]:
     for root, _dirs, files in os.walk(dir_to_scan):
         for src_file in files:

build/fbcode_builder/getdeps/cmd_base.py

@@ -973,7 +973,7 @@ def set(self, key: str, value: str | None) -> None:
         assert key in self.ALLOWED_VARIABLES or key.startswith("feature_")
         self.ctx_dict[key] = value
 
-    def features(self) -> set[str]:
+    def features(self) -> typing.Set[str]:
         return {
             k[len("feature_") :]
             for k, v in self.ctx_dict.items()