Bosatsu should stop relying on host-installed bdw-gc today, and should be able to add libuv soon, without requiring either library to be globally installed on the local machine. The right model is:
- each Bosatsu runtime version selects a dependency manifest;
- each dependency source archive is downloaded and hash-verified into
.bosatsuc; - each dependency is built into a Bosatsu-managed local prefix under
.bosatsuc; - built third-party artifacts are reused across Bosatsu compiler/runtime bumps whenever the dependency source identity and build context are unchanged;
- Bosatsu-generated C is linked against the local static artifacts, not a globally installed dynamic library.
This document proposes an implementation path that starts with vendored libgc, preserves the current runtime install flow as much as possible, and leaves a clean path for adding libuv.
Today the Bosatsu C runtime already has some useful building blocks:
bosatsu c-runtime installcan download a hash-pinned runtime archive into.bosatsuc, unpack it, and build locally.- the install step writes a
cc_conf.jsonfile consumed later by C compilation/linking. - Bosatsu already stores per-repo compiler/runtime state under
.bosatsuc.
The main problem is that bdw-gc is still discovered from the host machine:
- via
pkg-config bdw-gc, or - via hard-coded Homebrew/MacPorts/system fallbacks, or
- via
-lgcexpecting a system install.
That means the Bosatsu runtime is still coupled to whatever the machine has installed. The same problem would recur for libuv.
- Remove the requirement that
bdw-gcorlibuvbe globally installed. - Reuse previously built third-party dependencies across Bosatsu version bumps.
- Keep Bosatsu version selection in control of which dependency sources are acceptable.
- Support macOS and Linux in the initial implementation.
- Keep the design valid for Windows, even if Windows is not fully shipped in phase 1.
- Link Bosatsu-generated executables against Bosatsu-managed static artifacts under
.bosatsuc. - Preserve reproducibility by pinning source archives by hash.
- Preserve room for dependency-specific build requirements and future transitive dependencies.
- This plan does not attempt to make the final Bosatsu executable fully static against the system C runtime.
- This plan does not require Bosatsu to bundle prebuilt binaries for every platform.
- This plan does not require phase 1 to ship
libuv; phase 1 only needs to avoid painting us into a corner. - This plan does not require immediate end-to-end Windows support for the Bosatsu runtime itself.
Bosatsu version should determine which dependency descriptors are valid, but the reuse key for built libgc.a or libuv.a should not include the Bosatsu compiler/runtime version.
If Bosatsu X and Bosatsu Y both depend on the same libgc release tarball with the same hash, and the local build context is the same, we should reuse the same previously built artifact.
We need two levels of cache keys:
- source identity:
- dependency name
- dependency version
- verified source archive hash
- build identity:
- source identity
- target OS
- target architecture
- toolchain family and version
- relevant recipe options
- build profile
- recipe version
- transitive dependency build identities, if any
This is the minimum needed to safely reuse artifacts without conflating incompatible builds.
Every vendored dependency build should install into a Bosatsu-managed prefix under .bosatsuc, and cc_conf.json should then point at those local include and library paths directly.
We should not depend on:
- host
pkg-config; - host-wide library search paths;
- global
brew/apt/vcpkgstate; - dynamic lookup of
libgcorlibuvat executable run time.
Different C libraries have different build systems and different platform quirks. Bosatsu should represent that with explicit dependency recipes rather than pretending every library is built the same way.
That recipe layer is also where we record:
- how to configure the library;
- which artifacts to expect after install;
- which additional system link flags remain necessary;
- which compile defines Bosatsu runtime code must honor when using that library.
The Bosatsu runtime archive should contain a checked-in dependency manifest, for example c_runtime/deps.json or c_runtime/deps.toml.
At minimum each dependency entry should carry the conceptual tuple already discussed:
nameversionurihash
The actual serialized model should be slightly richer so Bosatsu can evolve cleanly. A good shape is:
{
"schema_version": 1,
"recipe_version": 1,
"dependencies": [
{
"name": "bdwgc",
"version": "8.2.8",
"uris": [
"https://github.com/bdwgc/bdwgc/releases/download/v8.2.8/gc-8.2.8.tar.gz"
],
"hash": "blake3:...",
"source_subdir": "gc-8.2.8",
"recipe": "bdwgc-cmake-static",
"options": {
"threadsafe": true
}
}
]
}Notes:
urisis preferable to a singleuriso we can add mirrors later without changing the model.recipe_versionis a Bosatsu-owned invalidation knob. If our build recipe changes, old cached builds should stop matching even if the source archive is unchanged.recipeidentifies Bosatsu-owned logic for configuring/building/installing that dependency.optionslets the manifest pin Bosatsu-relevant features, such as “thread support must be enabled”.
The manifest should live inside the Bosatsu C runtime source tree and therefore inside the runtime archive that bosatsu c-runtime install downloads or consumes locally.
That means:
- the Bosatsu release process produces a runtime archive that contains:
- the runtime source files
- the Bosatsu runtime
Makefileor later build files - the dependency manifest
bosatsu c-runtime installunpacks the archive;- Bosatsu reads the manifest from the unpacked runtime tree before building Bosatsu runtime artifacts.
This is the right placement because:
- the manifest is part of the Bosatsu runtime definition for that release;
- different Bosatsu runtime versions can pin different dependency sets or options;
- the dependency cache can still be shared across Bosatsu versions because the build reuse key is not the Bosatsu version itself.
In practice, Bosatsu version selects the manifest, but previously built dependencies are reused if the manifest entries resolve to the same source identity and build identity.
The cache should distinguish:
- downloaded archives;
- extracted source trees;
- built/install prefixes;
- runtime-version-specific Bosatsu artifacts.
Proposed layout:
.bosatsuc/
c_deps/
archives/
<name>/
<version>/
<hash>/
source.tar.gz
sources/
<name>/
<version>/
<hash>/
<source tree>
builds/
<build-key>/
prefix/
include/
lib/
metadata.json
build.log
success
locks/
<build-key>.lock
<runtime-id>/
include/
lib/
cc_conf.json
Where:
c_deps/is shared across Bosatsu runtime/compiler versions;<runtime-id>/remains Bosatsu-version-specific and contains Bosatsu’s own runtime artifacts such asbosatsu_platform.aand the generatedcc_conf.json.
This split is important because the current installer deletes the versioned runtime directory on reinstall. Third-party dep caches should live outside that per-runtime directory so they survive Bosatsu bumps and reinstalls.
A dependency build should be reused if and only if all of the following match:
- dependency name;
- dependency version;
- dependency source archive hash;
- Bosatsu recipe identifier;
- Bosatsu recipe version;
- target OS;
- target architecture;
- toolchain family;
- toolchain version fingerprint;
- build profile;
- relevant dependency build options;
- transitive dependency build keys, if any.
To be conservative, the toolchain fingerprint should include enough information to avoid ABI mismatches. For phase 1 that can be:
- resolved compiler path;
- compiler
--versionoutput; - archiver identity if we invoke a non-default
ar/lib; - environment knobs that materially affect output.
This means a Bosatsu version bump will not invalidate the cache, but an Xcode/Clang upgrade or GCC family switch usually will.
Dependency builds should keep release and debug separate. Even if the API/ABI is nominally compatible, debug-oriented builds often carry different flags, assertions, and symbol/debug settings.
Each successful dependency build should write a Bosatsu-owned metadata.json next to the installed prefix. Bosatsu should consume this metadata rather than rediscovering the library from the host.
Suggested shape:
{
"schema_version": 1,
"name": "bdwgc",
"version": "8.2.8",
"build_key": "...",
"target": {
"os": "macos",
"arch": "arm64",
"toolchain_family": "clang",
"toolchain_version": "Apple clang 16.0.0 ..."
},
"prefix": "/abs/path/.bosatsuc/c_deps/builds/<build-key>/prefix",
"include_dirs": [
"/abs/path/.bosatsuc/c_deps/builds/<build-key>/prefix/include"
],
"static_libs": [
"/abs/path/.bosatsuc/c_deps/builds/<build-key>/prefix/lib/libgc.a"
],
"system_link_flags": [
"-pthread"
],
"runtime_requirements": {
"bosatsu_runtime_cppflags": [],
"generated_c_cppflags": []
}
}Important separation:
static_libsare the vendored artifacts;system_link_flagsare non-vendored OS/toolchain libraries or frameworks still required at final link;runtime_requirementsare requirements Bosatsu runtime code must honor when using that dependency.
This separation becomes important for libuv, because even if libuv.a is vendored, some platform system libraries/frameworks may still need to be linked.
Bosatsu should model platform support at three layers:
- archive extraction;
- dependency recipe execution;
- final runtime compile/link configuration.
Phase 1 can still use the existing shell-heavy flow on macOS/Linux, but the dependency subsystem should not hard-code Unix assumptions into its API. In particular:
- dependency recipes should be keyed by platform and toolchain family;
- archive extraction should not permanently assume
tarandunzipare present; - recipe execution should not permanently assume GNU
make; - final metadata should be platform-neutral even if the phase 1 implementation is Unix-only.
macOS phase 1 requirements:
- support Apple Clang builds;
- build per-architecture artifacts rather than universal libraries by default;
- prefer vendored static archives under
.bosatsuc; - avoid relying on Homebrew installation paths for
bdw-gcorlibuv.
Implications:
arm64andx86_64should be separate build keys;- any remaining system frameworks needed by a vendored dependency should be recorded in Bosatsu metadata as final link flags;
- the design should not depend on
brewat all once vendoring is in place.
Linux phase 1 requirements:
- support common GCC/Clang toolchains;
- keep vendored static archives under
.bosatsuc; - allow final link metadata to include required system libraries such as threading or dynamic loader flags when applicable.
Implications:
- build keys should include compiler family/version and target architecture;
- the recipe layer should be allowed to record extra system flags instead of assuming the static archive is fully self-contained;
- no
aptor distro package should be required for vendoredlibgcor vendoredlibuv, except general build tools.
Windows is an explicit design target, but not a phase 1 deliverable. The design should make it possible without a second architectural rewrite.
Requirements:
- do not force Unix-only extraction/build APIs into the dependency subsystem;
- support multiple toolchain families in principle:
- MSVC
- clang-cl
- MinGW
- keep toolchain family in the build key;
- allow dependency metadata to emit
.libpaths and Windows system libraries instead of.aplus Unix-style flags.
Important caveat:
- the current Bosatsu runtime install path shells out to
make,tar, andunzip; - that is not a reliable long-term Windows foundation;
- phase 1 can keep that for macOS/Linux, but the vendored dependency subsystem should be designed so Bosatsu runtime build itself can later move to a CMake-driven or Bosatsu-owned build path.
Bosatsu’s current runtime already depends on Boehm GC, so libgc is the first dependency to vendor.
Per upstream bdwgc documentation:
- CMake is the recommended build path;
- the collector is built with thread support enabled by default unless explicitly disabled;
- on Windows/MSVC,
libatomic_opsmay still be needed because compiler intrinsics support differs there.
Bosatsu requirements for vendored libgc:
- build a static library;
- ensure thread support is not disabled;
- record the installed headers and static archive in dep metadata;
- record any required system link flags in dep metadata;
- do not rediscover
libgcvia hostpkg-configduring runtime install.
There are two separate concerns:
- how
libgcitself is built; - how Bosatsu runtime code is compiled when it uses
gc.h.
For phase 1:
- current Bosatsu runtime can keep its existing behavior, because it is not yet using
libuv; - vendored
libgcshould still be built with thread-safe support enabled, since that is the correct base for the upcominglibuvbackend; - phase 1 does not need to define
GC_THREADSeverywhere immediately if the current runtime backend does not require it.
For the future libuv backend:
- Bosatsu runtime translation units that include
gc.hand participate in multithreaded execution should defineGC_THREADS; - Bosatsu-generated C that includes
gc.hshould also receive the relevant compile define in the libuv-enabled runtime configuration; GC_allow_register_threads()and explicit worker thread registration/unregistration belong to Bosatsu runtime code, not to the vendored dependency recipe.
The vendored dependency system must be able to model transitive dependencies because bdwgc can require libatomic_ops on some toolchains, especially Windows/MSVC.
We should therefore not hard-code an assumption that every vendored dependency is leaf-only.
Phase 1 may start with the common macOS/Linux case where libgc builds without a separate vendored libatomic_ops, but the recipe/data model should allow:
bdwgcdepending onlibatomic_ops;- the
bdwgcbuild key depending on thelibatomic_opsbuild key.
libuv is not yet a Bosatsu runtime dependency, but the vendoring design should support it cleanly.
Per upstream libuv documentation:
- it has a global threadpool;
uv_queue_work()runs work callbacks on threadpool threads;- CMake is the supported build system on Windows and a supported build system on Unix-like platforms.
Bosatsu vendoring requirements for libuv:
- build a static library under
.bosatsuc; - disable or skip tests/benchmarks/docs in vendored builds where appropriate;
- record installed headers, static library path, and any remaining system link flags in dep metadata;
- avoid host discovery through system
pkg-config.
The main libuv/libgc integration constraint is not a libuv build flag. It is a Bosatsu runtime contract:
libuvworker threads are third-party-created threads fromlibgc’s perspective;- any such worker thread that touches GC-managed heap values must be explicitly registered with Boehm;
- those threads must be unregistered before returning to
libuv; - Bosatsu runtime code using the libuv backend should compile with
GC_THREADS.
That means the design must distinguish:
- dependency build metadata for
libuv; - dependency build metadata for
libgc; - runtime backend requirements for “Bosatsu with libuv”.
We should not encode GC_THREADS as a blanket libgc client requirement for every backend. It is specifically a requirement of Bosatsu’s future multithreaded/libuv runtime mode.
We can integrate vendored dependencies incrementally without rewriting the whole pipeline first.
bosatsu c-runtime installdownloads and unpacks the Bosatsu runtime archive as today.- before invoking
make install, Bosatsu reads the vendored dependency manifest from the runtime source tree. - Bosatsu ensures all declared dependencies are present in
.bosatsuc/c_deps/.... - Bosatsu aggregates dependency metadata into final include and link flags.
- Bosatsu invokes the Bosatsu runtime build with those exact flags.
install.pywritescc_conf.jsonusing the passed-in local include/library paths instead of rediscoveringlibgcfrom the host.
This preserves most of the current flow while removing host dependency discovery.
The Bosatsu runtime build step should receive:
- dependency include flags derived from vendored metadata;
- dependency library paths derived from vendored metadata;
- dependency-required system link flags derived from vendored metadata;
- runtime backend-specific compile defines such as future
GC_THREADS.
The easiest incremental step is to keep the current CcConf schema and just change its contents.
That means:
iflagsshould point at.bosatsuc/<runtime-id>/includeand vendored dependency include prefixes;libsshould start with Bosatsu’s ownbosatsu_platform.a, then vendored dependency archives in dependency order, then required system link flags;- no later C compile step should need to rediscover
libgcorlibuv.
c_runtime/install.py should become dumber, not smarter.
Instead of trying to rediscover bdw-gc itself, it should:
- accept the exact compile/link inputs passed by the Bosatsu installer;
- copy Bosatsu runtime headers and archive into
.bosatsuc/<runtime-id>/; - write those exact paths/flags to
cc_conf.json; - avoid host
pkg-configfallback logic for official vendored installs.
We may keep a developer-only fallback path temporarily, but the supported install path should stop depending on host discovery.
Bosatsu should add explicit recipe implementations, for example:
bdwgc-cmake-staticlibuv-cmake-static
Each recipe is responsible for:
- taking a verified source tree and a target build context;
- configuring/building/installing into a temp prefix;
- validating expected outputs;
- writing Bosatsu metadata;
- atomically moving the successful prefix/metadata into the shared build cache.
Dependency build invocation should be specified in two layers:
- the manifest chooses a Bosatsu-owned recipe and recipe options;
- Bosatsu code implements that recipe and executes the actual commands.
The manifest should not embed arbitrary shell commands such as raw ./configure, cmake, or make command lines.
Instead, the manifest should say things like:
- build dependency
bdwgcwith recipebdwgc-cmake-static; - enable thread-safe collector support;
- use these source hashes and these accepted source URIs.
Then Bosatsu-owned installer code translates that recipe into the concrete build steps for the current platform/toolchain.
For example:
- the
bdwgc-cmake-staticrecipe might invoke CMake configure/build/install steps; - the
libuv-cmake-staticrecipe might invoke a different set of CMake flags and validation rules; - later Windows-specific variants can reuse the same recipe identity but branch internally by toolchain family if needed.
This split keeps the manifest:
- declarative;
- stable across platforms;
- safe to evolve without shipping arbitrary command execution in release metadata.
It also keeps platform-specific command construction in Bosatsu code where it can be tested and versioned.
The dependency build orchestration should not primarily live in c_runtime/install.py.
Recommended responsibility split:
- Bosatsu installer code in Scala:
- reads the dependency manifest
- computes build keys
- fetches and verifies archives
- checks and populates the shared dependency cache
- executes dependency recipes
- gathers dependency metadata
- invokes the Bosatsu runtime build with the resulting include/link inputs
c_runtime/install.py:- copies Bosatsu runtime outputs into
.bosatsuc/<runtime-id>/ - writes
cc_conf.jsonfrom the exact arguments it is given - does not rediscover or build third-party dependencies in the supported path
- copies Bosatsu runtime outputs into
So the intended flow is:
bosatsu c-runtime installunpacks the Bosatsu runtime archive;- Scala-side installer code reads
c_runtime/deps.json; - Scala-side installer code ensures each dependency build exists in
.bosatsuc/c_deps/...; - Scala-side installer code invokes the Bosatsu runtime build using those vendored prefixes;
install.pyrecords the final Bosatsu runtime install and the finalcc_conf.json.
This keeps install.py narrow and avoids pushing dependency resolution logic into a script that currently only packages the final runtime outputs.
Dependency builds should be safe under concurrent Bosatsu invocations.
Recommended rules:
- build into a temp directory outside the final cache key path;
- acquire a per-build-key lock before publishing;
- publish by atomic rename/move into the final
<build-key>/directory; - reuse a cached build only if
metadata.jsonand a success marker are both present and valid.
Status: completed on 2026-04-04.
Deliverables:
- add a vendored dependency manifest to
c_runtime/; - add Bosatsu-side manifest parsing, metadata encoding, and build-key computation under
dev.bosatsu.cruntime; - seed the manifest with vendored
bdwgc; - add unit tests for manifest parsing, build-key stability, and metadata round-tripping.
Delivered in this phase:
c_runtime/deps.jsonnow exists and is part of the shipped runtime tree;core/src/main/scala/dev/bosatsu/cruntime/CDeps.scalanow owns the manifest schema, metadata schema, toolchain normalization helpers, and build-key hashing;core/src/test/scala/dev/bosatsu/cruntime/CDepsTest.scalacovers the pure data-contract pieces.
Status: completed on 2026-04-04.
Delivered in this phase:
bosatsu c-runtime installnow readsc_runtime/deps.json, computes a shared build key, and materializes vendored dependency state under.bosatsuc/c_deps/...;bdwgc-cmake-staticis implemented and installs into a Bosatsu-managed prefix instead of relying on host package discovery;- the runtime
Makefilenow has a supported vendored-dependency path that bypasses hostpkg-config/Homebrew lookup; install.pynow records the exact vendored include and library paths it is given, rather than rediscoveringbdw-gcitself;- local verification shows:
- the first install builds and caches
bdwgc; - a second install with a different runtime id reuses the cached
libgc.abuild.
- the first install builds and caches
Concrete file areas likely touched:
core/src/main/scala/dev/bosatsu/cruntime/Command.scala- new
core/src/main/scala/dev/bosatsu/cruntime/...support files for manifest parsing, build keys, recipes, and metadata c_runtime/Makefilec_runtime/install.py- new
c_runtime/deps.jsonorc_runtime/deps.toml - tests for installer/cache behavior
Recommended sequence:
- implement archive fetch and source extraction reuse;
- implement cache layout under
.bosatsuc/c_deps/; - implement the
bdwgc-cmake-staticrecipe and metadata output; - plumb vendored metadata into the runtime build and
cc_conf.json; - delete or bypass host
bdw-gcdiscovery in the supported installer path; - add install-path verification.
Status: completed on 2026-04-04.
Delivered in this phase:
- manifest dependency ordering now goes through explicit dependency-graph validation with missing-dependency and cycle errors;
- transitive dependency build keys now participate in cache identity, so future dependency edges can invalidate builds correctly;
- the recipe/metadata layer remains recipe-driven and now has direct tests around graph ordering, transitive build-key changes, and
.pcsystem-flag parsing; - the working vendored install path was rechecked after this refactor and still reused the cached
bdwgcbuild.
Deliverables:
- extend dependency resolution from “declared order only” to explicit dependency graph ordering with cycle detection;
- include transitive dependency build keys in cache identity so future
libatomic_opsorlibuvedges invalidate correctly; - keep the recipe/metadata layer generic enough that
libuv-cmake-staticis just another recipe entry, not a new architecture; - add tests for dependency ordering, cache-key invalidation through transitive deps, and metadata/system-flag parsing.
This phase does not require the libuv backend itself to land at the same time, but it should make adding it straightforward.
The dependency vendoring issue stops at shared-cache third-party dependency management. The following remain follow-up items after this issue lands:
- adding the actual
libuvdependency entry and recipe implementation; - building the libuv-backed Bosatsu runtime backend itself;
- replacing Unix-specific runtime build assumptions for full Windows runtime support.
- manifest parsing tests;
- build-key stability tests;
- cache hit/miss tests showing Bosatsu version bumps do not invalidate vendored dep reuse;
- metadata serialization/deserialization tests;
- invalid cache entry handling tests.
- first install downloads, builds, and publishes vendored
libgc; - second install with same manifest/toolchain/profile reuses cached
libgc; - changing Bosatsu runtime version but not dependency source/build context still reuses cached
libgc; - changing dependency hash causes a rebuild;
- changing recipe version causes a rebuild;
- changing toolchain fingerprint causes a rebuild.
- verify
cc_conf.jsonpoints at.bosatsucpaths rather than hostpkg-configoutput; - verify a generated executable links successfully without host-installed
libgc; - once
libuvis added, verify equivalent behavior for vendoredlibuv.
- macOS CI path for vendored
libgc; - Linux CI path for vendored
libgc; - when Windows work starts, add toolchain-specific CI separately rather than assuming Unix tests are enough.
bosatsu c-runtime installno longer requires host-installedbdw-gc.cc_conf.jsonno longer records host-discoveredbdw-gcinclude/library paths for the supported path.- third-party dependency builds are reused across Bosatsu compiler/runtime version bumps when the dependency source identity and build context match.
- a changed dependency source hash or build context causes a rebuild.
- the dependency subsystem can represent multiple dependencies, dependency order, and future transitive dependencies.
- the design cleanly accommodates future vendored
libuv. - the design leaves room for Windows without requiring another architectural rewrite.
-
Risk: we accidentally key the shared dependency cache too loosely and reuse incompatible artifacts. Mitigation: include toolchain family/version, target arch, build profile, recipe version, and dependency options in the build key.
-
Risk: vendored static archives still need non-vendored system link flags and builds fail late. Mitigation: make system link flags an explicit part of dep metadata instead of assuming the archive is fully self-contained.
-
Risk:
libgcrecipe needslibatomic_opson some platforms/toolchains. Mitigation: model transitive dependencies in the recipe/data model from the start, even if phase 1 common Unix builds do not need them. -
Risk: current installer remains Unix-specific. Mitigation: keep phase 1 scoped to macOS/Linux, but define the dependency subsystem in a platform-neutral way and plan explicit Windows enablement later.
-
Risk: recipe logic drifts from upstream-supported build modes. Mitigation: use upstream-recommended build systems where available, especially CMake for both
bdwgcandlibuv.
The first practical cut should be:
- add
c_runtime/deps.jsonwith a single vendoredbdwgcentry; - add
.bosatsuc/c_deps/shared cache layout; - implement
bdwgc-cmake-staticrecipe for macOS/Linux; - change
bosatsu c-runtime installto build/reuse vendoredlibgc; - change
install.pyandcc_conf.jsongeneration to use the vendored prefix directly; - keep the rest of the Bosatsu C compile/link pipeline unchanged.
That gives immediate value for the current dependency, solves the repeated-rebuild problem across Bosatsu bumps, and leaves the door open for libuv without forcing a second redesign.