HARPER_SET_CONFIG applied after component loading (thread-timing race) and not persisted at install — env-var-configured components silently fail to register

[heskew]

Summary

HARPER_SET_CONFIG (and by extension env-var-driven config) is unreliable for anything component-loading reads at boot: it is applied to the in-memory config after components load on some platforms/timings, and the install-time application does not persist it to the written config file. Net effect: a component key delivered via env var may be silently ignored — the component never registers, no error, no warning.

Found while debugging PR #1616's integration-test 404s (the modelsGateway component); the mechanism is general and affects any env-var-configured deployment (Docker/Fabric-style) activating or configuring components via HARPER_SET_CONFIG.

Evidence

1. Ordering race (Linux CI, deterministic there; opposite on macOS). From the failing suite's Harper instance log (PR #1616, run 28810803532, shard 4/6):

17:37:26.137 [main/0] [info]:  All root applications loaded          ← componentLoader ran
17:37:26.494 [http/1] [debug] [env-config]: Applied HARPER_SET_CONFIG at runtime   ← merged after
17:37:26.507 [http/1] [debug]: Config file updated with runtime env var values

Components were loaded from a config that did not yet contain the env-supplied key → /v1/* 404s. The same boot on macOS serves 200 — the runtime application happens to win there. Same code, thread-timing-dependent outcome.

2. Install-time application doesn't persist. config/configUtils.ts:163 calls applyRuntimeEnvVarConfig(configDoc, null, { isInstall: true }) before writing the fresh config file — but after a fresh local install with HARPER_SET_CONFIG='{"modelsGateway":{"enabled":true}}', the written harperdb-config.yaml does not contain the key (verified directly). So even a fresh install under env config produces a file that component loading can't see the env config in, leaving only the racy runtime path (configUtils.ts:341).

(There is also a known sub-issue that flattenObject in config/harperConfigEnvVars.ts silently drops empty-object values like modelsGateway: {} — no leaf paths — noted in PR #1616's history. The non-empty {enabled: true} form still hits the two problems above, so fixing the flatten alone is insufficient.)

Why this matters

• Env-var config is the primary mechanism for containerized deployments and the integration-test harness (@harperfast/integration-testing passes suite config exclusively via HARPER_SET_CONFIG — see its harperLifecycle).
• Silent mis-registration: nothing logs that a component key arrived too late to matter.
• Direction from Nathan: fix the core loading order properly rather than working around it (e.g., PR feat(models): OpenAI-compatible /v1/* gateway as built-in Resources (#631, Phase 4 of #510) #1616 deliberately keeps its component default-off and its integration test depends on this fix; enabling things by default to dodge the race is not an acceptable pattern).

Suggested shape of the fix

• Apply HARPER_DEFAULT_CONFIG / HARPER_CONFIG / HARPER_SET_CONFIG synchronously during initial config resolution on every thread, before any component loading — i.e. the resolved config object handed to componentLoader must already include env-var config, deterministically, on all platforms.
• Make the install-time application actually persist into the freshly written config file (or make the boot path not depend on the file for env-supplied values).
• Keep the existing drift-detection/restore semantics (hasPersistedEnvConfigState) intact.
• Consider a debug log when env config introduces keys that affect component activation, to make future ordering regressions visible.

Acceptance

• Fresh boot with HARPER_SET_CONFIG='{"modelsGateway":{"enabled":true}}' registers the component on Linux and macOS, first boot and subsequent boots.
• PR feat(models): OpenAI-compatible /v1/* gateway as built-in Resources (#631, Phase 4 of #510) #1616's integration test (integrationTests/server/v1-gateway.test.ts) passes in CI without workarounds.
• Empty-object config values survive the env-var path or are rejected loudly (no silent drop).

Refs

• PR feat(models): OpenAI-compatible /v1/* gateway as built-in Resources (#631, Phase 4 of #510) #1616 (blocked integration test), issue [Models] Phase 4 — OpenAI-compatible /v1/* gateway as built-in Resources #631 / Add unified model-access API (scope.models) #510.
• config/configUtils.ts:163 (install-time apply), :341 (runtime apply), :915 (applyRuntimeEnvVarConfig); config/harperConfigEnvVars.ts (flattenObject, applyRuntimeEnvConfig); components/componentLoader.ts:387-394 (config-keyed component loading).

---

🤖 Generated with Claude Code

[kriszyp]

Characterization from exploratory QA (harper 228eacc0f) that may be the crux here: HARPER_SET_CONFIG env config is continuously env-gated, not install-time-persisted.

• No registration race: an env-only-configured component (a non-default componentsRoot set solely via HARPER_SET_CONFIG) registered cleanly on cold boot (1 worker), 4-worker boot (80 concurrent hits → all 200, all 4 worker threads served, no 404 subset), and warm restart.
• But removing the env var and restarting → get_configuration.componentsRoot is absent and the routes 404. config/harperConfigEnvVars.ts cleanupRemovedEnvVar() deliberately un-applies (restores the original, or deletes if no original was recorded) any path the env var owned; componentsRoot's on-disk default is null and storeOriginals skips recording null → the key is deleted on removal.

So env config is a "force/lock-against-drift" mechanism, not install-time persistence. That's plausibly the surprise users are filing against — if the expectation is that install-time env config becomes permanent, that's a doc-vs-behavior gap (or a request to make it persistable). Flagging the design intent so it can be documented or changed deliberately.

Repro: local qa-scratch/qa473-env-config-boot.test.ts.

[kriszyp]

Appears to be a duplicate of / consolidate under #1513 — root config composed/memoized before components load.

— Claude (Opus 4.8), issue-backlog triage

[heskew]

Root cause pinned via instrumented CI runs on #1616 (the failure reproduces ~50% per run on Linux CI, run-to-run on identical code) — and it's neither of the mechanisms this issue originally hypothesized, so corrections first:

• My original "install-time non-persistence" evidence was invalid — it grepped harperdb-config.yaml; the file is harper-config.yaml. With the right filename, install-time persistence works.
• The QA characterization above (continuously env-gated, no registration race for an env-only componentsRoot) is confirmed and consistent: that key is read by the componentLoader from the resolved in-memory config, which is reliably env-merged on every thread.

The actual bug: scope.options and the componentLoader read different sources. OptionsWatcher (what every scope.options is) re-parses the config file from disk; the loader uses the resolved in-memory config. At boot the env-applied values reach the file only when the first thread's runtime re-apply flushes them ([http/N] Config file updated with runtime env var values), so a component's boot-time scope.options read — e.g. handleApplication's enabled gate — races the flush. Probe evidence from a failing run: componentLoader logged modelsGateway config={"enabled":true} on both main/0 and http/1, yet the routes 404'd because scope.options.get(['enabled']) saw the pre-env file. macOS wins the race; Linux CI loses it about half the time.

Fix PR incoming on this issue's branch: OptionsWatcher composes the env layers (composeConfigFromEnv, side-effect-free, same precedence as the runtime pipeline) over every root-config read, including the install window where the file doesn't exist yet; application config.yaml scopes untouched. Plus a once-per-path warning when an env-var config value is an empty object (componentName: {} flattens to nothing — that's load-bearing for the restore-on-removal semantics, so it can't be "fixed", but it should never again be silent).

On consolidation with #1513: same family, distinct layer — #1513 is loadEnv-delivered config arriving after the root config is composed at all; this is the scope.options view diverging from the already-composed config. The fix here doesn't close #1513. Happy to fold this issue in as a sub-item if that's still preferred once the PR is reviewed.