main has failing and flaky tests: 6 in cook::tests, 1 in http_api

[chubes4]

Found while verifying unrelated branches during the tech-debt sweep — every branch I checked showed these, and none of them caused them.

homeboy-core — 1 deterministic failure

test http_api::http_api_test::artifact_content_serves_encoded_artifact_store_locator ... FAILED

Confirmed on origin/main at 0cad5a542 with a clean, isolated target dir. Fails identically on branches that do not touch http_api.rs.

homeboy-agents — 6 failures in cook::tests, and they are flaky

On origin/main:

cook_successful_concrete_attempt_publishes_reviewer_body
manual_preflight_intent_does_not_block_normal_cook_finalization
manual_preflight_recovers_without_a_persisted_promotion_and_rejects_tampering
recovered_cook_finalization_uses_latest_resumed_gate_contract
re_materialize_follow_up_baseline_recovers_after_worktree_deletion
replacement_gate_proof_recovers_failed_candidate_without_hiding_evidence_or_republishing

They are not stable. Consecutive runs of the same tree produced 9, then 5, then 6 failures, with overlapping but non-identical sets. The names point at worktree materialization, baseline re-materialization, and gate-proof recovery — all filesystem- and ordering-sensitive.

Why this matters more than the count

I only detected these by diffing failure lists between a branch and its merge base. A branch that genuinely broke one of these tests would be indistinguishable from the noise, and a reviewer seeing "6 failed" on their branch would reasonably assume it was theirs.

It also interacts badly with two things the sweep just landed:

• CI shards are --changed-since-scoped, so a given shard may never run these.
• #11835 bounded several poll loops with deadline-based tests. Flaky wall-clock tests in the same suite make it harder to tell a real timing regression from this background noise.

Suggested handling

1. Fix or #[ignore] the deterministic http_api failure — a permanently-red test teaches people to ignore red.
2. Triage the cook::tests six as a group. If they share a fixture-isolation root cause (likely, given the names), one fix probably covers them.
3. Until then, consider a documented known-failing list so a branch author can diff against it, which is what I had to reconstruct by hand.

[chubes4]

Update: the deterministic half is fixed; the scale was understated

#11919 fixes 8 of these. What is left is larger and more important than this issue originally described.

What the 6 "flaky" cook tests actually were

Five were deterministic, not flaky, and shared one root cause: the promotion() fixture hardcoded /repo, and a50702c9d began validating that --path overrides exist. One fix, five tests. cook::tests is now 132 passed, 0 failed.

Only re_materialize_follow_up_baseline_recovers_after_worktree_deletion was genuinely order-dependent — and it is not a product defect. It passes in isolation, under --test-threads=1, and under nextest. HomeGuard serializes writers via home_lock() but readers never take it, so a concurrent home repoint can be observed between that test's deliberate worktree deletion and its re-materialization. CI runs nextest and does not see it. Documented in place.

Two more came from a single stale assumption: with_isolated_home sets HOMEBOY_DATA_DIR, which homeboy_data() checks before the XDG fallback, so tests hardcoding <home>/.local/share/homeboy/... look in the wrong place. homeboy_data()'s own comment claimed the two resolve identically — false, and now corrected.

The actual scale

I originally counted from filtered runs. The unfiltered homeboy-core --lib suite has 35 failures on main (34 after #11919):

module	count
daemon::*	8
observation::store::*	6
cleanup::*	5
extension_execution / lab_routing	4
deps / engine / git::pr_land / server::client / setup / controller_runtime	6

That is homeboy-core alone. homeboy-agents, homeboy-cli and the integration binaries are not counted here.

The two things that make this urgent

1. A branch that genuinely breaks something is indistinguishable from the background. I could only establish that my own changes were clean by reconstructing main's failure list and diffing it — not a reasonable thing to ask of every author, and not something a reviewer will do.
2. Nobody can currently tell real from environmental. Some are clearly host-dependent: server::client::tests::timed_piped_ssh_envelope... needs ssh, setup::tests::setup_installs_configured_extensions_from_source needs network, cleanup::self_artifacts::* needs a managed checkout. But others look real — daemon::daemon_test::controller_job_errors_use_driver_safe_public_projection fails a plain assertion on driver-safe public projection, and that is exactly the surface feat(agent-task): make the detached local cook a daemon-owned controller job #11857 and feat(agent-task): make the detached local fanout a daemon-owned controller job #11871 now depend on for redaction of cook and fanout job state.

Also confirmed pre-existing and genuinely flaky in isolation (2 of 3 runs): daemon::control::control_test::candidate_attribution_reads_command_env_store_identity_from_procfs.

Suggested next step

Triage the remaining 34 into environmental vs real, starting with the daemon::daemon_test controller-job group, since those guard the redaction and cancellation semantics the recent orchestration work relies on. A documented known-failing list would let authors diff against it in the meantime — which is what I had to build by hand.

[chubes4]

daemon:: triage complete — #11927 fixes 4 more

Running total across #11919 + #11927: 12 tests fixed, 0 new failures.

The key result: these are not one problem

The daemon:: group alone decomposes into five distinct causes, and most are test debt rather than product defects:

cause	n	status
path drift (HOMEBOY_DATA_DIR vs hardcoded XDG)	4	fixed (2 in #11919, 1 in #11927, 1 earlier)
racing a documented async contract	3	fixed (#11927)
process-global registry interference	2	not a defect — pass in isolation, CI immune
host-dependent flake (/proc reads)	1	not a defect
genuine, needs investigation	2	open

On the redaction concern I raised earlier

I flagged controller_job_errors_use_driver_safe_public_projection as worrying because #11857/#11871 depend on that surface. Redaction was never broken. All three of its no-leak assertions were already passing; only the "safe classification is present" assertion failed, because the supervisor thread had not yet written the projection. Race, not leak.

Not caused by the sweep

The three controller-job failures already fail at 99e15769c, the commit the sweep started from, and pass at 2778da01b. They regressed in between, before any of this work.

The 2 that are genuinely broken

Both fail in isolation, so neither is interference or timing:

• stop_refuses_stale_lease_that_points_at_reused_pid
• legacy_child_recovery_exact_evidence_is_idempotent_and_conflicts_fail_closed — asserts a recovery count of 1, gets 0

Both concern daemon lease and child-recovery semantics. I deliberately did not attempt fixes: unlike the cancel race, the intended contract is not documented anywhere I could point to, and this is the subsystem the sweep independently found to be an implicit state machine spread over 8 files with 14 recovery entry points and no explicit state type. Guessing at lease-recovery semantics is how a daemon becomes subtly wrong.

These two are the real remaining work in this group, and they would be far easier to reason about after the daemon lifecycle state machine is made explicit.

Method note for whoever continues

The 0..N { if cond { break } sleep(ms) } idiom appears throughout these tests with 250–500ms budgets. On a loaded host that is not enough, and it fails as an assertion rather than as a timeout — which reads like a logic bug. #11927 replaces four of them with a bounded wait_for(label, condition) at 10s. Worth applying wherever else that idiom appears.

[chubes4]

observation::store + cleanup triage — #11941 fixes 7 more

Running total: 19 tests fixed across #11919, #11927, #11941. Zero new failures introduced. homeboy-core --lib: 35 → 23.

main did not compile again

release_readiness_source was added to both TestArgs and LintArgs without updating tests/self_checks_test.rs. Same E0063 class as #11767, second occurrence this session. Anyone running cargo check --workspace --tests was blocked, which is the gate that would have caught it.

The store failures were three unrelated bugs, not one

• FK-enforcement drift (3) — seed_owned_and_orphaned_children creates orphaned rows on purpose; that is what migration 13 reaps, and one test is named ..._orphaned_while_enforcement_was_off. enforce_foreign_keys now enables enforcement at open, so the seed died with FOREIGN KEY constraint failed before the assertions ran.
• Path drift (2) — instances 5 and 6 of the HOMEBOY_DATA_DIR vs XDG assumption. test_database_path additionally set XDG_DATA_HOME and asserted an XDG path that could never resolve: the home-root override outranks XDG deliberately and the harness always registers one. It was pinning a resolution order that does not exist.
• Swapped assertions (1) — test_record_directory_artifact asserted the tree digest in url and None in sha256; the code writes it to sha256, which is also what the idempotent-reuse check compares.

cleanup::self_artifacts (5) — environmental, dismissible

They require a managed Homeboy source checkout to own the running test binary. I tested the obvious fix by building target/debug/homeboy — it did not help. They need real host state an isolated home lacks by construction, so they cannot pass in a hermetic run. These should probably be #[ignore]d with a reason rather than left red forever.

(Incidental data point: that binary is 816 MB, corroborating the ~700 MB figure #11775's lazy-fixture change depends on.)

The remaining genuine defects — now a short, specific list

This is what is actually worth engineering time:

1. stop_refuses_stale_lease_that_points_at_reused_pid — daemon lease semantics
2. legacy_child_recovery_exact_evidence_is_idempotent_and_conflicts_fail_closed — recovery count 1 expected, 0 observed
3. publication_manifest_artifact_store_refs_are_indexed — manifest-referenced artifacts are not being indexed
4. publication_manifest_public_url_refs_are_indexed_from_artifact_origin — same subsystem
5. migration_from_version_8_preserves_artifacts_and_adds_query_indexes — apply migration 13: no such table: findings. Possibly a real migration bug: migration 13 assumes findings exists, and a database legitimately at v8 may not have it. Same migration whose fixtures fix(test): unbreak the observation store suite and main's test build #11941 just repaired.

1 and 2 sit in the daemon lease/recovery subsystem the sweep independently flagged as an implicit state machine across 8 files with 14 recovery entry points and no explicit state type — they would be far easier to reason about after that is made explicit.

Final tally of the decomposition

cause	n	status
path drift (HOMEBOY_DATA_DIR vs XDG)	6	fixed
racing a documented async contract	3	fixed
FK-enforcement drift	3	fixed
swapped assertions	1	fixed
unreachable test premises	2	fixed
compile breaks (E0063)	2	fixed
process-global interference	2	not defects, CI immune
host-dependent	6	not defects
genuine defects	5	open, listed above

The headline stands: "the suite is red" was mostly test debt with a small real core. That core is now identified and small enough to fix deliberately.