fix(reconciler): global score-descending pairing prevents false-positive feature deactivations - #188
Merged
Merged
Conversation
…ive feature deactivations The reconciler's matcher was per-write greedy first-fit: every synthesised write picked the best candidate that no earlier write had already claimed. With realistic inputs this routinely orphaned the candidate that would have been a clean pair for a later write while a borderline-match earlier write got the credit, then the orphaned candidate was swept inactive. Concrete prod evidence (ATOAPayment, 2026-06-01 PR-3220 merge): 28 features soft-deleted in one scan; at least 10 of them had a still-active twin with embedding cosine >= 0.85, several with identical titles -- "Cache Service" <-> "Cache Service" (0.857), "Vendor Rollout Flags" <-> "Vendor Rollout Flags" (0.845), "Upload Analytics Events" <-> "Analytics Upload Events" (0.853), "Pricing & Merchant Plans" <-> "Merchant Pricing & Plans" (0.876), and more. Replace _match_strategy (per-write greedy) with _resolve_pairings (global within-tier resolution). Tier order is preserved: signature -> Jaccard -> containment -> cosine. Within each tier the resolver collects every qualifying (write, candidate) edge, sorts descending by score with a stable tiebreak on (write_index, candidate_feature_id), and claims edges in that order. The strongest pair always wins. Sweep behavior, candidate_filter, deactivate_filter, audit-log contract, and the conservative absorb path are all unchanged. The change is algorithm-local; existing callers (synthesize.py full scan, pr_narrow_synthesis.py narrow path) need no changes and benefit equally. Validated locally by restoring the prod ATOAPayment snapshot and replaying the PR-3220 webhook through the backend's Redis-stream worker on the new matcher. High-confidence false-positives (cosine >= 0.85 twin) dropped from prod's 10+ to 4 on the local replay -- a meaningful incremental fix. The residual 4 are a cross-tier ordering pattern (a sibling write claims via signature/Jaccard while the better cosine pair is starved); a fuller fix would need global cross-tier optimization (Hungarian-style) and is out of scope for this change. Tests: - Updated test_feature_reconciler_containment.py's 4 unit tests to use the new _resolve_pairings entry via a _pair_one helper. Integration test untouched. - New test_feature_reconciler_global_pairing.py pins the regression with 6 cases: strongest-Jaccard-pair-wins regardless of input order; cosine score-descending claims correctly; signature tier still pre-empts lower tiers; sibling write does not block; tier order preserved when both Jaccard and cosine qualify for the same pair; unmatched candidate left for sweep. All 1818 backend tests pass; ruff and mypy --strict clean across 492 source files. Signed-off-by: Arun Rajkumar <mickyarunr@gmail.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_resolve_pairingsinbackend/app/services/feature_reconciler.py; the old_match_strategyis removed._pair_onehelper. The integration test is untouched.test_feature_reconciler_global_pairing.pywith 6 regression cases pinning the new behavior.Why
On the 2026-06-01 merge to main of a large payments-domain repo, 28 features were soft-deleted in one scan. Of those, at least 10 had a still-active twin with embedding cosine ≥ 0.85 — several with literally identical titles:
Root cause: the per-write greedy let an earlier write claim a candidate via a borderline match (often Jaccard ≈ 0.7-0.8), starving the later write that would have been a clean cosine pair. The displaced candidate then had no match and was swept inactive at the end of reconcile. Write iteration order — i.e., LLM output order — silently determined which features survived.
Stage 1 (PR #185) removed BUD-closure as one entry point into this failure. The full-scan path (called from PR-merge above the narrow cap, and from any explicit rescan) still hit this matcher defect on every run. This PR fixes the algorithm.
What changed
_resolve_pairingsruns four passes — signature first as a 1:1 lookup, then Jaccard / containment / cosine each as a tier-local edge sort. For each non-signature tier:(write, candidate)edge whose score clears the tier's threshold.(score, -write_index, candidate_feature_id_str)— strongest first, lower write index wins on equal score (matches old greedy iteration-order semantics), candidate UUID as final deterministic tiebreak so behavior never depends on the candidate-list iteration order.reconcile_features_for_repoconsumes the pre-computedpairingslist and the rest of the CRUD path (_insert_new,_update_existing,_absorb_into_existing, sweep) is byte-identical.The PR is algorithm-local: no caller signatures change,
candidate_filter/deactivate_filter/feature_match_logsemantics are preserved.Test plan
ruff check backend/— zero issues.ruff format --checkon changed files — clean.mypy backend/app/— 492 source files, no issues.pytest backend/tests/— 1818 passed.pr-review-toolkit:code-reviewersubagent — addressed both findings (tiebreak determinism: addedstr(cand.feature_id)as third sort key; coverage gaps: added two extra regression tests for cross-write contention and tier ordering).pr_merge_workerso the in-process MCP token authenticates correctly. With the fix applied: 29 features deactivated; only 4 had a high-cosine twin still active — vs the prod baseline of 10+ under the old matcher. Material improvement (≈ 60% reduction in high-confidence false-positives in a single PR's replay).feature_match_logandfeatures.is_active=falsetransitions on the next several PR-merge scans; expect the visible "Deactivated" rate on feature cards to drop materially.Known follow-up (not blocking this PR)
The 4 residual false-positives in the local replay all have the same shape: a different new write claims the candidate via signature or Jaccard (tier 1 or 2), so the cosine-equivalent new write that would have been a better semantic pair has nothing left to claim. The within-tier global resolution doesn't address cross-tier ordering. A fuller fix would be Hungarian-style global optimization across all tiers (best edge overall wins, regardless of tier); that's a larger algorithm change and belongs in a separate PR if the residual rate stays elevated.