diff --git a/README.md b/README.md index 66f6450..ee1b0dc 100644 --- a/README.md +++ b/README.md @@ -16,8 +16,8 @@ GitHub PR reviews → download → extract → synthesize → dedupe → place ``` 1. **Download** — fetch PR data (reviews, comments, diffs) via `gh` CLI -2. **Extract** — use Claude to identify actionable changes and generalizable rules from each comment (bot comments are filtered out automatically) -3. **Synthesize** — embed generalizations, cluster by similarity, extract validated rules. Each rule is scored based on the LLM's confidence multiplied by a factor for how many unique PRs the rule's evidence spans (1 PR = 0.6×, 2 = 0.85×, 3 = 0.95×, 4+ = 1.0×), so rules that came up across more reviews score higher. +2. **Extract** — use Claude to identify actionable changes and generalizable rules from each comment. Bot comments are filtered out automatically, *except* those a human has endorsed with a 👍 reaction — a thumbs-up reclaims a bot's autoreview comment as human-vetted signal (see [Reactions](#reactions)) +3. **Synthesize** — embed generalizations, cluster by similarity, extract validated rules. Each rule is scored based on the LLM's confidence multiplied by a factor for how many unique PRs the rule's evidence spans (1 PR = 0.6×, 2 = 0.85×, 3 = 0.95×, 4+ = 1.0×), so rules that came up across more reviews score higher. A [reaction](#reactions) factor then adjusts the score: rules whose source comments were endorsed (👍) are boosted (1.25×), vetoed (👎) are penalized (0.4×). 4. **Dedupe** — three-pass deduplication (embedding clusters + category review + post-consolidation review) 5. **Place** — determine where each rule belongs (root, directory, file, cross-file), filtering by `min_score` floor (default 0.5) 6. **Group** — filter by `min_score` threshold (default 0.5) and organize by topic for progressive disclosure @@ -123,6 +123,7 @@ uv run braindump --repo owner/repo run [--from STAGE] [--skip STAGE ...] [--sinc - `--authors`: Author filter for extract (default: `all`) - `--min-score`: Override rule score threshold (default: 0.5) - `--max-rules N`: Cap the number of rules in group stage (top-scored) +- `--reaction-authors`: Comma-separated logins whose 👍/👎 reactions count (default: anyone). See [Reactions](#reactions) - `--fresh`: Wipe all stage outputs and start from scratch ### `download` — Fetch PR data @@ -140,6 +141,7 @@ uv run braindump --repo owner/repo extract [--authors USER] [--limit N] [--rando - `--prs`: Filter to specific PR numbers (comma-separated) - `--limit N`: Limit number of comments to process - `--random`: Randomly sample comments (with `--seed` for reproducibility) +- `--reaction-authors`: Comma-separated logins whose 👍/👎 reactions count (default: anyone). See [Reactions](#reactions) ### `synthesize` — Cluster and validate rules @@ -182,6 +184,25 @@ uv run braindump --repo owner/repo run --from group --max-rules 80 uv run braindump --repo owner/repo generate [--dry-run] [--concurrency 10] ``` +### `apply` — Merge generated files into a repo + +`generate` writes to the staging directory (`data///7-generate/`). `apply` merges those files into an actual repo **without clobbering hand-written content**: + +```bash +uv run braindump --repo owner/repo apply --into /path/to/repo [--dry-run] +``` + +Every generated file wraps braindump's output between `` and `` markers. When merging: + +- If the target file already has these markers → only the fenced block is replaced; anything you wrote above or below it is preserved. +- If the target exists without markers → the block is appended after your content. +- If the target doesn't exist → it's created. + +This applies uniformly to `AGENTS.md` and topic docs under `agent_docs/`, so it's safe to keep your own additions in any of them. Files braindump doesn't generate are never touched. Use `--dry-run` to preview which files would be created/merged/left unchanged. + +> [!NOTE] +> Edits made *inside* the fenced block are overwritten on the next `apply`. Keep your hand-written content outside the markers. + ### `status` — Pipeline status ```bash @@ -206,7 +227,7 @@ All per-repo data lives under `data///`: data/ pydantic/ pydantic-ai/ - 1-download/ # Raw GitHub API data (PRs, diffs, review comments) + 1-download/ # Raw GitHub API data (PRs, diffs, review comments, reactions) 2-extract/ # extractions.jsonl, checkpoint.json 3-synthesize/ # rules.jsonl, embeddings, clusters 4-dedupe/ # rules.jsonl (deduped), embeddings, merge_log @@ -233,6 +254,25 @@ Overrides cluster with similar extracted rules and replace them during the `dedu uv run braindump --repo owner/repo run --from dedupe --fresh ``` +## Reactions + +A 👍/👎 reaction on a review comment is an explicit human verdict on that comment — the highest-precision signal in the pipeline. `download` fetches reaction attribution (who reacted) for any comment that has reactions, and it feeds two things: + +- **Reclaiming bot comments.** Bot autoreview comments (e.g. `github-actions[bot]`, `coderabbitai[bot]`) are normally filtered out. But if you 👍 one, it's reclaimed and its rules flow into the pipeline as human-vetted signal. This lets you curate a bot reviewer's output by simply reacting to the good comments. +- **Scoring.** The net thumbs signal (👍 minus 👎) across a rule's source comments adjusts its score: endorsed rules are boosted (1.25×), vetoed rules penalized (0.4×). An endorsed rule from a single PR can clear the score floor it would otherwise miss; a vetoed rule can drop below it and be excluded. + +By default any user's reactions count. Use `--reaction-authors` to restrict this to specific logins (e.g. maintainers): + +```bash +uv run braindump --repo owner/repo run --reaction-authors alice,bob +``` + +Reactions are captured at `download`, but the allowlist is applied at `extract`, so changing `--reaction-authors` only requires re-running from extract — no re-download: + +```bash +uv run braindump --repo owner/repo run --from extract --reaction-authors alice +``` + ## Model configuration By default, braindump uses the [Pydantic AI Gateway](https://ai.pydantic.dev/gateway/) with `anthropic:claude-sonnet-4-5` for LLM tasks and `openai:text-embedding-3-small` for embeddings. Both models are configurable via flags or environment variables: diff --git a/src/braindump/cli.py b/src/braindump/cli.py index b7cb411..9c3e0d3 100644 --- a/src/braindump/cli.py +++ b/src/braindump/cli.py @@ -51,6 +51,7 @@ def main( # Register all commands +from braindump.commands.apply import apply as _apply # noqa: E402 from braindump.commands.dedupe import dedupe as _dedupe # noqa: E402 from braindump.commands.download import download as _download # noqa: E402 from braindump.commands.extract import extract as _extract # noqa: E402 @@ -69,6 +70,7 @@ def main( app.command(name="place")(_place) app.command(name="group")(_group) app.command(name="generate")(_generate) +app.command(name="apply")(_apply) app.command(name="run")(_run) app.command(name="lookup")(_lookup) app.command(name="status")(_status) diff --git a/src/braindump/commands/apply.py b/src/braindump/commands/apply.py new file mode 100644 index 0000000..fd9ddb2 --- /dev/null +++ b/src/braindump/commands/apply.py @@ -0,0 +1,58 @@ +"""Apply generated files into a target repo via non-destructive block merge.""" + +from __future__ import annotations + +from pathlib import Path + +import typer + +from braindump.commands.generate import merge_braindump_block +from braindump.config import RepoConfig +from braindump.progress import console + + +def _run(config: RepoConfig, into: Path, dry_run: bool = False) -> None: + output_dir = config.output_dir + generated = sorted(output_dir.rglob("*.md")) if output_dir.exists() else [] + if not generated: + console.print( + f"[red]No generated .md files under {output_dir}. Run `generate` first.[/red]" + ) + raise typer.Exit(1) + + into = into.resolve() + created = merged = unchanged = 0 + for src in generated: + rel = src.relative_to(output_dir) + target = into / rel + content = src.read_text() + + if target.exists(): + existing = target.read_text() + new_content = merge_braindump_block(existing, content) + if new_content == existing: + verb, unchanged = "unchanged", unchanged + 1 + else: + verb, merged = "merge", merged + 1 + else: + new_content, verb, created = content, "create", created + 1 + + console.print(f"[dim]{verb:9}[/dim] {target}") + if not dry_run and verb != "unchanged": + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(new_content) + + prefix = "[yellow]Dry run — nothing written.[/yellow] " if dry_run else "" + console.print( + f"\n{prefix}{created} created, {merged} merged, {unchanged} unchanged (into {into})" + ) + + +def apply( + ctx: typer.Context, + into: Path = typer.Option(..., "--into", help="Target repo root to merge generated files into"), + dry_run: bool = typer.Option(False, "--dry-run", help="Show what would change without writing"), +) -> None: + """Merge generated AGENTS.md files into a repo, preserving hand-written content.""" + config: RepoConfig = ctx.obj["config"] + _run(config, into=into, dry_run=dry_run) diff --git a/src/braindump/commands/dedupe.py b/src/braindump/commands/dedupe.py index 2e3b61a..7c95b73 100644 --- a/src/braindump/commands/dedupe.py +++ b/src/braindump/commands/dedupe.py @@ -198,6 +198,7 @@ def load_overrides(overrides_path: Path) -> list[dict]: "source_comments": [], "source_prs": [], "unique_pr_count": 5, + "reaction_net": 0, "cluster_coherence": 1.0, "common_pattern": "", "_is_override": True, @@ -303,6 +304,7 @@ def _build_output_rules( combined_comments = list({c for r in source_rules for c in r["source_comments"]}) combined_prs = sorted({p for r in source_rules for p in r["source_prs"]}) unique_pr_count = len(combined_prs) + combined_reaction_net = sum(r.get("reaction_net", 0) for r in source_rules) # Pick the lowest unused source rule ID to avoid duplicates # when the LLM references overlapping sources across output rules available_ids = sorted(r["rule_id"] for r in source_rules) @@ -327,7 +329,9 @@ def _build_output_rules( "text": consolidated.text, "reason": consolidated.reason, "confidence": consolidated.confidence, - "rule_score": calculate_rule_score(consolidated.confidence, unique_pr_count), + "rule_score": calculate_rule_score( + consolidated.confidence, unique_pr_count, combined_reaction_net + ), "category": consolidated.category, "scope": consolidated.scope, "example_bad": example_bad, @@ -336,6 +340,7 @@ def _build_output_rules( "source_comments": combined_comments, "source_prs": combined_prs, "unique_pr_count": unique_pr_count, + "reaction_net": combined_reaction_net, "cluster_coherence": max(r.get("cluster_coherence", 0) for r in source_rules), "common_pattern": best_source.get("common_pattern", ""), "_merged_from": sorted(r["rule_id"] for r in source_rules), @@ -356,6 +361,7 @@ async def consolidate_cluster(cluster_rules: list[dict]) -> tuple[list[dict], De combined_comments = list({c for r in extracted for c in r.get("source_comments", [])}) combined_prs = sorted({p for r in extracted for p in r.get("source_prs", [])}) unique_pr_count = max(len(combined_prs), override["unique_pr_count"]) + combined_reaction_net = sum(r.get("reaction_net", 0) for r in extracted) example_bad = override.get("example_bad") example_good = override.get("example_good") for r in extracted: @@ -382,7 +388,9 @@ async def consolidate_cluster(cluster_rules: list[dict]) -> tuple[list[dict], De "text": override["text"], "reason": override["reason"], "confidence": override["confidence"], - "rule_score": calculate_rule_score(override["confidence"], unique_pr_count), + "rule_score": calculate_rule_score( + override["confidence"], unique_pr_count, combined_reaction_net + ), "category": category, "scope": scope, "example_bad": example_bad, @@ -391,6 +399,7 @@ async def consolidate_cluster(cluster_rules: list[dict]) -> tuple[list[dict], De "source_comments": combined_comments, "source_prs": combined_prs, "unique_pr_count": unique_pr_count, + "reaction_net": combined_reaction_net, "cluster_coherence": max( (r.get("cluster_coherence", 0) for r in extracted), default=1.0 ), diff --git a/src/braindump/commands/download.py b/src/braindump/commands/download.py index fae6527..3ad552f 100644 --- a/src/braindump/commands/download.py +++ b/src/braindump/commands/download.py @@ -30,6 +30,42 @@ async def _gh_async(*args: str) -> tuple[int, str, str]: return proc.returncode or 0, stdout.decode(), stderr.decode() +async def _fetch_reactions(repo: str, output_dir, pr_num: int, comments_json: str) -> None: + """Fetch reactor attribution for a PR's reacted review comments → sidecar. + + Writes ``review_comment_reactions/{pr_num}.json`` mapping comment id (str) + to a list of ``{"content", "user"}`` entries. Comments with no reactions are + skipped, so the file is empty ``{}`` for most PRs. + """ + try: + comments = json.loads(comments_json) + except json.JSONDecodeError: + comments = [] + + reactions_by_comment: dict[str, list[dict]] = {} + for c in comments: + if c.get("reactions", {}).get("total_count", 0) <= 0: + continue + cid = c["id"] + with logfire.span("fetch reactions: comment {cid}", cid=cid): + rc, stdout, _ = await _gh_async( + "api", f"repos/{repo}/pulls/comments/{cid}/reactions", "--paginate" + ) + if rc != 0: + continue + try: + entries = json.loads(stdout) + except json.JSONDecodeError: + entries = [] + reactions_by_comment[str(cid)] = [ + {"content": e.get("content"), "user": e.get("user", {}).get("login")} for e in entries + ] + + (output_dir / "review_comment_reactions" / f"{pr_num}.json").write_text( + json.dumps(reactions_by_comment) + ) + + @logfire.instrument("download", extract_args={"config": ["repo"]}) def _run( config: RepoConfig, @@ -54,7 +90,14 @@ def _run( shutil.rmtree(output_dir) # Create directories - for subdir in ["prs", "reviews", "review_comments", "diffs", "issue_comments"]: + for subdir in [ + "prs", + "reviews", + "review_comments", + "review_comment_reactions", + "diffs", + "issue_comments", + ]: (output_dir / subdir).mkdir(parents=True, exist_ok=True) # Build search filter @@ -105,12 +148,13 @@ async def _async_download( semaphore = asyncio.Semaphore(concurrency) def _pr_already_downloaded(pr_num: int) -> bool: - """Check if all 5 expected files exist for a PR.""" + """Check if all expected files exist for a PR.""" return all( [ (output_dir / "prs" / f"{pr_num}.json").exists(), (output_dir / "reviews" / f"{pr_num}.json").exists(), (output_dir / "review_comments" / f"{pr_num}.json").exists(), + (output_dir / "review_comment_reactions" / f"{pr_num}.json").exists(), (output_dir / "diffs" / f"{pr_num}.diff").exists(), (output_dir / "issue_comments" / f"{pr_num}.json").exists(), ] @@ -159,6 +203,12 @@ async def download_pr(pr_num: int) -> None: stderr=stderr[:200], ) + # Reaction attribution: for comments that carry reactions, fetch + # who reacted (a 👍/👎 is an explicit human verdict on the comment). + # Only reacted comments are queried, so this is cheap. + if rc == 0: + await _fetch_reactions(repo, output_dir, pr_num, stdout) + # Diff with logfire.span("fetch diff: #{pr_num}", pr_num=pr_num): rc, stdout, stderr = await _gh_async( diff --git a/src/braindump/commands/extract.py b/src/braindump/commands/extract.py index 027834b..f05897f 100644 --- a/src/braindump/commands/extract.py +++ b/src/braindump/commands/extract.py @@ -16,6 +16,7 @@ from braindump.config import RepoConfig, compute_multi_file_hash from braindump.progress import StageStats, console, format_cost, get_result_cost, stage_progress +from braindump.reactions import parse_reaction_authors, reaction_signal # ============================================================================ # Models @@ -64,6 +65,7 @@ class ExtractionOutput(BaseModel): comment_id: int pr_number: int author: str + reaction_signal: int = 0 is_actionable: bool rejection_reason: str | None = None rejection_explanation: str | None = None @@ -196,26 +198,41 @@ def get_agent() -> Agent[None, CommentExtraction]: # ============================================================================ -def load_comments(review_dir: Path) -> list[dict]: +def load_comments(review_dir: Path, reactions_dir: Path | None = None) -> list[dict]: all_comments = [] for filepath in sorted(review_dir.glob("*.json")): with open(filepath) as f: comments = json.load(f) - pr_number = int(filepath.stem) - for comment in comments: - comment["_pr_number"] = pr_number - all_comments.append(comment) + pr_number = int(filepath.stem) + reactions = {} + if reactions_dir is not None: + reactions_file = reactions_dir / filepath.name + if reactions_file.exists(): + reactions = json.loads(reactions_file.read_text()) + for comment in comments: + comment["_pr_number"] = pr_number + comment["_reactions"] = reactions.get(str(comment["id"]), []) + all_comments.append(comment) return all_comments +def is_bot_comment(comment: dict) -> bool: + user = comment.get("user", {}) + return user.get("type") == "Bot" or user.get("login", "").endswith("[bot]") + + +def annotate_reaction_signal(comments: list[dict], allowlist: set[str] | None) -> None: + """Attach `_reaction_signal` (allowlist-filtered net thumbs) to each comment.""" + for c in comments: + c["_reaction_signal"] = reaction_signal(c.get("_reactions", []), allowlist) + + def filter_bots(comments: list[dict]) -> list[dict]: - """Exclude comments from bot accounts.""" - return [ - c - for c in comments - if c.get("user", {}).get("type") != "Bot" - and not c.get("user", {}).get("login", "").endswith("[bot]") - ] + """Exclude bot comments, except those a human has endorsed with a 👍. + + Requires `annotate_reaction_signal` to have run first. + """ + return [c for c in comments if not is_bot_comment(c) or c.get("_reaction_signal", 0) > 0] def filter_by_author(comments: list[dict], author: str | None) -> list[dict]: @@ -406,6 +423,7 @@ async def process_one( comment_id=comment_id, pr_number=pr_number, author=author, + reaction_signal=comment.get("_reaction_signal", 0), is_actionable=result.is_actionable, rejection_reason=result.rejection.reason if result.rejection else None, rejection_explanation=result.rejection.explanation @@ -482,6 +500,7 @@ def _run( seed: int = 42, prs: str | None = None, concurrency: int = 10, + reaction_authors: str | None = None, is_pipeline: bool = False, fresh: bool = False, ) -> StageStats | None: @@ -489,6 +508,7 @@ def _run( import shutil review_dir = config.review_comments_dir + reactions_dir = config.review_comment_reactions_dir output_dir = config.stage_dir("2-extract") # Input hash: detect if download data changed @@ -512,13 +532,21 @@ def _run( if not is_pipeline: console.print("Loading comments...") - all_comments = load_comments(review_dir) + all_comments = load_comments(review_dir, reactions_dir) if not is_pipeline: console.print(f"Loaded {len(all_comments)} total comments") + allowlist = parse_reaction_authors(reaction_authors) + annotate_reaction_signal(all_comments, allowlist) + before_bots = len(all_comments) all_comments = filter_bots(all_comments) + reclaimed = sum(1 for c in all_comments if is_bot_comment(c)) if not is_pipeline: - console.print(f"After excluding bots: {len(all_comments)} comments") + note = f" (reclaimed {reclaimed} endorsed bot comment(s))" if reclaimed else "" + console.print( + f"After excluding bots: {len(all_comments)} comments" + f" [dropped {before_bots - len(all_comments)}]{note}" + ) if prs: pr_numbers = set(int(p.strip()) for p in prs.split(",")) @@ -580,6 +608,11 @@ def extract( seed: int = typer.Option(42, help="Random seed for reproducible sampling"), prs: str | None = typer.Option(None, help="Comma-separated list of PR numbers"), concurrency: int = typer.Option(10, help="Number of parallel LLM requests"), + reaction_authors: str | None = typer.Option( + None, + "--reaction-authors", + help="Comma-separated logins whose 👍/👎 reactions count (default: anyone)", + ), fresh: bool = typer.Option(False, "--fresh", help="Wipe output and start from scratch"), ) -> None: """Extract potential rules from review comments.""" @@ -597,5 +630,6 @@ def extract( seed=seed, prs=prs, concurrency=concurrency, + reaction_authors=reaction_authors, fresh=fresh, ) diff --git a/src/braindump/commands/generate.py b/src/braindump/commands/generate.py index 7bc0a44..8eed212 100644 --- a/src/braindump/commands/generate.py +++ b/src/braindump/commands/generate.py @@ -4,6 +4,7 @@ import asyncio import json +import re import shutil from collections import defaultdict from decimal import Decimal @@ -19,6 +20,36 @@ from braindump.config import RepoConfig, compute_file_hash from braindump.progress import StageStats, console, format_cost, get_result_cost, stage_progress +# ============================================================================ +# Managed-block markers +# ============================================================================ + +# Every generated file wraps its content between these markers so `apply` can +# splice braindump's output into a file that also holds hand-written content, +# replacing only the managed region. +BLOCK_OPEN = "" +BLOCK_CLOSE = "" +_BLOCK_OPEN_RE = re.compile(r"", re.IGNORECASE) +_BLOCK_CLOSE_RE = re.compile(r"", re.IGNORECASE) + + +def merge_braindump_block(existing: str, generated: str) -> str: + """Splice a braindump-generated file into existing file content. + + `generated` is a full file fenced by the open/close markers. Content + outside the markers in `existing` is preserved — only the fenced region is + replaced. If `existing` has no markers, the block is appended after it. + """ + open_m = _BLOCK_OPEN_RE.search(existing) + close_m = _BLOCK_CLOSE_RE.search(existing) + if open_m and close_m and close_m.end() > open_m.start(): + return existing[: open_m.start()] + generated + existing[close_m.end() :] + if not existing.strip(): + return generated + prefix = existing if existing.endswith("\n") else existing + "\n" + return f"{prefix}\n{generated}" + + # ============================================================================ # Models # ============================================================================ @@ -302,7 +333,7 @@ def generate_agents_md( subdirectories: list[str] | None = None, ) -> str: lines = [] - lines.append("") + lines.append(BLOCK_OPEN) lines.append("") if location == "root": lines.append("# Coding Guidelines") @@ -345,7 +376,7 @@ def generate_agents_md( lines.append(format_rule_line(rid, rephrased[rid])) lines.append("") - lines.append("") + lines.append(BLOCK_CLOSE) return "\n".join(lines) @@ -355,6 +386,8 @@ def generate_topic_doc( placements: dict[int, dict], ) -> str: lines = [] + lines.append(BLOCK_OPEN) + lines.append("") lines.append(f"# {topic['topic_name']}") lines.append("") lines.append(f"> {topic['topic_description']}") @@ -367,6 +400,7 @@ def generate_topic_doc( if rid in rephrased: lines.append(format_rule_line(rid, rephrased[rid])) lines.append("") + lines.append(BLOCK_CLOSE) return "\n".join(lines) diff --git a/src/braindump/commands/run.py b/src/braindump/commands/run.py index 6821884..5031ac0 100644 --- a/src/braindump/commands/run.py +++ b/src/braindump/commands/run.py @@ -49,6 +49,11 @@ def run( ), skip: list[str] | None = typer.Option(None, help="Skip specific stages (repeatable)"), authors: str = typer.Option("all", help="Author filter for extract stage"), + reaction_authors: str | None = typer.Option( + None, + "--reaction-authors", + help="Comma-separated logins whose 👍/👎 reactions count (default: anyone)", + ), since: str | None = typer.Option(None, help="Date filter for download stage (YYYY-MM-DD)"), min_score: float | None = typer.Option(None, help="Minimum rule score (default: 0.5)"), max_rules: int | None = typer.Option( @@ -91,7 +96,13 @@ def run( elif stage == "extract": from braindump.commands.extract import _run as extract_run - stats = extract_run(config, authors=authors, is_pipeline=True, fresh=fresh) + stats = extract_run( + config, + authors=authors, + reaction_authors=reaction_authors, + is_pipeline=True, + fresh=fresh, + ) elif stage == "synthesize": from braindump.commands.synthesize import _run as synthesize_run diff --git a/src/braindump/commands/synthesize.py b/src/braindump/commands/synthesize.py index 6700baf..1ae5354 100644 --- a/src/braindump/commands/synthesize.py +++ b/src/braindump/commands/synthesize.py @@ -78,6 +78,7 @@ class RuleOutput(BaseModel): source_comments: list[int] source_prs: list[int] unique_pr_count: int + reaction_net: int = 0 cluster_coherence: float common_pattern: str @@ -86,10 +87,25 @@ class RuleOutput(BaseModel): PR_FACTORS: dict[int, float] = {1: 0.6, 2: 0.85, 3: 0.95} PR_FACTOR_DEFAULT: float = 1.0 # 4+ PRs +# Multiplier for the net human reaction verdict on a rule's source comments. +# Endorsed lifts a rule (can rescue a low-PR bot rule above min_score); vetoed +# sinks it (can push it below the placement floor). +REACTION_FACTOR_ENDORSED = 1.25 +REACTION_FACTOR_NEUTRAL = 1.0 +REACTION_FACTOR_VETOED = 0.4 -def calculate_rule_score(confidence: float, unique_prs: int) -> float: + +def reaction_factor(reaction_net: int) -> float: + if reaction_net > 0: + return REACTION_FACTOR_ENDORSED + if reaction_net < 0: + return REACTION_FACTOR_VETOED + return REACTION_FACTOR_NEUTRAL + + +def calculate_rule_score(confidence: float, unique_prs: int, reaction_net: int = 0) -> float: pr_factor = PR_FACTORS.get(unique_prs, PR_FACTOR_DEFAULT) - return min(1.0, confidence * pr_factor) + return min(1.0, confidence * pr_factor * reaction_factor(reaction_net)) # ============================================================================ @@ -208,6 +224,7 @@ def __init__( pr_number: int, source_change_index: int, source_context: dict, + reaction_signal: int = 0, ): self.generalization = generalization self.motivation = motivation @@ -216,6 +233,7 @@ def __init__( self.pr_number = pr_number self.source_change_index = source_change_index self.source_context = source_context + self.reaction_signal = reaction_signal self.embedding: list[float] | None = None @@ -237,7 +255,9 @@ def load_extractions(input_path: Path) -> list[dict]: for line in f: if line.strip(): ext = json.loads(line) - if is_bot_author(ext.get("author", "")): + # Drop bot-authored extractions, except those a human endorsed + # (reclaimed in the extract stage, carried via reaction_signal). + if is_bot_author(ext.get("author", "")) and ext.get("reaction_signal", 0) <= 0: continue extractions.append(ext) return extractions @@ -251,6 +271,7 @@ def extract_generalizations(extractions: list[dict]) -> list[GeneralizationItem] comment_id = ext["comment_id"] pr_number = ext["pr_number"] source_context = ext.get("source", {}) + signal = ext.get("reaction_signal", 0) for rule in ext.get("potential_rules", []): gen = rule.get("generalization", "") if not gen: @@ -263,6 +284,7 @@ def extract_generalizations(extractions: list[dict]) -> list[GeneralizationItem] pr_number=pr_number, source_change_index=rule.get("source_change_index", 0), source_context=source_context, + reaction_signal=signal, ) items.append(item) return items @@ -692,8 +714,11 @@ async def analyze_one(i: int) -> None: progress.advance(task_id) return - # Build comment_id -> pr_number mapping from cluster items + # Build comment_id -> pr_number / reaction mappings from cluster items comment_to_pr: dict[int, int] = {item.comment_id: item.pr_number for item in cluster} + comment_to_signal: dict[int, int] = { + item.comment_id: item.reaction_signal for item in cluster + } cluster_comment_ids_set = set(comment_ids) async with write_lock: @@ -738,7 +763,12 @@ async def analyze_one(i: int) -> None: rule_pr_numbers = pr_numbers rule_unique_pr_count = unique_pr_count - score = calculate_rule_score(rule.confidence, rule_unique_pr_count) + rule_reaction_net = sum( + comment_to_signal.get(cid, 0) for cid in rule_comment_ids + ) + score = calculate_rule_score( + rule.confidence, rule_unique_pr_count, rule_reaction_net + ) rule_output = RuleOutput( rule_id=rule_id, cluster_id=i, @@ -754,6 +784,7 @@ async def analyze_one(i: int) -> None: source_comments=rule_comment_ids, source_prs=rule_pr_numbers, unique_pr_count=rule_unique_pr_count, + reaction_net=rule_reaction_net, cluster_coherence=analysis.cluster_coherence, common_pattern=analysis.common_pattern, ) diff --git a/src/braindump/config.py b/src/braindump/config.py index 500f63b..16d17ba 100644 --- a/src/braindump/config.py +++ b/src/braindump/config.py @@ -124,6 +124,10 @@ def pr_data_dir(self) -> Path: def review_comments_dir(self) -> Path: return self.pr_data_dir / "review_comments" + @property + def review_comment_reactions_dir(self) -> Path: + return self.pr_data_dir / "review_comment_reactions" + @property def extractions_path(self) -> Path: return self.stage_dir("2-extract") / "extractions.jsonl" diff --git a/src/braindump/reactions.py b/src/braindump/reactions.py new file mode 100644 index 0000000..0a1029d --- /dev/null +++ b/src/braindump/reactions.py @@ -0,0 +1,42 @@ +"""Reaction-derived validity signal for review comments. + +A 👍/👎 on a review comment is an explicit human verdict on that comment. It is +the highest-precision provenance signal available: it can reclaim a bot-authored +autoreview comment (endorsed) or veto a comment regardless of who wrote it. +Only thumbs reactions carry a verdict — other reactions (heart, rocket, …) are +ignored. +""" + +from __future__ import annotations + +_POSITIVE = "+1" +_NEGATIVE = "-1" + + +def parse_reaction_authors(value: str | None) -> set[str] | None: + """Parse a comma-separated allowlist of reactor logins. + + None/empty → count reactions from anyone. Otherwise only these logins count. + """ + if not value: + return None + logins = {a.strip() for a in value.split(",") if a.strip()} + return logins or None + + +def reaction_signal(reactions: list[dict], allowlist: set[str] | None) -> int: + """Net thumbs signal for a comment: (count of 👍) − (count of 👎). + + `reactions` is a list of ``{"content", "user"}`` entries. When `allowlist` + is given, only reactions from those logins count; otherwise all count. + """ + net = 0 + for r in reactions: + if allowlist is not None and r.get("user") not in allowlist: + continue + content = r.get("content") + if content == _POSITIVE: + net += 1 + elif content == _NEGATIVE: + net -= 1 + return net