Skip to content

Commit 08d7af2

Browse files
rnetserclaude
andauthored
fix: reduce false positives in pytest_marker_analyzer smoke detection (#4344)
## Summary - Five fixes to reduce false positive smoke test triggers in `scripts/tests_analyzer/pytest_marker_analyzer.py` - Tested on 150 recent PRs: false positives reduced from 57 to 34 (40% reduction) - Many PRs that previously flagged all 83 smoke tests now correctly flag only the 1-2 actually affected ## Changes 1. **Conftest import-only changes** — `_get_modified_function_names` returns `None` (diff failure) vs empty `set()` (no functions modified), preventing the all-fixtures fallback when only imports change in conftest files 2. **Module-level tracking** — `_extract_modified_symbols` continues past unattributed lines (imports, comments between functions) instead of returning `None`, preserving symbol info for narrowing 3. **API fetch fallback** — failed `_fetch_file_at_ref` falls through to local file read instead of returning `None` 4. **Checkout mode API cache** — `github_pr_info` is now preserved in checkout mode so the GitHub API diffs cache is available as fallback when local git diff fails 5. **Conftest no-fixture-match** — when a conftest imports a modified symbol but no fixture used by the test calls it, resolve as safe instead of conservatively flagging ## Test plan - [x] All 74 unit tests pass - [x] Ran script on 150 recent PRs (merged + open, excluding draft/WIP) - [x] Verified previously false-positive PRs (#4308, #4314, #4310, #4312, #4242, #4105, #4092, etc.) now correctly return False - [x] Verified legitimate True PRs (#4277, #4159, #4177, etc.) still correctly return True Co-authored-by: Claude <noreply@anthropic.com> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * More precise test-impact detection: harmless module edits (imports/comments/blank/docstrings) are treated safe while executable module changes are still flagged. * Renamed or pure-deletion cases now yield explicit empty results instead of ambiguous fallbacks. * Improved fallback semantics: checkout mode uses local analysis when remote fetches fail; remote mode remains conservative. * Tracks unattributed changed lines to reduce false positives. * **Tests** * Expanded coverage for diff retrieval, change-classification semantics, conftest interactions, renamed-file handling, and mode-specific fallback behaviors. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4fa833a commit 08d7af2

File tree

2 files changed

+1102
-42
lines changed

2 files changed

+1102
-42
lines changed

scripts/tests_analyzer/pytest_marker_analyzer.py

Lines changed: 114 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -709,6 +709,15 @@ class SymbolClassification:
709709
modified_members: dict[str, set[str]] = field(default_factory=dict)
710710
"""class_name -> set of modified member names (after transitive expansion).
711711
Absent class = no member-level info, fall back to class-level."""
712+
has_unattributed_changes: bool = False
713+
"""True when some changed lines could not be mapped to any named symbol.
714+
715+
This covers import reorderings, comments between functions, and other
716+
module-level code that is not a function, class, or variable assignment.
717+
When set, callers that already have symbol-level import data can still
718+
use the ``modified_symbols`` set for narrowing instead of falling back
719+
to file-level dependency tracking.
720+
"""
712721

713722

714723
@dataclass
@@ -1623,6 +1632,7 @@ def _extract_modified_symbols(
16231632
pr_diffs_cache: dict[str, str] | None = None,
16241633
file_status: str | None = None,
16251634
pr_head_ref: str | None = None,
1635+
is_checkout: bool = False,
16261636
) -> SymbolClassification | None:
16271637
"""Determine which top-level symbols were modified or added in a file.
16281638
@@ -1644,13 +1654,23 @@ def _extract_modified_symbols(
16441654
pr_head_ref: Optional PR HEAD commit SHA. When provided in remote
16451655
mode, the symbol map is built from the PR's version of the
16461656
file so that line numbers align with the diff.
1657+
is_checkout: Whether the analyzer is running in checkout mode
1658+
(the local working tree is the PR HEAD). When ``True``,
1659+
falling back to the local file after a fetch failure is safe.
1660+
When ``False`` (remote analysis), the local file may be on a
1661+
different branch and must not be used as fallback.
16471662
16481663
Returns:
16491664
``SymbolClassification`` with modified and new symbol sets, or
16501665
``None`` when symbol-level analysis is not possible (diff failure,
1651-
module-level changes outside any symbol, or parse errors).
1652-
A ``None`` return signals the caller to fall back to file-level
1653-
dependency tracking.
1666+
pure deletion, or parse errors). A ``None`` return signals the
1667+
caller to fall back to file-level dependency tracking.
1668+
1669+
When some changed lines fall outside any named symbol (e.g. import
1670+
reorderings or comments between functions), the returned
1671+
``SymbolClassification`` will have ``has_unattributed_changes=True``
1672+
but still contain any symbols that *were* identified from other
1673+
changed lines.
16541674
"""
16551675
diff_content = _get_diff_content(
16561676
file_path=file_path,
@@ -1659,14 +1679,20 @@ def _extract_modified_symbols(
16591679
github_pr_info=github_pr_info,
16601680
pr_diffs_cache=pr_diffs_cache,
16611681
)
1682+
16621683
if diff_content is None:
1684+
if file_status == "renamed":
1685+
# Renamed with no diff — content unchanged, no symbols modified
1686+
return SymbolClassification(modified_symbols=set(), new_symbols=set())
16631687
return None
16641688

16651689
has_deletions = _diff_has_deletions(diff_content=diff_content)
16661690

16671691
changed_lines = _parse_diff_for_changed_lines(diff_content=diff_content)
16681692
if not changed_lines:
16691693
if has_deletions:
1694+
if file_status == "renamed":
1695+
return SymbolClassification(modified_symbols=set(), new_symbols=set())
16701696
return None # Pure deletion — cannot safely narrow impact
16711697
return SymbolClassification(modified_symbols=set(), new_symbols=set())
16721698

@@ -1683,12 +1709,21 @@ def _extract_modified_symbols(
16831709
token=github_pr_info.get("token"),
16841710
)
16851711
if source is None:
1686-
logger.info(
1687-
msg="Failed to fetch PR file version, falling back to file-level analysis",
1688-
extra={"file": str(file_path)},
1689-
)
1690-
return None
1712+
if is_checkout:
1713+
logger.info(
1714+
msg="Failed to fetch PR file version, falling back to local file (checkout mode)",
1715+
extra={"file": str(file_path)},
1716+
)
1717+
else:
1718+
# Remote mode: local file may not match PR HEAD.
1719+
# Fall back conservatively to file-level analysis.
1720+
logger.info(
1721+
msg="Failed to fetch PR file version, falling back to file-level analysis",
1722+
extra={"file": str(file_path)},
1723+
)
1724+
return None
16911725
if source is None:
1726+
# pr_head_ref was None — pure local mode, local file is authoritative
16921727
source = file_path.read_text(encoding="utf-8")
16931728
symbol_map = _build_line_to_symbol_map(source=source)
16941729
except (SyntaxError, UnicodeDecodeError, OSError) as exc: # fmt: skip
@@ -1698,6 +1733,8 @@ def _extract_modified_symbols(
16981733
)
16991734
return None
17001735

1736+
has_unattributed = False
1737+
source_lines = source.splitlines()
17011738
modified_symbols: set[str] = set()
17021739
for line_number in changed_lines:
17031740
found = False
@@ -1707,9 +1744,23 @@ def _extract_modified_symbols(
17071744
found = True
17081745
break
17091746
if not found:
1710-
# Changed line is outside any top-level symbol (module-level code).
1711-
# Conservative fallback: cannot safely narrow impact.
1712-
return None
1747+
# Changed line is outside any top-level symbol. Check whether
1748+
# the line is "safe" (import, comment, blank, or string literal)
1749+
# or potentially impactful executable code.
1750+
if line_number <= len(source_lines):
1751+
line_content = source_lines[line_number - 1].strip()
1752+
if (
1753+
not line_content
1754+
or line_content.startswith("#")
1755+
or line_content.startswith(("import ", "from "))
1756+
or line_content.startswith(('"""', "'''", '"', "'"))
1757+
):
1758+
has_unattributed = True
1759+
else:
1760+
# Executable module-level code — conservative fallback
1761+
return None
1762+
else:
1763+
return None
17131764

17141765
# --- Member-level analysis for modified classes ---
17151766
modified_members: dict[str, set[str]] = {}
@@ -1760,6 +1811,7 @@ def _extract_modified_symbols(
17601811
modified_symbols=set(),
17611812
new_symbols=modified_symbols,
17621813
modified_members=modified_members,
1814+
has_unattributed_changes=has_unattributed,
17631815
)
17641816

17651817
# Identify candidate new symbols: symbols whose ENTIRE line range
@@ -1778,6 +1830,7 @@ def _extract_modified_symbols(
17781830
modified_symbols=modified_symbols,
17791831
new_symbols=set(),
17801832
modified_members=modified_members,
1833+
has_unattributed_changes=has_unattributed,
17811834
)
17821835

17831836
# Optimization: if the diff has no deletions, existing code was not
@@ -1789,6 +1842,7 @@ def _extract_modified_symbols(
17891842
modified_symbols=truly_modified,
17901843
new_symbols=truly_new,
17911844
modified_members=modified_members,
1845+
has_unattributed_changes=has_unattributed,
17921846
)
17931847

17941848
# Deletions exist — need to check old file to distinguish rewrites
@@ -1805,6 +1859,7 @@ def _extract_modified_symbols(
18051859
modified_symbols=modified_symbols,
18061860
new_symbols=set(),
18071861
modified_members=modified_members,
1862+
has_unattributed_changes=has_unattributed,
18081863
)
18091864

18101865
old_symbols, old_class_members = old_result
@@ -1832,6 +1887,7 @@ def _extract_modified_symbols(
18321887
modified_symbols=truly_modified,
18331888
new_symbols=truly_new,
18341889
modified_members=modified_members,
1890+
has_unattributed_changes=has_unattributed,
18351891
)
18361892

18371893

@@ -2034,15 +2090,13 @@ def _check_conftest_pathway(
20342090
break
20352091

20362092
if not fixture_match:
2037-
# Overlap exists but no fixture calls them — could be module-level usage
2038-
# Conservative: flag test
2039-
symbols_str = ", ".join(sorted(overlapping))
2040-
matching_deps.append(
2041-
f"{changed_file.relative_to(repo_root)} (via {conftest_path.relative_to(repo_root)}, symbols: {symbols_str})"
2042-
)
2093+
# Safe for this specific conftest path, but other conftest.py
2094+
# files higher in the hierarchy may still create a dependency.
2095+
conftest_resolved = True
2096+
continue
20432097

20442098
conftest_resolved = True
2045-
break
2099+
return True, matching_deps
20462100

20472101
if not conftest_resolved:
20482102
# No conftest pathway found — file-level fallback (conservative)
@@ -2065,6 +2119,7 @@ def _check_test_impact(
20652119
conftest_opaque_deps: dict[Path, set[Path]] | None = None,
20662120
pr_diffs_cache: dict[str, str] | None = None,
20672121
pr_file_statuses: dict[str, str] | None = None,
2122+
is_checkout: bool = False,
20682123
) -> dict[str, Any] | None:
20692124
"""Check if a single test is affected by changed files (for parallel execution).
20702125
@@ -2179,6 +2234,7 @@ def _check_test_impact(
21792234
github_pr_info=github_pr_info,
21802235
pr_diffs_cache=pr_diffs_cache,
21812236
file_status=file_status_conftest,
2237+
is_checkout=is_checkout,
21822238
)
21832239

21842240
# Get all transitively affected fixtures
@@ -2334,6 +2390,7 @@ def _extract_modified_items_from_conftest(
23342390
github_pr_info: dict[str, Any] | None,
23352391
pr_diffs_cache: dict[str, str] | None = None,
23362392
file_status: str | None = None,
2393+
is_checkout: bool = False,
23372394
) -> tuple[set[str], set[str]]:
23382395
"""Extract modified fixtures and functions from conftest.py.
23392396
@@ -2355,8 +2412,8 @@ def _extract_modified_items_from_conftest(
23552412
Tuple of (modified_fixtures, modified_functions) containing only
23562413
symbols that existed in the base version and were modified.
23572414
"""
2358-
if file_status == "added":
2359-
return set(), set() # New conftest cannot break existing tests
2415+
if file_status in ("added", "renamed"):
2416+
return set(), set() # New/renamed conftest cannot break existing tests
23602417

23612418
modified_fixtures: set[str] = set()
23622419
modified_functions: set[str] = set()
@@ -2384,16 +2441,17 @@ def _extract_modified_items_from_conftest(
23842441
repo_root=repo_root,
23852442
github_pr_info=github_pr_info,
23862443
pr_diffs_cache=pr_diffs_cache,
2444+
is_checkout=is_checkout,
23872445
)
23882446

2389-
# Preserve the raw set before additive filtering for fallback logic:
2390-
# if diff parsing found functions but additive filtering emptied the
2391-
# set, the conftest only contains new additions and should NOT trigger
2392-
# the conservative all-fixtures fallback.
2393-
raw_modified_function_names = set(modified_function_names)
2447+
# None means diff retrieval failed — fall back conservatively
2448+
if modified_function_names is None:
2449+
if changed_file.exists():
2450+
return all_fixtures, set()
2451+
return set(), set()
23942452

23952453
# Filter out purely new functions/fixtures (additive-change detection)
2396-
if modified_function_names and file_status != "added":
2454+
if modified_function_names and file_status not in ("added", "renamed"):
23972455
old_result = _get_old_file_symbols(
23982456
file_path=changed_file,
23992457
base_branch=base_branch,
@@ -2411,11 +2469,6 @@ def _extract_modified_items_from_conftest(
24112469
else:
24122470
modified_functions.add(func_name)
24132471

2414-
# Fallback: only trigger when diff parsing itself failed (raw set empty),
2415-
# NOT when additive filtering legitimately emptied the set.
2416-
if not raw_modified_function_names and changed_file.exists():
2417-
return all_fixtures, set()
2418-
24192472
except (SyntaxError, UnicodeDecodeError, OSError, subprocess.SubprocessError) as exc: # fmt: skip
24202473
logger.info(
24212474
msg="Error extracting modified items from conftest", extra={"file": str(changed_file), "error": str(exc)}
@@ -2441,10 +2494,16 @@ def _get_modified_function_names(
24412494
repo_root: Path,
24422495
github_pr_info: dict[str, Any] | None,
24432496
pr_diffs_cache: dict[str, str] | None = None,
2444-
) -> set[str]:
2445-
"""Get names of functions modified in a file based on diff analysis."""
2446-
modified: set[str] = set()
2497+
is_checkout: bool = False,
2498+
) -> set[str] | None:
2499+
"""Get names of functions modified in a file based on diff analysis.
24472500
2501+
Returns:
2502+
Set of modified function names, or None if diff retrieval failed.
2503+
An empty set means the diff was successfully obtained but no
2504+
functions were in the changed lines (e.g., only module-level
2505+
import changes).
2506+
"""
24482507
# Try pre-fetched cache first
24492508
if pr_diffs_cache is not None:
24502509
try:
@@ -2468,8 +2527,10 @@ def _get_modified_function_names(
24682527

24692528
diff_content = get_pr_file_diff(repo=repo, pr_number=pr_number, file_path=str(relative_path), token=token)
24702529
if diff_content:
2471-
modified = _parse_diff_for_functions(diff_content=diff_content)
2472-
return modified
2530+
return _parse_diff_for_functions(diff_content=diff_content)
2531+
if not is_checkout:
2532+
return None # Remote mode: no local fallback
2533+
# Checkout mode: fall through to local git diff
24732534

24742535
# Use local git
24752536
try:
@@ -2481,11 +2542,11 @@ def _get_modified_function_names(
24812542
timeout=10,
24822543
)
24832544
if result.returncode == 0:
2484-
modified = _parse_diff_for_functions(diff_content=result.stdout)
2545+
return _parse_diff_for_functions(diff_content=result.stdout)
2546+
return None # git diff failed
24852547
except (subprocess.SubprocessError, OSError) as e: # fmt: skip
24862548
logger.info(msg="Error getting modified function names", extra={"file": str(file_path), "error": str(e)})
2487-
2488-
return modified
2549+
return None # Diff retrieval failed
24892550

24902551

24912552
def _parse_diff_for_functions(diff_content: str) -> set[str]:
@@ -2721,12 +2782,14 @@ def __init__(
27212782
repo_root: Path | None = None,
27222783
base_branch: str = "main",
27232784
github_pr_info: dict[str, Any] | None = None,
2785+
is_checkout: bool = False,
27242786
) -> None:
27252787
self.marker_expression = marker_expression
27262788
self.marker_names = extract_marker_names(marker_expression=marker_expression)
27272789
self.repo_root = repo_root or Path.cwd()
27282790
self.base_branch = base_branch
27292791
self.github_pr_info = github_pr_info # Contains repo, pr_number, token for GitHub API calls
2792+
self.is_checkout = is_checkout
27302793
self.marked_tests: dict[str, MarkedTest] = {}
27312794
self.conftest_files: list[Path] = []
27322795
self.fixtures: dict[str, Fixture] = {} # name -> Fixture
@@ -3309,6 +3372,7 @@ def analyze_impact(self, changed_files: list[Path]) -> AnalysisResult:
33093372
pr_diffs_cache=pr_diffs_cache,
33103373
file_status=file_status,
33113374
pr_head_ref=pr_head_ref,
3375+
is_checkout=self.is_checkout,
33123376
)
33133377

33143378
# Check each marked test for dependency matches in parallel using ThreadPoolExecutor
@@ -3330,6 +3394,7 @@ def analyze_impact(self, changed_files: list[Path]) -> AnalysisResult:
33303394
conftest_opaque_deps=self.conftest_opaque_deps,
33313395
pr_diffs_cache=pr_diffs_cache,
33323396
pr_file_statuses=pr_file_statuses,
3397+
is_checkout=self.is_checkout,
33333398
): node_id
33343399
for node_id, marked_test in self.marked_tests.items()
33353400
}
@@ -3597,8 +3662,14 @@ def run_github_mode(args: argparse.Namespace) -> tuple[AnalysisResult | None, in
35973662
)
35983663
# Continue anyway - the branch might already be available
35993664

3600-
# When we checkout, we can use local git diff, so no need to pass github_pr_info
3601-
github_pr_info = None
3665+
# Pass github_pr_info even in checkout mode so the GitHub API
3666+
# diffs cache is available when local git diff fails (e.g.,
3667+
# base branch fetch failure or shallow clone limitations).
3668+
github_pr_info = {
3669+
"repo": args.repo,
3670+
"pr_number": args.pr,
3671+
"token": token,
3672+
}
36023673
else:
36033674
# Remote mode: use current directory and GitHub API for diffs
36043675
repo_root = Path.cwd()
@@ -3615,7 +3686,8 @@ def run_github_mode(args: argparse.Namespace) -> tuple[AnalysisResult | None, in
36153686
repo_root=repo_root,
36163687
base_branch=base_branch,
36173688
github_pr_info=github_pr_info,
3618-
) # Constructor already uses keyword arguments
3689+
is_checkout=args.checkout,
3690+
)
36193691
analyzer.discover_marked_tests()
36203692

36213693
if not analyzer.marked_tests:

0 commit comments

Comments
 (0)