feat(ci_visibility): coverage itr backfill

[gnufede]

Summary

Implements ITR coverage backfilling so that test.code_coverage.lines_pct is accurate even when
ITR skips tests. Without this, skipped tests' covered lines are missing from the tally, producing
artificially low coverage percentages.

What it does:

• Parses meta.coverage (base64 bitmaps) and _missing_line_code_coverage from the
  /api/v2/ci/tests/skippable API response
• Accumulates per-test coverage bitmaps into a session-level aggregate during test execution
• At session end, OR-merges session coverage with the skipped-tests coverage from meta.coverage
  and recomputes lines_pct from the merged result
• Sets test.code_coverage.backfilled = "true" on session, module, and suite events whenever
  ITR skipping is enabled
• Tests with _missing_line_code_coverage are excluded from skippable_items (they run fresh
  to collect live coverage), matching Java tracer behaviour

xdist support: Workers propagate their _session_coverage bitmaps via workeroutput to the
main process, which merges all workers' coverage before computing the final percentage.

Key design decisions

• _missing_line_code_coverage is per-test, not per-session. Flagged tests are not skipped
  (excluded from skippable_items), so their coverage is always collected. This matches Java and
  avoids losing coverage for those tests entirely.
• Backfill tag set whenever ITR skipping is enabled, regardless of whether meta.coverage is
  empty or tests were actually skipped. This tells the backend to trust lines_pct.
• Coverage percentage denominator uses the highest set bit position per file (not full
  byte-aligned bitmap length) to avoid counting intra-byte padding as phantom lines.
• CoverageLines re-exported from ddtrace.testing.internal.tracer_api.coverage — the
  module's docstring already declares it as the coverage API boundary for ddtrace.testing.

Files changed

File	Change
api_client.py	Parse meta.coverage, exclude _missing_line_code_coverage items from skippable
cached_file_provider.py	Update TestOptDataProvider protocol + CachedFileDataProvider signatures
session_manager.py	Store itr_covered_files from API response
test_data.py	Add CODE_COVERAGE_BACKFILLED tag constant
plugin.py	Accumulate session coverage, OR-merge + recompute percentage, set tag on session/module/suite, xdist propagation
tracer_api/coverage.py	Re-export CoverageLines

Test plan

• Unit tests for API client parsing: coverage bitmap decode, path normalization,
  _missing_line_code_coverage exclusion, malformed base64 handling
• Integration tests for backfill tag on session/module/suite events (5 scenarios: ITR
  enabled with coverage, missing flag, empty coverage, ITR disabled, no tests skipped)
• xdist subprocess tests: tag set with 2 workers, missing flag, empty coverage
• Existing test suites updated for new return types

🤖 Generated with Claude Code

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show existing cycles (showing 5 of 4974 shortest)

ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kafka -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams

To see all cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json

[cit-pr-commenter-54b7da]

Codeowners resolved as

ddtrace/testing/internal/api_client.py                                  @DataDog/ci-app-libraries
ddtrace/testing/internal/cached_file_provider.py                        @DataDog/ci-app-libraries
ddtrace/testing/internal/pytest/plugin.py                               @DataDog/ci-app-libraries
ddtrace/testing/internal/session_manager.py                             @DataDog/ci-app-libraries
tests/testing/internal/pytest/test_pytest_itr.py                        @DataDog/ci-app-libraries
tests/testing/internal/pytest/test_pytest_xdist.py                      @DataDog/ci-app-libraries
tests/testing/internal/test_api_client.py                               @DataDog/ci-app-libraries
tests/testing/internal/test_cached_file_provider.py                     @DataDog/ci-app-libraries
tests/testing/internal/test_session_manager.py                          @DataDog/ci-app-libraries
tests/testing/mocks.py                                                  @DataDog/ci-app-libraries

[datadog-datadog-prod-us1]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 26 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741357-d2b8243-manylinux2014_aarch64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64]     

DataDog/apm-reliability/dd-trace-py | build linux: [arm64, cp315-cp315, v113741589-d2b8243-musllinux_1_2_aarch64]     

View all 26 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 2 jobs - 2 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e16737a | Docs | Datadog PR Page | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-07-10 11:27:52

Comparing candidate commit e16737a in PR branch gnufede/coverage-itr-fill with baseline commit af36221 in branch gnufede/coverage-upload-conflict.

Found 0 performance improvements and 7 performance regressions! Performance is the same for 613 metrics, 10 unstable metrics.

scenario:iastaspects-add_aspect

• 🟥 execution_time [+7.900µs; +10.269µs] or [+7.372%; +9.583%]

scenario:iastaspects-casefold_noaspect

• 🟥 execution_time [+28.391µs; +38.895µs] or [+8.884%; +12.171%]

scenario:iastaspects-strip_aspect

• 🟥 execution_time [+74.992µs; +80.611µs] or [+23.745%; +25.524%]

scenario:iastaspects-upper_noaspect

• 🟥 execution_time [+19.708µs; +29.276µs] or [+7.964%; +11.830%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+81.526µs; +92.476µs] or [+18.917%; +21.458%]

scenario:iastaspectssplit-rsplit_aspect

• 🟥 execution_time [+11.912µs; +17.849µs] or [+7.755%; +11.621%]

scenario:span-start

• 🟥 execution_time [+1.320ms; +1.462ms] or [+8.576%; +9.496%]

[gnufede]

E2E Test Report: SUCCESS ✅

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)

Test Environment

• Method: Local testing with real test playgrounds + mockdog backend
• Revision tested: e16737a
• Playgrounds: flask (Python) + jolokia (Java, dd-trace-java v1.64.0 for comparison)

Results

Python — flask playground

Check	Status
Tests skipped by ITR (4 in test_appctx.py)	✅
Test with _missing_line_code_coverage=true runs fresh (not skipped)	✅
test.code_coverage.backfilled=true on session	✅
test.code_coverage.backfilled=true on module	✅
test.code_coverage.backfilled=true on suite	✅
citestcov payloads: 497 (501 total − 4 skipped)	✅
skippable.tests_returned: 5 (all entries including _missing_line_code_coverage one)	✅

Java — jolokia playground (dd-trace-java v1.64.0, for comparison)

Check	Status
Tests skipped by ITR (1 of 2 in PropertyUtilTest)	✅
Test with _missing_line_code_coverage=true runs fresh	✅
test.code_coverage.backfilled=true on modules (jolokia-json, jolokia-core)	✅
test.code_coverage.backfilled NOT set on session (session-level Jacoco aggregation fails across 32 modules)	documented
test.code_coverage.backfilled NOT set on suites (Java doesn't propagate to suite events)	documented

Issues Found During Testing

1. Python: 0 tests were being skipped — root cause: module field was missing from the scenario's skippable test entries. Python's ModuleRef lookup compares against the module name from the test's nodeid path (e.g., "tests" from tests/test_appctx.py::...). The mockdog skippable response was using an empty fallback (".") because configurations.test.bundle was absent. Fixed by adding module: "tests" to each entry in the scenario YAML.

2. Python: skippable.tests_returned assertion — the server returns all 5 entries (including the _missing_line_code_coverage one). The library internally filters to 4 skip candidates but the count reflects what the server returned. Fixed assertion to {exact: 5}.

Behavioral Comparison: Python vs Java

Behavior	dd-trace-py	dd-trace-java v1.64.0
backfilled on session	✅	❌ (Jacoco aggregation exception across many modules)
backfilled on modules	✅	✅
backfilled on suites	✅	❌ (not propagated to suite events)
_missing_line_code_coverage respected	✅ always	✅ in v1.64.0 (unconditionally)
Skippable requests	1 per session	1 per Maven module
meta.coverage path prefix	adds /	strips /

Test Methodology

1. Created flask-itr-coverage-backfill scenario with meta.coverage bitmaps and _missing_line_code_coverage flag
2. Ran dd-trace-py against flask playground using --dep dd-trace-py=feat/coverage-itr-backfill with mockdog backend
3. Debugged and fixed module lookup mismatch by reading ddtrace/testing/internal/api_client.py (EMPTY_NAME fallback) and ddtrace/testing/internal/pytest/utils.py (nodeid-to-module extraction)
4. Ran Java comparison tests against jolokia with dd-trace-java v1.64.0, investigating Jacoco injection requirements
5. Documented 6 behavioral differences between Python and Java implementations

---

This E2E test was performed by Shepherd — autonomous QA agent for Datadog Test Optimization