fix(ci_visibility): do not use is_user_code [backport 3.14] #14809
Conversation
![dd-octo-sts[bot]](https://avatars.githubusercontent.com/in/1157446?s=80&v=4){kind=small}
|
|
Currently, in Test Impact Analysis code coverage, we use `is_user_code()` to decide whether to instrument a module. This function was conceived for use in non-CI contexts where user code may be installed in site-packages like external dependencies. This is not necessary in Test Optimization, where we only want to cover code inside the current repo (TIA only works with filenames that are relative to the repo root, and we already set the repo root as the include path of `ModuleCodeCollector`). Removing this check also resolves a bug where test packages are incorrectly detected as third-party if a third-party package contains files under `test/*` or `tests/*` that get installed at site-packages level (`sumy` and `sendgrid` do this, for instance). Unit tests + manual testing in dogweb. Potentially covering more code than expected. Should not be an issue because we already exclude site-packages and similar [here](https://github.com/DataDog/dd-trace-py/blob/vitor-de-araujo%2FSDTEST-2747%2Fcoverage-third-party/ddtrace/internal/coverage/code.py#L53). None. (cherry picked from commit 7cd4d9c)
vitor-de-araujo
force-pushed
the
backport-14794-to-3.14
branch
from
79586db to
760ac12
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 268 ± 3 ms. The average import time from base is: 270 ± 3 ms. The import time difference between this PR and base is: -2.4 ± 0.1 ms. Import time breakdownThe following import paths have shrunk:
|
Performance SLOsComparing candidate backport-14794-to-3.14 (760ac12) with baseline 3.14 (8592132) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 0.332µs (SLO: <10.000µs 📉 -96.7%) vs baseline: -1.1% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.9% ✅ add_inplace_aspectTime: ✅ 0.334µs (SLO: <10.000µs 📉 -96.7%) vs baseline: +0.8% Memory: ✅ 37.591MB (SLO: <39.000MB -3.6%) vs baseline: +5.1% ✅ add_inplace_noaspectTime: ✅ 0.317µs (SLO: <10.000µs 📉 -96.8%) vs baseline: +0.4% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.0% ✅ add_noaspectTime: ✅ 0.277µs (SLO: <10.000µs 📉 -97.2%) vs baseline: +1.7% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.7% ✅ bytearray_aspectTime: ✅ 1.373µs (SLO: <10.000µs 📉 -86.3%) vs baseline: +4.1% Memory: ✅ 37.454MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ bytearray_extend_aspectTime: ✅ 1.419µs (SLO: <10.000µs 📉 -85.8%) vs baseline: +0.9% Memory: ✅ 37.415MB (SLO: <39.000MB -4.1%) vs baseline: +4.5% ✅ bytearray_extend_noaspectTime: ✅ 0.613µs (SLO: <10.000µs 📉 -93.9%) vs baseline: +0.9% Memory: ✅ 37.454MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ bytearray_noaspectTime: ✅ 0.484µs (SLO: <10.000µs 📉 -95.2%) vs baseline: +0.8% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.1% ✅ bytes_aspectTime: ✅ 1.493µs (SLO: <10.000µs 📉 -85.1%) vs baseline: 📈 +13.9% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.7% ✅ bytes_noaspectTime: ✅ 0.493µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.7% Memory: ✅ 37.434MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ bytesio_aspectTime: ✅ 1.320µs (SLO: <10.000µs 📉 -86.8%) vs baseline: -0.6% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +4.9% ✅ bytesio_noaspectTime: ✅ 0.493µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.4% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +4.8% ✅ capitalize_aspectTime: ✅ 0.736µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +0.5% Memory: ✅ 37.572MB (SLO: <39.000MB -3.7%) vs baseline: +5.2% ✅ capitalize_noaspectTime: ✅ 0.435µs (SLO: <10.000µs 📉 -95.7%) vs baseline: +0.4% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.1% ✅ casefold_aspectTime: ✅ 0.732µs (SLO: <10.000µs 📉 -92.7%) vs baseline: -0.8% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.8% ✅ casefold_noaspectTime: ✅ 0.370µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +0.2% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.8% ✅ decode_aspectTime: ✅ 0.722µs (SLO: <10.000µs 📉 -92.8%) vs baseline: -0.8% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +4.9% ✅ decode_noaspectTime: ✅ 0.413µs (SLO: <10.000µs 📉 -95.9%) vs baseline: -1.1% Memory: ✅ 37.591MB (SLO: <39.000MB -3.6%) vs baseline: +5.1% ✅ encode_aspectTime: ✅ 0.706µs (SLO: <10.000µs 📉 -92.9%) vs baseline: -0.7% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.0% ✅ encode_noaspectTime: ✅ 0.402µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -0.3% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +5.0% ✅ format_aspectTime: ✅ 3.307µs (SLO: <10.000µs 📉 -66.9%) vs baseline: +0.4% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.7% ✅ format_map_aspectTime: ✅ 3.174µs (SLO: <10.000µs 📉 -68.3%) vs baseline: +0.7% Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.1% ✅ format_map_noaspectTime: ✅ 0.785µs (SLO: <10.000µs 📉 -92.2%) vs baseline: +1.3% Memory: ✅ 37.454MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ format_noaspectTime: ✅ 0.593µs (SLO: <10.000µs 📉 -94.1%) vs baseline: -0.3% Memory: ✅ 37.532MB (SLO: <39.000MB -3.8%) vs baseline: +5.0% ✅ index_aspectTime: ✅ 0.345µs (SLO: <10.000µs 📉 -96.6%) vs baseline: +0.1% Memory: ✅ 37.454MB (SLO: <39.000MB -4.0%) vs baseline: +4.6% ✅ index_noaspectTime: ✅ 0.274µs (SLO: <10.000µs 📉 -97.3%) vs baseline: -1.9% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.9% ✅ join_aspectTime: ✅ 1.236µs (SLO: <10.000µs 📉 -87.6%) vs baseline: +1.0% Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.2% ✅ join_noaspectTime: ✅ 0.488µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.2% Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.1% ✅ ljust_aspectTime: ✅ 10.932µs (SLO: <20.000µs 📉 -45.3%) vs baseline: +5.0% Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +5.1% ✅ ljust_noaspectTime: ✅ 0.405µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +1.4% Memory: ✅ 37.572MB (SLO: <39.000MB -3.7%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 2.244µs (SLO: <10.000µs 📉 -77.6%) vs baseline: +0.7% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.9% ✅ lower_noaspectTime: ✅ 0.364µs (SLO: <10.000µs 📉 -96.4%) vs baseline: -1.7% Memory: ✅ 37.532MB (SLO: <39.000MB -3.8%) vs baseline: +4.9% ✅ lstrip_aspectTime: ✅ 11.170µs (SLO: <20.000µs 📉 -44.2%) vs baseline: +6.9% Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 0.381µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.7% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +4.8% ✅ modulo_aspectTime: ✅ 0.932µs (SLO: <10.000µs 📉 -90.7%) vs baseline: +0.4% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +5.0% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 1.435µs (SLO: <10.000µs 📉 -85.7%) vs baseline: +0.7% Memory: ✅ 37.434MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ modulo_aspect_for_bytesTime: ✅ 0.910µs (SLO: <10.000µs 📉 -90.9%) vs baseline: -0.6% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +4.9% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 1.132µs (SLO: <10.000µs 📉 -88.7%) vs baseline: +0.9% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.0% ✅ modulo_noaspectTime: ✅ 0.624µs (SLO: <10.000µs 📉 -93.8%) vs baseline: -0.7% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +5.0% ✅ replace_aspectTime: ✅ 5.195µs (SLO: <10.000µs 📉 -48.1%) vs baseline: 📈 +15.3% Memory: ✅ 37.532MB (SLO: <39.000MB -3.8%) vs baseline: +4.6% ✅ replace_noaspectTime: ✅ 0.460µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.1% Memory: ✅ 37.572MB (SLO: <39.000MB -3.7%) vs baseline: +5.0% ✅ repr_aspectTime: ✅ 0.906µs (SLO: <10.000µs 📉 -90.9%) vs baseline: -0.4% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +5.0% ✅ repr_noaspectTime: ✅ 0.415µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +0.4% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +4.8% ✅ rstrip_aspectTime: ✅ 10.332µs (SLO: <20.000µs 📉 -48.3%) vs baseline: ~same Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +4.9% ✅ rstrip_noaspectTime: ✅ 0.379µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.5% Memory: ✅ 37.454MB (SLO: <39.000MB -4.0%) vs baseline: +4.8% ✅ slice_aspectTime: ✅ 0.487µs (SLO: <10.000µs 📉 -95.1%) vs baseline: +0.7% Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.0% ✅ slice_noaspectTime: ✅ 0.448µs (SLO: <10.000µs 📉 -95.5%) vs baseline: ~same Memory: ✅ 37.375MB (SLO: <39.000MB -4.2%) vs baseline: +4.6% ✅ stringio_aspectTime: ✅ 1.537µs (SLO: <10.000µs 📉 -84.6%) vs baseline: -0.8% Memory: ✅ 37.415MB (SLO: <39.000MB -4.1%) vs baseline: +4.7% ✅ stringio_noaspectTime: ✅ 0.721µs (SLO: <10.000µs 📉 -92.8%) vs baseline: +1.4% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.9% ✅ strip_aspectTime: ✅ 10.470µs (SLO: <20.000µs 📉 -47.7%) vs baseline: +0.3% Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.8% ✅ strip_noaspectTime: ✅ 0.383µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.2% Memory: ✅ 37.473MB (SLO: <39.000MB -3.9%) vs baseline: +4.8% ✅ swapcase_aspectTime: ✅ 2.587µs (SLO: <10.000µs 📉 -74.1%) vs baseline: +6.6% Memory: ✅ 37.532MB (SLO: <39.000MB -3.8%) vs baseline: +5.0% ✅ swapcase_noaspectTime: ✅ 0.534µs (SLO: <10.000µs 📉 -94.7%) vs baseline: -1.2% Memory: ✅ 37.434MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ title_aspectTime: ✅ 2.574µs (SLO: <10.000µs 📉 -74.3%) vs baseline: 📈 +10.1% Memory: ✅ 37.591MB (SLO: <39.000MB -3.6%) vs baseline: +5.1% ✅ title_noaspectTime: ✅ 0.497µs (SLO: <10.000µs 📉 -95.0%) vs baseline: ~same Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.2% ✅ translate_aspectTime: ✅ 3.529µs (SLO: <10.000µs 📉 -64.7%) vs baseline: +9.5% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +4.8% ✅ translate_noaspectTime: ✅ 1.033µs (SLO: <10.000µs 📉 -89.7%) vs baseline: -0.6% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +5.0% ✅ upper_aspectTime: ✅ 2.609µs (SLO: <10.000µs 📉 -73.9%) vs baseline: 📈 +16.1% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.1% ✅ upper_noaspectTime: ✅ 0.370µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +0.4% Memory: ✅ 37.415MB (SLO: <39.000MB -4.1%) vs baseline: +4.6% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 4.143µs (SLO: <10.000µs 📉 -58.6%) vs baseline: -1.0% Memory: ✅ 37.395MB (SLO: <39.000MB -4.1%) vs baseline: +4.6% ✅ ospathbasename_noaspectTime: ✅ 1.089µs (SLO: <10.000µs 📉 -89.1%) vs baseline: ~same Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.1% ✅ ospathjoin_aspectTime: ✅ 6.972µs (SLO: <10.000µs 📉 -30.3%) vs baseline: 📈 +16.8% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.0% ✅ ospathjoin_noaspectTime: ✅ 2.291µs (SLO: <10.000µs 📉 -77.1%) vs baseline: +0.2% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +4.7% ✅ ospathnormcase_aspectTime: ✅ 3.390µs (SLO: <10.000µs 📉 -66.1%) vs baseline: -0.5% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +5.0% ✅ ospathnormcase_noaspectTime: ✅ 0.573µs (SLO: <10.000µs 📉 -94.3%) vs baseline: +1.3% Memory: ✅ 37.454MB (SLO: <39.000MB -4.0%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.851µs (SLO: <10.000µs 📉 -51.5%) vs baseline: +2.2% Memory: ✅ 37.552MB (SLO: <39.000MB -3.7%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 1.589µs (SLO: <10.000µs 📉 -84.1%) vs baseline: +0.1% Memory: ✅ 37.434MB (SLO: <39.000MB -4.0%) vs baseline: +4.7% ✅ ospathsplitdrive_aspectTime: ✅ 3.623µs (SLO: <10.000µs 📉 -63.8%) vs baseline: +0.2% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +4.9% ✅ ospathsplitdrive_noaspectTime: ✅ 0.702µs (SLO: <10.000µs 📉 -93.0%) vs baseline: -0.4% Memory: ✅ 37.415MB (SLO: <39.000MB -4.1%) vs baseline: +4.7% ✅ ospathsplitext_aspectTime: ✅ 4.454µs (SLO: <10.000µs 📉 -55.5%) vs baseline: -0.3% Memory: ✅ 37.493MB (SLO: <39.000MB -3.9%) vs baseline: +5.1% ✅ ospathsplitext_noaspectTime: ✅ 1.396µs (SLO: <10.000µs 📉 -86.0%) vs baseline: +0.4% Memory: ✅ 37.513MB (SLO: <39.000MB -3.8%) vs baseline: +5.1% 📈 telemetryaddmetric - 30/30✅ 1-count-metric-1-timesTime: ✅ 3.593µs (SLO: <20.000µs 📉 -82.0%) vs baseline: 📈 +11.3% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.7% ✅ 1-count-metrics-100-timesTime: ✅ 219.902µs (SLO: <250.000µs 📉 -12.0%) vs baseline: -0.8% Memory: ✅ 31.457MB (SLO: <34.000MB -7.5%) vs baseline: +4.9% ✅ 1-distribution-metric-1-timesTime: ✅ 3.004µs (SLO: <20.000µs 📉 -85.0%) vs baseline: -0.1% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 198.283µs (SLO: <220.000µs -9.9%) vs baseline: +0.2% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.156µs (SLO: <20.000µs 📉 -89.2%) vs baseline: +1.6% Memory: ✅ 31.497MB (SLO: <34.000MB -7.4%) vs baseline: +4.8% ✅ 1-gauge-metrics-100-timesTime: ✅ 125.216µs (SLO: <150.000µs 📉 -16.5%) vs baseline: +1.1% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.9% ✅ 1-rate-metric-1-timesTime: ✅ 3.606µs (SLO: <20.000µs 📉 -82.0%) vs baseline: 📈 +11.9% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.8% ✅ 1-rate-metrics-100-timesTime: ✅ 219.535µs (SLO: <250.000µs 📉 -12.2%) vs baseline: ~same Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.8% ✅ 100-count-metrics-100-timesTime: ✅ 22.666ms (SLO: <23.500ms -3.6%) vs baseline: +3.2% Memory: ✅ 31.497MB (SLO: <34.000MB -7.4%) vs baseline: +4.9% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.080ms (SLO: <2.250ms -7.6%) vs baseline: +2.0% Memory: ✅ 31.713MB (SLO: <34.000MB -6.7%) vs baseline: +4.4% ✅ 100-gauge-metrics-100-timesTime: ✅ 1.285ms (SLO: <1.550ms 📉 -17.1%) vs baseline: -0.5% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.9% ✅ 100-rate-metrics-100-timesTime: ✅ 2.344ms (SLO: <2.550ms -8.1%) vs baseline: +3.6% Memory: ✅ 31.477MB (SLO: <34.000MB -7.4%) vs baseline: +4.9% ✅ flush-1-metricTime: ✅ 4.199µs (SLO: <20.000µs 📉 -79.0%) vs baseline: -0.5% Memory: ✅ 31.890MB (SLO: <34.000MB -6.2%) vs baseline: +5.0% ✅ flush-100-metricsTime: ✅ 185.155µs (SLO: <250.000µs 📉 -25.9%) vs baseline: +0.3% Memory: ✅ 31.890MB (SLO: <34.000MB -6.2%) vs baseline: +4.9% ✅ flush-1000-metricsTime: ✅ 2.232ms (SLO: <2.500ms 📉 -10.7%) vs baseline: +0.4% Memory: ✅ 32.637MB (SLO: <34.500MB -5.4%) vs baseline: +4.7% 🟡 Near SLO Breach (6 suites)🟡 djangosimple - 26/26✅ appsecTime: ✅ 20.562ms (SLO: <22.300ms -7.8%) vs baseline: +0.5% Memory: ✅ 64.350MB (SLO: <66.000MB -2.5%) vs baseline: +4.8% ✅ exception-replay-enabledTime: ✅ 1.364ms (SLO: <1.450ms -5.9%) vs baseline: -0.3% Memory: ✅ 63.304MB (SLO: <66.000MB -4.1%) vs baseline: +4.9% ✅ iastTime: ✅ 20.591ms (SLO: <22.250ms -7.5%) vs baseline: +0.3% Memory: ✅ 64.389MB (SLO: <66.000MB -2.4%) vs baseline: +5.0% ✅ profilerTime: ✅ 15.150ms (SLO: <16.550ms -8.5%) vs baseline: -0.4% Memory: ✅ 53.064MB (SLO: <53.500MB 🟡 -0.8%) vs baseline: +5.1% ✅ span-code-originTime: ✅ 26.113ms (SLO: <28.200ms -7.4%) vs baseline: -0.1% Memory: ✅ 66.590MB (SLO: <68.500MB -2.8%) vs baseline: +4.7% ✅ tracerTime: ✅ 20.520ms (SLO: <21.750ms -5.7%) vs baseline: ~same Memory: ✅ 64.271MB (SLO: <66.000MB -2.6%) vs baseline: +4.6% ✅ tracer-and-profilerTime: ✅ 22.133ms (SLO: <23.500ms -5.8%) vs baseline: -0.5% Memory: ✅ 65.662MB (SLO: <67.000MB 🟡 -2.0%) vs baseline: +4.8% ✅ tracer-dont-create-db-spansTime: ✅ 19.444ms (SLO: <21.500ms -9.6%) vs baseline: ~same Memory: ✅ 64.350MB (SLO: <66.000MB -2.5%) vs baseline: +4.8% ✅ tracer-nativeTime: ✅ 20.424ms (SLO: <21.750ms -6.1%) vs baseline: -0.2% Memory: ✅ 65.785MB (SLO: <66.000MB 🟡 -0.3%) vs baseline: +5.0% ✅ tracer-no-cachesTime: ✅ 18.601ms (SLO: <19.650ms -5.3%) vs baseline: +0.6% Memory: ✅ 64.386MB (SLO: <66.000MB -2.4%) vs baseline: +4.9% ✅ tracer-no-databasesTime: ✅ 18.920ms (SLO: <20.100ms -5.9%) vs baseline: +0.2% Memory: ✅ 64.250MB (SLO: <66.000MB -2.7%) vs baseline: +4.8% ✅ tracer-no-middlewareTime: ✅ 20.168ms (SLO: <21.500ms -6.2%) vs baseline: ~same Memory: ✅ 64.448MB (SLO: <66.000MB -2.4%) vs baseline: +5.0% ✅ tracer-no-templatesTime: ✅ 20.377ms (SLO: <22.000ms -7.4%) vs baseline: +0.2% Memory: ✅ 64.388MB (SLO: <66.000MB -2.4%) vs baseline: +5.1% 🟡 errortrackingdjangosimple - 6/6✅ errortracking-enabled-allTime: ✅ 18.071ms (SLO: <19.850ms -9.0%) vs baseline: -0.2% Memory: ✅ 64.277MB (SLO: <65.500MB 🟡 -1.9%) vs baseline: +4.8% ✅ errortracking-enabled-userTime: ✅ 18.066ms (SLO: <19.400ms -6.9%) vs baseline: +0.2% Memory: ✅ 64.315MB (SLO: <65.500MB 🟡 -1.8%) vs baseline: +4.8% ✅ tracer-enabledTime: ✅ 18.051ms (SLO: <19.450ms -7.2%) vs baseline: -0.3% Memory: ✅ 64.221MB (SLO: <65.500MB 🟡 -2.0%) vs baseline: +4.7% 🟡 flasksimple - 17/17✅ appsec-getTime: ✅ 4.579ms (SLO: <4.750ms -3.6%) vs baseline: +0.1% Memory: ✅ 60.870MB (SLO: <64.500MB -5.6%) vs baseline: +4.8% ✅ appsec-postTime: ✅ 6.597ms (SLO: <6.750ms -2.3%) vs baseline: +0.2% Memory: ✅ 60.988MB (SLO: <64.500MB -5.4%) vs baseline: +4.9% ✅ appsec-telemetryTime: ✅ 4.571ms (SLO: <4.750ms -3.8%) vs baseline: -0.2% Memory: ✅ 60.870MB (SLO: <64.500MB -5.6%) vs baseline: +4.7% ✅ debuggerTime: ✅ 1.861ms (SLO: <2.000ms -6.9%) vs baseline: -0.2% Memory: ✅ 44.650MB (SLO: <45.000MB 🟡 -0.8%) vs baseline: +4.7% ✅ iast-getTime: ✅ 1.859ms (SLO: <2.000ms -7.0%) vs baseline: +0.2% Memory: ✅ 41.799MB (SLO: <49.000MB 📉 -14.7%) vs baseline: +5.1% ✅ profilerTime: ✅ 1.920ms (SLO: <2.100ms -8.6%) vs baseline: +0.3% Memory: ✅ 44.394MB (SLO: <46.500MB -4.5%) vs baseline: +4.7% ✅ tracerTime: ✅ 3.371ms (SLO: <3.650ms -7.7%) vs baseline: ~same Memory: ✅ 51.374MB (SLO: <53.500MB -4.0%) vs baseline: +4.9% ✅ tracer-nativeTime: ✅ 3.369ms (SLO: <3.650ms -7.7%) vs baseline: ~same Memory: ✅ 52.534MB (SLO: <53.500MB 🟡 -1.8%) vs baseline: +4.7% 🟡 flasksqli - 6/6✅ appsec-enabledTime: ✅ 3.937ms (SLO: <4.200ms -6.3%) vs baseline: +0.5% Memory: ✅ 60.873MB (SLO: <66.000MB -7.8%) vs baseline: +4.8% ✅ iast-enabledTime: ✅ 2.513ms (SLO: <2.800ms 📉 -10.3%) vs baseline: ~same Memory: ✅ 58.118MB (SLO: <59.000MB 🟡 -1.5%) vs baseline: +4.9% ✅ tracer-enabledTime: ✅ 2.080ms (SLO: <2.250ms -7.6%) vs baseline: ~same Memory: ✅ 51.102MB (SLO: <53.500MB -4.5%) vs baseline: +4.9% 🟡 otelsdkspan - 24/24✅ add-eventTime: ✅ 41.189ms (SLO: <42.000ms 🟡 -1.9%) vs baseline: +1.9% Memory: ✅ 34.269MB (SLO: <39.000MB 📉 -12.1%) vs baseline: +4.8% ✅ add-linkTime: ✅ 36.392ms (SLO: <38.550ms -5.6%) vs baseline: -0.2% Memory: ✅ 34.249MB (SLO: <39.000MB 📉 -12.2%) vs baseline: +4.7% ✅ add-metricsTime: ✅ 222.228ms (SLO: <232.000ms -4.2%) vs baseline: +1.2% Memory: ✅ 34.190MB (SLO: <39.000MB 📉 -12.3%) vs baseline: +4.5% ✅ add-tagsTime: ✅ 211.011ms (SLO: <221.600ms -4.8%) vs baseline: ~same Memory: ✅ 34.190MB (SLO: <39.000MB 📉 -12.3%) vs baseline: +4.6% ✅ get-contextTime: ✅ 29.637ms (SLO: <31.300ms -5.3%) vs baseline: +1.7% Memory: ✅ 34.249MB (SLO: <39.000MB 📉 -12.2%) vs baseline: +4.8% ✅ is-recordingTime: ✅ 29.350ms (SLO: <31.000ms -5.3%) vs baseline: -0.4% Memory: ✅ 34.269MB (SLO: <39.000MB 📉 -12.1%) vs baseline: +4.9% ✅ record-exceptionTime: ✅ 64.119ms (SLO: <65.850ms -2.6%) vs baseline: +1.4% Memory: ✅ 34.328MB (SLO: <39.000MB 📉 -12.0%) vs baseline: +5.2% ✅ set-statusTime: ✅ 32.180ms (SLO: <34.150ms -5.8%) vs baseline: +0.1% Memory: ✅ 34.328MB (SLO: <39.000MB 📉 -12.0%) vs baseline: +5.2% ✅ startTime: ✅ 29.341ms (SLO: <30.150ms -2.7%) vs baseline: +1.4% Memory: ✅ 34.308MB (SLO: <39.000MB 📉 -12.0%) vs baseline: +4.9% ✅ start-finishTime: ✅ 33.981ms (SLO: <35.350ms -3.9%) vs baseline: -0.1% Memory: ✅ 34.229MB (SLO: <39.000MB 📉 -12.2%) vs baseline: +4.6% ✅ start-finish-telemetryTime: ✅ 33.985ms (SLO: <35.450ms -4.1%) vs baseline: -0.7% Memory: ✅ 34.288MB (SLO: <39.000MB 📉 -12.1%) vs baseline: +4.8% ✅ update-nameTime: ✅ 31.049ms (SLO: <33.400ms -7.0%) vs baseline: -0.1% Memory: ✅ 34.229MB (SLO: <39.000MB 📉 -12.2%) vs baseline: +4.7% 🟡 otelspan - 22/22✅ add-eventTime: ✅ 45.360ms (SLO: <47.150ms -3.8%) vs baseline: ~same Memory: ✅ 44.627MB (SLO: <46.500MB -4.0%) vs baseline: +4.8% ✅ add-metricsTime: ✅ 320.393ms (SLO: <344.800ms -7.1%) vs baseline: +0.3% Memory: ✅ 553.902MB (SLO: <562.000MB 🟡 -1.4%) vs baseline: +4.8% ✅ add-tagsTime: ✅ 291.326ms (SLO: <314.000ms -7.2%) vs baseline: +0.4% Memory: ✅ 554.525MB (SLO: <563.500MB 🟡 -1.6%) vs baseline: +4.8% ✅ get-contextTime: ✅ 82.263ms (SLO: <92.350ms 📉 -10.9%) vs baseline: +0.1% Memory: ✅ 39.784MB (SLO: <46.500MB 📉 -14.4%) vs baseline: +5.0% ✅ is-recordingTime: ✅ 42.948ms (SLO: <44.500ms -3.5%) vs baseline: -0.5% Memory: ✅ 43.975MB (SLO: <46.500MB -5.4%) vs baseline: +4.5% ✅ record-exceptionTime: ✅ 61.672ms (SLO: <67.650ms -8.8%) vs baseline: -0.5% Memory: ✅ 40.100MB (SLO: <46.500MB 📉 -13.8%) vs baseline: +5.0% ✅ set-statusTime: ✅ 49.012ms (SLO: <50.400ms -2.8%) vs baseline: +0.1% Memory: ✅ 44.127MB (SLO: <46.500MB -5.1%) vs baseline: +5.0% ✅ startTime: ✅ 42.475ms (SLO: <43.450ms -2.2%) vs baseline: +0.4% Memory: ✅ 44.097MB (SLO: <46.500MB -5.2%) vs baseline: +4.9% ✅ start-finishTime: ✅ 82.737ms (SLO: <88.000ms -6.0%) vs baseline: +0.4% Memory: ✅ 33.974MB (SLO: <46.500MB 📉 -26.9%) vs baseline: +5.0% ✅ start-finish-telemetryTime: ✅ 84.079ms (SLO: <89.000ms -5.5%) vs baseline: ~same Memory: ✅ 33.954MB (SLO: <46.500MB 📉 -27.0%) vs baseline: +4.7% ✅ update-nameTime: ✅ 44.388ms (SLO: <45.150ms 🟡 -1.7%) vs baseline: +0.7% Memory: ✅ 44.387MB (SLO: <46.500MB -4.5%) vs baseline: +4.7%
|
2 participants
Backport 7cd4d9c from #14794 to 3.14.
Description
Currently, in Test Impact Analysis code coverage, we use
is_user_code()to decide whether to instrument a module. This function was conceived for use in non-CI contexts where user code may be installed in site-packages like external dependencies. This is not necessary in Test Optimization, where we only want to cover code inside the current repo (TIA only works with filenames that are relative to the repo root, and we already set the repo root as the include path ofModuleCodeCollector). Removing this check also resolves a bug where test packages are incorrectly detected as third-party if a third-party package contains files undertest/*ortests/*that get installed at site-packages level (sumyandsendgriddo this, for instance).Testing
Unit tests + manual testing in dogweb.
Risks
Potentially covering more code than expected. Should not be an issue because we already exclude site-packages and similar here.
Additional Notes
None.