-
Notifications
You must be signed in to change notification settings - Fork 461
feat(profiling): add Reentrant Lock (RLock) support to the Lock profiler #14823
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
944b340
to
c17cd90
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 221 ± 3 ms. The average import time from base is: 217 ± 4 ms. The import time difference between this PR and base is: 3.6 ± 0.2 ms. Import time breakdownThe following import paths have grown:
|
Performance SLOsComparing candidate vlad/lock-profiler-add-rlock (5faff64) with baseline main (3b0136e) 📈 Performance Regressions (1 suite)📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 4.323µs (SLO: <10.000µs 📉 -56.8%) vs baseline: +0.7% Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9% ✅ ospathbasename_noaspectTime: ✅ 1.080µs (SLO: <10.000µs 📉 -89.2%) vs baseline: ~same Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.0% ✅ ospathjoin_aspectTime: ✅ 6.154µs (SLO: <10.000µs 📉 -38.5%) vs baseline: -0.4% Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.8% ✅ ospathjoin_noaspectTime: ✅ 2.320µs (SLO: <10.000µs 📉 -76.8%) vs baseline: +1.0% Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.7% ✅ ospathnormcase_aspectTime: ✅ 3.510µs (SLO: <10.000µs 📉 -64.9%) vs baseline: +1.2% Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.1% ✅ ospathnormcase_noaspectTime: ✅ 0.566µs (SLO: <10.000µs 📉 -94.3%) vs baseline: -1.3% Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +4.9% ✅ ospathsplit_aspectTime: ✅ 4.871µs (SLO: <10.000µs 📉 -51.3%) vs baseline: -0.5% Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +5.1% ✅ ospathsplit_noaspectTime: ✅ 1.586µs (SLO: <10.000µs 📉 -84.1%) vs baseline: ~same Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.0% ✅ ospathsplitdrive_aspectTime: ✅ 4.031µs (SLO: <10.000µs 📉 -59.7%) vs baseline: 📈 +10.2% Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.0% ✅ ospathsplitdrive_noaspectTime: ✅ 0.698µs (SLO: <10.000µs 📉 -93.0%) vs baseline: ~same Memory: ✅ 37.631MB (SLO: <39.000MB -3.5%) vs baseline: +4.7% ✅ ospathsplitext_aspectTime: ✅ 5.277µs (SLO: <10.000µs 📉 -47.2%) vs baseline: 📈 +15.3% Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.9% ✅ ospathsplitext_noaspectTime: ✅ 1.388µs (SLO: <10.000µs 📉 -86.1%) vs baseline: ~same Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9% 🟡 Near SLO Breach (5 suites)🟡 djangosimple - 30/30✅ appsecTime: ✅ 20.507ms (SLO: <22.300ms -8.0%) vs baseline: ~same Memory: ✅ 65.381MB (SLO: <67.000MB -2.4%) vs baseline: +4.8% ✅ exception-replay-enabledTime: ✅ 1.344ms (SLO: <1.450ms -7.3%) vs baseline: -0.5% Memory: ✅ 64.541MB (SLO: <67.000MB -3.7%) vs baseline: +4.9% ✅ iastTime: ✅ 20.484ms (SLO: <22.250ms -7.9%) vs baseline: -0.1% Memory: ✅ 65.503MB (SLO: <67.000MB -2.2%) vs baseline: +4.9% ✅ profilerTime: ✅ 15.565ms (SLO: <16.550ms -6.0%) vs baseline: +2.1% Memory: ✅ 53.866MB (SLO: <54.500MB 🟡 -1.2%) vs baseline: +5.1% ✅ resource-renamingTime: ✅ 20.530ms (SLO: <21.750ms -5.6%) vs baseline: -0.2% Memory: ✅ 65.388MB (SLO: <67.000MB -2.4%) vs baseline: +4.8% ✅ span-code-originTime: ✅ 26.251ms (SLO: <28.200ms -6.9%) vs baseline: +0.5% Memory: ✅ 67.599MB (SLO: <69.500MB -2.7%) vs baseline: +4.9% ✅ tracerTime: ✅ 20.427ms (SLO: <21.750ms -6.1%) vs baseline: -0.4% Memory: ✅ 65.389MB (SLO: <67.000MB -2.4%) vs baseline: +4.8% ✅ tracer-and-profilerTime: ✅ 22.625ms (SLO: <23.500ms -3.7%) vs baseline: +2.6% Memory: ✅ 66.894MB (SLO: <67.500MB 🟡 -0.9%) vs baseline: +5.4% ✅ tracer-dont-create-db-spansTime: ✅ 19.337ms (SLO: <21.500ms 📉 -10.1%) vs baseline: ~same Memory: ✅ 65.398MB (SLO: <66.000MB 🟡 -0.9%) vs baseline: +5.0% ✅ tracer-minimalTime: ✅ 16.679ms (SLO: <17.500ms -4.7%) vs baseline: +0.2% Memory: ✅ 65.463MB (SLO: <66.000MB 🟡 -0.8%) vs baseline: +5.0% ✅ tracer-nativeTime: ✅ 20.544ms (SLO: <21.750ms -5.5%) vs baseline: +0.4% Memory: ✅ 71.386MB (SLO: <72.500MB 🟡 -1.5%) vs baseline: +4.7% ✅ tracer-no-cachesTime: ✅ 18.473ms (SLO: <19.650ms -6.0%) vs baseline: ~same Memory: ✅ 65.502MB (SLO: <67.000MB -2.2%) vs baseline: +5.1% ✅ tracer-no-databasesTime: ✅ 18.804ms (SLO: <20.100ms -6.5%) vs baseline: ~same Memory: ✅ 65.185MB (SLO: <67.000MB -2.7%) vs baseline: +4.9% ✅ tracer-no-middlewareTime: ✅ 20.159ms (SLO: <21.500ms -6.2%) vs baseline: -0.2% Memory: ✅ 65.471MB (SLO: <67.000MB -2.3%) vs baseline: +5.0% ✅ tracer-no-templatesTime: ✅ 20.311ms (SLO: <22.000ms -7.7%) vs baseline: ~same Memory: ✅ 65.496MB (SLO: <67.000MB -2.2%) vs baseline: +5.0% 🟡 errortrackingdjangosimple - 6/6✅ errortracking-enabled-allTime: ✅ 18.033ms (SLO: <19.850ms -9.2%) vs baseline: -0.5% Memory: ✅ 65.195MB (SLO: <66.500MB 🟡 -2.0%) vs baseline: +4.7% ✅ errortracking-enabled-userTime: ✅ 18.034ms (SLO: <19.400ms -7.0%) vs baseline: -0.6% Memory: ✅ 65.352MB (SLO: <66.500MB 🟡 -1.7%) vs baseline: +5.1% ✅ tracer-enabledTime: ✅ 18.074ms (SLO: <19.450ms -7.1%) vs baseline: -0.2% Memory: ✅ 65.254MB (SLO: <66.500MB 🟡 -1.9%) vs baseline: +4.9% 🟡 flasksimple - 18/18✅ appsec-getTime: ✅ 4.567ms (SLO: <4.750ms -3.9%) vs baseline: ~same Memory: ✅ 62.010MB (SLO: <65.000MB -4.6%) vs baseline: +4.9% ✅ appsec-postTime: ✅ 6.566ms (SLO: <6.750ms -2.7%) vs baseline: -0.2% Memory: ✅ 61.971MB (SLO: <65.000MB -4.7%) vs baseline: +4.8% ✅ appsec-telemetryTime: ✅ 4.575ms (SLO: <4.750ms -3.7%) vs baseline: +0.1% Memory: ✅ 62.049MB (SLO: <65.000MB -4.5%) vs baseline: +5.1% ✅ debuggerTime: ✅ 1.855ms (SLO: <2.000ms -7.3%) vs baseline: -0.3% Memory: ✅ 45.515MB (SLO: <47.000MB -3.2%) vs baseline: +4.7% ✅ iast-getTime: ✅ 1.860ms (SLO: <2.000ms -7.0%) vs baseline: ~same Memory: ✅ 42.290MB (SLO: <49.000MB 📉 -13.7%) vs baseline: +4.5% ✅ profilerTime: ✅ 1.911ms (SLO: <2.100ms -9.0%) vs baseline: -0.1% Memory: ✅ 46.517MB (SLO: <47.000MB 🟡 -1.0%) vs baseline: +4.9% ✅ resource-renamingTime: ✅ 3.377ms (SLO: <3.650ms -7.5%) vs baseline: -0.4% Memory: ✅ 52.219MB (SLO: <53.500MB -2.4%) vs baseline: +4.8% ✅ tracerTime: ✅ 3.367ms (SLO: <3.650ms -7.8%) vs baseline: ~same Memory: ✅ 52.180MB (SLO: <53.500MB -2.5%) vs baseline: +4.8% ✅ tracer-nativeTime: ✅ 3.362ms (SLO: <3.650ms -7.9%) vs baseline: ~same Memory: ✅ 58.284MB (SLO: <60.000MB -2.9%) vs baseline: +4.8% 🟡 otelspan - 22/22✅ add-eventTime: ✅ 42.318ms (SLO: <47.150ms 📉 -10.2%) vs baseline: -0.1% Memory: ✅ 44.428MB (SLO: <47.000MB -5.5%) vs baseline: +4.7% ✅ add-metricsTime: ✅ 316.310ms (SLO: <344.800ms -8.3%) vs baseline: -1.3% Memory: ✅ 595.816MB (SLO: <600.000MB 🟡 -0.7%) vs baseline: +5.0% ✅ add-tagsTime: ✅ 287.127ms (SLO: <314.000ms -8.6%) vs baseline: -0.4% Memory: ✅ 597.475MB (SLO: <600.000MB 🟡 -0.4%) vs baseline: +4.9% ✅ get-contextTime: ✅ 80.563ms (SLO: <92.350ms 📉 -12.8%) vs baseline: -0.2% Memory: ✅ 39.947MB (SLO: <46.500MB 📉 -14.1%) vs baseline: +5.1% ✅ is-recordingTime: ✅ 38.983ms (SLO: <44.500ms 📉 -12.4%) vs baseline: -0.1% Memory: ✅ 43.903MB (SLO: <47.500MB -7.6%) vs baseline: +4.8% ✅ record-exceptionTime: ✅ 58.596ms (SLO: <67.650ms 📉 -13.4%) vs baseline: -0.3% Memory: ✅ 40.251MB (SLO: <47.000MB 📉 -14.4%) vs baseline: +4.7% ✅ set-statusTime: ✅ 44.798ms (SLO: <50.400ms 📉 -11.1%) vs baseline: +0.3% Memory: ✅ 43.942MB (SLO: <47.000MB -6.5%) vs baseline: +4.8% ✅ startTime: ✅ 38.269ms (SLO: <43.450ms 📉 -11.9%) vs baseline: ~same Memory: ✅ 43.947MB (SLO: <47.000MB -6.5%) vs baseline: +5.1% ✅ start-finishTime: ✅ 82.651ms (SLO: <88.000ms -6.1%) vs baseline: +0.2% Memory: ✅ 34.564MB (SLO: <46.500MB 📉 -25.7%) vs baseline: +5.0% ✅ start-finish-telemetryTime: ✅ 83.783ms (SLO: <89.000ms -5.9%) vs baseline: ~same Memory: ✅ 34.583MB (SLO: <46.500MB 📉 -25.6%) vs baseline: +4.8% ✅ update-nameTime: ✅ 40.507ms (SLO: <45.150ms 📉 -10.3%) vs baseline: +1.1% Memory: ✅ 44.164MB (SLO: <47.000MB -6.0%) vs baseline: +4.7% 🟡 span - 26/26✅ add-eventTime: ✅ 20.644ms (SLO: <22.500ms -8.3%) vs baseline: -0.2% Memory: ✅ 50.334MB (SLO: <53.000MB -5.0%) vs baseline: +4.9% ✅ add-metricsTime: ✅ 90.620ms (SLO: <93.500ms -3.1%) vs baseline: -0.2% Memory: ✅ 661.217MB (SLO: <961.000MB 📉 -31.2%) vs baseline: +4.8% ✅ add-tagsTime: ✅ 148.467ms (SLO: <155.000ms -4.2%) vs baseline: -0.2% Memory: ✅ 661.760MB (SLO: <962.500MB 📉 -31.2%) vs baseline: +4.9% ✅ get-contextTime: ✅ 19.336ms (SLO: <20.500ms -5.7%) vs baseline: ~same Memory: ✅ 49.138MB (SLO: <53.000MB -7.3%) vs baseline: +4.7% ✅ is-recordingTime: ✅ 19.555ms (SLO: <20.500ms -4.6%) vs baseline: -0.2% Memory: ✅ 49.199MB (SLO: <53.000MB -7.2%) vs baseline: +5.0% ✅ record-exceptionTime: ✅ 38.484ms (SLO: <40.000ms -3.8%) vs baseline: +0.5% Memory: ✅ 42.759MB (SLO: <53.000MB 📉 -19.3%) vs baseline: +4.8% ✅ set-statusTime: ✅ 21.250ms (SLO: <22.000ms -3.4%) vs baseline: +0.5% Memory: ✅ 49.083MB (SLO: <53.000MB -7.4%) vs baseline: +4.6% ✅ startTime: ✅ 19.065ms (SLO: <20.500ms -7.0%) vs baseline: +0.8% Memory: ✅ 49.134MB (SLO: <53.000MB -7.3%) vs baseline: +4.7% ✅ start-finishTime: ✅ 51.520ms (SLO: <52.500ms 🟡 -1.9%) vs baseline: +0.2% Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.7% ✅ start-finish-telemetryTime: ✅ 53.057ms (SLO: <54.500ms -2.6%) vs baseline: +0.5% Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +4.9% ✅ start-finish-traceid128Time: ✅ 55.110ms (SLO: <56.000ms 🟡 -1.6%) vs baseline: +0.6% Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.7% ✅ start-traceid128Time: ✅ 19.505ms (SLO: <22.500ms 📉 -13.3%) vs baseline: -0.4% Memory: ✅ 49.111MB (SLO: <53.000MB -7.3%) vs baseline: +4.9% ✅ update-nameTime: ✅ 20.487ms (SLO: <22.000ms -6.9%) vs baseline: +2.0% Memory: ✅ 49.826MB (SLO: <53.000MB -6.0%) vs baseline: +4.8%
|
195b3c8
to
22f1608
Compare
cbe5cdd
to
a7fb6e2
Compare
0520a6b
to
4c62a2c
Compare
79da982
to
3f50634
Compare
3f50634
to
7e7549f
Compare
assert "mock failure message" in message | ||
|
||
|
||
@pytest.mark.skipif(not sys.platform.startswith("linux"), reason="only works on linux") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was moved here from test_threading.py
, since it has nothing to do with synchronization. Here is a better location for this test.
def test_global_locks(self): | ||
with collector_threading.ThreadingLockCollector(capture_pct=100): | ||
from tests.profiling.collector import global_locks | ||
def test_global_locks(self) -> None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This test has been reworked to not import global locks from an external module. Instead, it uses new global variables in its own module. Functionally, it is the same exact behavior, but we are avoiding several tricky dependency issues stemming from importing a global object.
This will simplify the test setup, and allow to delete the global_locks.py
file completely.
7e7549f
to
5e35980
Compare
…belong with threading)
435e9ec
to
5faff64
Compare
https://datadoghq.atlassian.net/browse/PROF-12721
Description
This PR adds profiling support for a new kind of synchronization mechanism: Reentrant Lock. A RLock is a lock that can be acquired by the same thread multiple times without blocking on itself. From the documentation:
This feature was identified as one of the most - low-hanging-fruits, since
RLock
s are being used extensively in Python world.Changes
ThreadingRLockCollector
class inthreading.py
profiler.py
test_threading.py
module has been updated to reuse existing tests for bothLock
andRLock
primitives (via new inheritance)test_global_locks
test rewritten extensively to support the unit test code reuse betweenLock
andRLock
(effectively duplicating the existing extensiveLock
unit tests forRLock
... without the code duplication)Testing
Risks
none
Additional Notes