[BUG]: +150 MB RSS per process in a Django app, from 4.9.0 onward, only when tracing is enabled

[bellini666]

Tracer Version(s)

4.9.0 (first affected), still present until even on 4.13.0rc3

Python Version(s)

3.13.13

Pip Version(s)

uv 0.11.1

Bug Report

Upgrading from 4.8.13 to any 4.9+ release costs us a flat +150 MB RSS per worker process in a large Django app. The jump is present from the moment the process finishes booting, it does not grow over time, and it is entirely attributable to the tracer being enabled.

We hit this in production first: a Dramatiq worker pod sat at a rock-steady 665 MB for weeks, went to 868 MB when we deployed 4.11.0, and returned to 665 MB the moment we reverted. Same story on every worker deployment we checked. Our gunicorn deployments have a wide enough RSS band (±300 MB) that a step this size is invisible there, so we cannot tell whether they are affected too.

Then we reproduced it locally, ran the version matrix, and ablated every 4.9.0 default change we could find. Numbers below.

Version matrix (identical container, identical env, ddtrace swapped by prepending a pip install --target directory to PYTHONPATH, 70 s settle after boot, VmRSS from /proc/<pid>/status):

ddtrace	dramatiq master	worker	total
4.8.13	61 MB	577-582 MB	634-643 MB
4.9.0	64 MB	738 MB	802 MB
4.9.7	58 MB	731 MB	789 MB
4.10.11	62 MB	736 MB	799 MB
4.11.0	61 MB	735 MB	796 MB
4.11.4	61 MB	736 MB	797 MB
4.12.2	63 MB	738 MB	801 MB
4.13.0rc3	58 MB	738 MB	796 MB

4.8.13 was measured four times, interleaved between the other runs, and landed at 577, 577, 581, and 582 MB. The entire regression appears at 4.9.0 and has not moved since. Current stable and the 4.13 candidate carry it unchanged.

Ablations (all negative):

Change	4.8.13	4.9.7
DD_TRACE_ENABLED=false	558 MB	558 MB
agent reachable (local sink on :8126)	582 MB	731 MB
agent unreachable	581 MB	732 MB
DD_CODE_ORIGIN_FOR_SPANS_ENABLED=false		732 MB
DD_SYMBOL_DATABASE_UPLOAD_ENABLED=false		735 MB (on 4.11.4)
DD_CRASHTRACKING_COLLECT_ALL_THREADS=false		735 MB (on 4.11.4)

With DD_TRACE_ENABLED=false the two versions agree to within 1 MB, so this is not the import hooks, the integration patching, or module-load cost. It is the tracer. Agent reachability makes no difference either way, which rules out buffering against a dead agent.

We started with code origin for spans as the prime suspect, since dd.code_origin.for_spans.enabled flips from False to True in 4.9.0 (ddtrace/internal/settings/code_origin.py). Disabling it changes nothing. Same for the symbol database and for crashtracker.collect_all_threads, the other two defaults that changed in that release.

Where the memory sits. /proc/<pid>/smaps for the worker, aggregated by mapping:

                4.8.13      4.9.7
[anon]          349.8 MB    411.0 MB
[heap]          182.4 MB    274.7 MB
_native...so      4.7 MB      4.5 MB
libpython3.13     3.8 MB      3.8 MB
(all other mappings identical to within 0.2 MB)
TOTAL           582.1 MB    735.3 MB

Ordinary process allocation, brk heap plus mmap. No mapped library grew.

When it happens. Importing the tracer is cheap: RSS after import ddtrace.auto is 47.8 MB on 4.8.13 and 49.2 MB on 4.9.7. The delta lands during django.setup(), which takes the process to 579.2 MB on 4.8.13 and 734.1 MB on 4.9.7. Our AppConfig.ready() hooks warm several caches with Postgres and Redis queries, so a few hundred spans are created and flushed while the app boots. That is the only tracing activity before the measurement.

App size matters. A bare ddtrace.auto process in the same container, generating 20k spans by hand and then idling, shows 89.6 MB on 4.8.13 versus 93.3 MB on 4.9.7. So whatever this is scales with the process, and I could not shrink it to a small standalone script. Sorry about that. Hopefully the ablation matrix narrows the 4.8 to 4.9 diff enough to be useful.

Existing issues I checked and ruled out. #18781 is also a 4.9 boundary but a linear leak tied to ClientResponse retention in the aiohttp client, and we do not use aiohttp. #17063 is gradual growth plus large disk usage on falcon, reported against 1.20.8 and 4.2.3. #16491 is the memory profiler, which we have never enabled. #18800 needs DD_DBM_PROPAGATION_MODE set, and ours is at the disabled default.

NOTE: This report was generated with Claude 🤖

Reproduction Code

I have no minimal repro, only a recipe. Any Django project big enough to reach a few hundred MB of resident heap should show it. Dramatiq is not required, it is just where our signal was cleanest: one process, one thread, flat baseline. A --preload gunicorn worker shows the same step, it is only harder to see under normal traffic variance.

Install the version under test next to the app venv, so nothing in the venv changes:

python -m pip install --no-deps --target /tmp/dd/4.9.0 ddtrace==4.9.0

Same env for every run:

export DD_TRACE_ENABLED=true DD_SERVICE=app DD_ENV=probe
export DD_PSYCOPG_TRACE_CONNECT=true DD_INSTRUMENTATION_TELEMETRY_ENABLED=true

Boot the worker, let it settle, read VmRSS of both the dramatiq master and the worker child:

measure() {
  ddtrace-run python manage.py rundramatiq \
    --processes 1 --threads 1 --queues some_queue >/tmp/run.log 2>&1 &
  sleep 70
  for p in $(pgrep -f 'bin/dramatiq --path' | sort -n); do
    awk -v p=$p '/VmRSS/ {printf "  pid=%s rss=%dMB\n", p, $2/1024}' /proc/$p/status
  done
  pkill -f 'bin/dramatiq --path'; sleep 5; pkill -9 -f 'bin/dramatiq --path'; sleep 3
}

# baseline, 4.8.13 installed in the venv
measure

# 4.9.0
PYTHONPATH=/tmp/dd/4.9.0:$PYTHONPATH measure

Expect roughly 61 MB master plus 580 MB worker on 4.8.13, and 64 MB plus 738 MB on 4.9.0.

Repeat both with DD_TRACE_ENABLED=false and the two versions collapse onto the same number (558 MB worker for us), which is what points at the tracer rather than at import or patching cost.

To confirm a reachable agent makes no difference, run a sink on port 8126 that answers every request with 200 and a body of {"rate_by_service":{}}, then re-run with DD_TRACE_AGENT_URL=http://127.0.0.1:8126. We measured 582 MB versus 731 MB with the sink up, and 581 MB versus 732 MB with nothing listening.

For the mapping breakdown, aggregate Rss: per mapping name from /proc/<worker-pid>/smaps:

python3 - "$WORKER_PID" <<'PY'
import sys, re, collections
pid = sys.argv[1]
agg, name = collections.Counter(), None
for line in open(f"/proc/{pid}/smaps"):
    m = re.match(r"^[0-9a-f]+-[0-9a-f]+ \S+ \S+ \S+ \S+\s*(.*)$", line)
    if m:
        n = m.group(1).strip() or "[anon]"
        name = n if n.startswith("[") else n.split("/")[-1]
        continue
    if line.startswith("Rss:") and name:
        agg[name] += int(line.split()[1])
for n, kb in agg.most_common(12):
    print(f"{kb/1024:9.1f}MB  {n}")
print(f"{sum(agg.values())/1024:9.1f}MB  TOTAL")
PY

Error Logs

None. Nothing unusual in the tracer's startup or debug logs, and no warnings from the writer.

Libraries in Use

• Django 5.2.16
• dramatiq 1.18.0
• psycopg 3.x, redis, kombu, requests (all patched; the startup line reports django,requests,redis,kombu,dramatiq)

Relevant env in both the local repro and production:

DD_TRACE_ENABLED=true
DD_PATCH_MODULES=django:false
DD_DJANGO_TRACING_MINIMAL=true
DD_PSYCOPG_TRACE_CONNECT=true
DD_TRACE_SAMPLING_RULES=[...]

Profiling, AppSec, IAST, DBM propagation, and dynamic instrumentation are all off. There are no datadog.profiling.* metrics anywhere in our org, so the profiler has never run.

Operating System

Debian 13 (trixie), glibc 2.41, aarch64, 6 CPUs, cgroup v2 with no CPU quota.

[bellini666]

Found it, and it is not a tracer memory cost at all. The tracer is importing our whole URLconf.

Root cause

ddtrace/contrib/internal/django/patch.py, added in #17695 and first shipped in 4.9.0rc1:

# Eager endpoint discovery: walk the ROOT_URLCONF resolver now that apps
# are ready, so the telemetry "app-endpoints" payload is populated at
# startup (not only when a request arrives).
try:
    from django.urls import get_resolver

    _collect_routes_once(get_resolver(None))
except Exception:
    log.debug("Error collecting Django routes for endpoint discovery", exc_info=True)

That sits at the end of traced_populate, the wrapper around django.apps.registry.Apps.populate.
django.setup() calls populate() directly, so it fires in every process that sets Django up, not
only where HTTP is served. Reading resolver.url_patterns imports ROOT_URLCONF, which through
include() imports every view module and everything those views pull in behind them.

A dramatiq worker never resolves a URL. On 4.8.13 none of that graph is ever imported. On 4.9 all of
it loads at boot. For our app that is 977 extra modules and 308k extra live GC objects, which is the
154 MB.

The cost is the import closure behind the routes, not the number of routes. 12k routes declared in
modules that are already imported cost nothing, so reproducing this needs view modules that nothing
else imports:

ddtrace	RSS	modules	url-only modules imported
4.8.13	71.5 MB	968	0
4.9.0rc1	183.6 MB	1272	180

A minimal reproduction is small: one Django project, views in modules imported only from the
URLconf, then

import ddtrace.auto
import django, sys
django.setup()
print("myproj.urls" in sys.modules)

4.8.13 prints False, 4.9.0rc1 prints True. Scale the delta by making those view modules import
something heavy; ours pull in django-ninja and Pydantic schema construction.

Supporting measurements

• The memory is live, not retained heap. malloc_trim(0) reclaims 31.3 MB on 4.8.13 and 0.5-0.7 MB
  on 4.9.x. malloc_info process <system type="current"> goes 183.6 to 275.4 MB, and
  <total type="rest"> is 0 on every version. No fragmentation involved.
• It is module objects. sys.modules 6468 to 7443, GC objects 893,789 to 1,184,595, and
  <ourapp>.urls is absent on 4.8.13 and present on 4.9. The per-package breakdown is dominated by
  our API and frontend view packages and the SDKs they import.
• The boundary is rc1, not rc2. 4.9.0rc1 already shows the full step, so neither of the 4.9 native
  storage migrations is involved. rc2 adds about 2.6 MB of native span attribute work on top.
• Under PYTHONTRACEMALLOC the new-on-4.9 allocation sites are ninja/schema.py,
  pydantic/_internal/_model_construction.py and _fields.py, which only run when the API view
  modules import. No single site owns the 154 MB; it is spread across roughly a thousand imports.

Not a configuration bug

DD_PATCH_MODULES=django:false and DD_TRACE_DJANGO_ENABLED=false both work correctly. Checking
type(Apps.populate) right after import ddtrace.auto shows a plain function under either flag and
a BoundFunctionWrapper without them. Our own settings module calls
patch(django=True, requests=True, redis=True, kombu=True, dramatiq=True) unconditionally, which is
what re-enabled the integration in our earlier measurements. Nothing for ddtrace to fix there.

Note that "django" in ddtrace._monkey._PATCHED_MODULES is not a useful check: it reports True on
4.8.13 too, and lists integrations whose libraries are not even installed.

Impact beyond memory

Every view module, serializer, and third-party SDK behind the URLconf now imports in every worker,
management command, and cron job. Slower boots, and a much larger import surface in processes that
were never meant to touch the HTTP layer.

PR to follow: walk the resolver at startup only when ROOT_URLCONF is already in sys.modules, and
otherwise leave it to the existing per-request walk. That keeps the startup payload wherever it costs
nothing and forces no imports.

---

Generated by claude 🤖

[christophe-papazian]

Hi @bellini666
Thanks for the detailed report and the PR proposition, I'll work on it!

[christophe-papazian]

Hi @bellini666
Would you be able to test #19467 ?
The last commit f0b229c should be the right candidate to test it.

[bellini666]

Hi @bellini666 Would you be able to test #19467 ? The last commit f0b229c should be the right candidate to test it.

Asked claude to test it and compare with the baseline and also my previous attempt to fix it, here are the results

So yep, it fixes it 😊

Thank you very much!

[christophe-papazian]

The fix was merged and is scheduled to be included in the release 4.14.0rc1 and above.

[bellini666]

The fix was merged and is scheduled to be included in the release 4.14.0rc1 and above.

Thank you! 😊

Is it only going to be available on 4.14.x, or will it be backported to previous versions?

[christophe-papazian]

As it is a substantial change, I would prefer to wait for our quality process/internal deployment to ensure all is well.
It could be later backported to 4.13, but probably not much before.
Upgrading from 4.9 to 4.14 should be painless.