contributing-testing.rst

Testing

Imagine you're making a change to the library.

If your change touches Python code, it should probably include at least one test.

What kind of tests should I write?

We use heuristics to decide when and what sort of tests to write. For example, a pull request implementing a new feature should include enough unit tests to cover the feature's "happy path" use cases in addition to any known likely edge cases. If the feature involves a new form of communication with another component (like the Datadog Agent or libddwaf), it should probably include at least one integration test exercising the end-to-end communication.

If a pull request fixes a bug, it should include a test that, on the trunk branch, would replicate the bug. Seeing this test pass on the fix branch gives us confidence that the bug was actually fixed.

Where do I put my tests?

Put your code's tests in the appropriate subdirectory of the tests directory based on what they are testing. If your feature is substantially new, you may decide to create a new tests subdirectory in the interest of code organization.

How do I run the test suite?

Prerequisites

Install and run:

• docker
• uv

Easy way: Use scripts/run-tests

The scripts/run-tests script handles this automatically:

# Run test suites for your current changes
$ scripts/run-tests

# Run test suites affected by source changes
$ scripts/run-tests ddtrace/contrib/django/patch.py
$ scripts/run-tests ddtrace/internal/core/event_hub.py

# Run test suites containing these tests
$ scripts/run-tests tests/contrib/django/
$ scripts/run-tests tests/contrib/flask/test_flask.py

Manual approach with ddtest

This repo includes a Docker container definition that provides a pre-built test environment. You can access it by running

$ scripts/ddtest

Some of our test suites are managed with Riot.

You can run riot commands and lint checks in the test runner container with commands like these:

$ scripts/ddtest riot run -p 3.10
$ scripts/ddtest scripts/lint style

How do I run only the tests I care about?

Easy way: Use scripts/run-tests

The scripts/run-tests script handles this automatically:

# Add riot arguments to avoid unnecessary compilation
$ scripts/run-tests tests/contrib/django/ -- -s

# Add pytest arguments for test selection
$ scripts/run-tests tests/contrib/django/ -- -- -k test_specific_function

# Add both riot (first) and pytest (second) arguments
$ scripts/run-tests ddtrace/contrib/django/patch.py -- -s -- -vvv -s --tb=short

# Run specific test functions
$ scripts/run-tests tests/contrib/flask/ -- -k "test_request or test_response"

Manual way: Direct riot commands

If you prefer manual control:

1. Note the names of the tests you care about - these are the "test names".
2. Find the Venv in the riotfile whose command contains the tests you're interested in. Note the Venv's name - this is the "suite name".
3. Find the suite in the file ./tests/contrib/suitespec.yml whose pattern is equal to the suite name. Note the docker_services section of the directive, if present - these are the "suite services".
4. Start the suite services, if applicable, with $ docker compose up -d service1 service2.
5. Start the test-runner Docker container with $ scripts/ddtest.
6. In the test-runner shell, run the tests with $ riot -v run --pass-env -p 3.10 <suite_name> -- -s -vv -k 'test_name1 or test_name2'.

Anatomy of a Riot Command

$ riot -v run --pass-env -s -p 3.10 <suite_name> -- -s -vv -k 'test_name1 or test_name2'

• -v: Print verbose output
• --pass-env: Pass all environment variables in the current shell to the pytest invocation
• -s: Skips base install. Ensure you have already generated the base virtual environment(s) before using this flag.
• -p 3.10: Run the tests using Python 3.10. You can change the version string if you want.
• <suite_name>: A regex matching the names of the Riot Venv instances to run
• --: Everything after this gets treated as a pytest argument
• -s: Make potential uses of pdb work properly
• -vv: Be loud about which tests are being run
• -k 'test1 or test2': Test selection by keyword expression

How CI allocates Riot environments

Semantic test ownership remains in tests/suitespec.py and the distributed suitespec.yml files. CI resolves every selected suite to its Riot environment hashes, then assigns those hashes to physical GitLab shards. A Riot environment is the smallest allocation unit, so duration-based balancing cannot change its command, Python version, services, environment, retry policy, or timeout.

scripts/gen_gitlab_config.py writes .gitlab/ci-allocation-plan.json with the current round-robin plan and a duration-balanced plan. Generation fails unless both plans contain the exact same Riot hash set, with no duplicates or empty shards, and the execution metadata is represented by the same digest. The plan is retained as a CI artifact for review.

The active strategy and promotion thresholds are in ci/ci-allocation-policy.json. legacy is the rollback-safe default. The duration estimates in ci/ci-allocation-runtime-model.json are generated from Datadog Test Visibility session exports joined by test.configuration.riot_hash. The model uses a time-decayed p90, conservative fallbacks for sparse hashes, and a recent holdout that is not used for fitting. CI job events are joined by pipeline and job identity to account for setup overhead, queue time, and total runner consumption. Failed and cancelled observations are retained as censored reliability evidence but are not treated as normal durations.

Use the allocation helper to normalize an export, build a candidate model, and replay it against the untouched holdout:

$ scripts/ci_allocation_cli.py ingest-datadog \
    --input test-sessions.json --output observations.jsonl
$ scripts/ci_allocation_cli.py ingest-jobs \
    --input ci-jobs.json --output jobs.jsonl
$ scripts/ci_allocation_cli.py build-model \
    --observations observations.jsonl --jobs jobs.jsonl \
    --output candidate-model.json --report historical-replay.json
$ scripts/ci_allocation_cli.py check-ratchet \
    --report historical-replay.json

Historical pull-request paths provide a second workload view. They are replayed through the current suitespec rules, so common PR cohorts cannot hide a regression in less frequent AppSec, integration, CI, or core workloads:

$ scripts/ci_allocation_cli.py export-pr-history \
    --since "2 years ago" --output pr-shapes.jsonl
$ scripts/ci_allocation_cli.py replay-pr-history \
    --pr-history pr-shapes.jsonl --model candidate-model.json \
    --output pr-replay.json
$ scripts/ci_allocation_cli.py check-ratchet --report pr-replay.json

After historical validation, set CI_ALLOCATION_SHADOW=true on an explicitly requested pipeline to add non-blocking balanced jobs beside the required legacy jobs. Export both strategies' Test Visibility sessions, then build and check the same-head report:

$ scripts/ci_allocation_cli.py build-live-report \
    --observations shadow-observations.jsonl --jobs shadow-jobs.jsonl \
    --output live-shadow.json
$ scripts/ci_allocation_cli.py check-ratchet --report live-shadow.json

Before promotion, download the legacy and balanced JUnit artifacts and prove that the collected (Riot hash, class, test, file) identity multisets and Riot execution metadata are identical. Allocation jobs encode the strategy and Riot hash in each JUnit filename as a fallback for xdist runs that omit testsuite properties:

$ scripts/ci_allocation_cli.py verify-junit \
    --legacy legacy/test-results/junit*.xml \
    --balanced balanced/test-results/junit*.xml \
    --output junit-parity.json

Promotion requires the checked thresholds for sample count, median improvement, p75, p90, runner time, clean-success rate, and retry rate. A scheduled retuning task may propose a new model, but it must not change the active strategy automatically. Activate balanced only after the historical and live ratchets pass; reverting the policy to legacy restores the previous assignment without changing suite authoring.

Why are my tests failing with 404 errors?

If your test relies on the testagent service, you might see it fail with a 404 error. To fix this:

# outside of the testrunner shell
$ docker compose up -d testagent

# inside the testrunner shell, started with scripts/ddtest
$ DD_AGENT_PORT=9126 riot -v run --pass-env ...

Why are my Docker tests failing with permission errors on Linux?

On Linux systems, when running tests with scripts/ddtest or scripts/run-tests, you may encounter permission errors or file ownership issues. This happens because the container user's ID and group ID must match your local user's IDs.

To fix this, create a docker-compose.override.yml file in the repository root with the following contents:

services:
  testrunner:
    user: "${UID}:${GID}"

Then, ensure your shell has the UID and GID environment variables set:

export UID="$(id -u)"
export GID="$(id -g)"

You can add these exports to your shell profile (e.g., .bashrc, .zshrc) to make them persistent across sessions.

After setting this up, run your tests normally:

$ scripts/ddtest
$ scripts/run-tests

The docker-compose.override.yml file is git-ignored and won't be committed, so each developer can have their own local configuration.

Build issues when running tests with Riot

If you encounter build failures, CMake errors, or stale native extension issues when running tests:

• Installing ddtrace locally (e.g. pip install -e .): See :ref:`build-failures-local-install` for the clean command.
• Using scripts/ddtest: The project is mounted from the host, so run scripts/clean on the host first. The container sees the cleaned project on the next run.

Then run Riot without the -s flag so that ddtrace is rebuilt from source. The -s flag skips the base install; omitting it forces a fresh build:

$ riot -v run --pass-env -p 3.10 <suite_name> -- -vv -k 'test_name'

Once the build succeeds, you can use -s again for faster subsequent runs.

Why is my CI run failing with a message about requirements files?

.riot/requirements contains requirements files generated with pip-compile for every environment specified by riotfile.py. Riot uses these files to build its environments, and they do not get rebuilt automatically when the riotfile changes. Thus, if you make changes to the riotfile, you need to rebuild them.

$ scripts/ddtest scripts/compile-and-prune-test-requirements

You can commit and pull request the resulting changes to files in .riot/requirements alongside the changes you made to riotfile.py.

Why is my CI run failing with benchmark or Service Level Objective (SLO) threshold breaches?

The library includes automated SLO checks that monitor performance thresholds for execution time and memory usage. If your pull request causes these checks to fail, you'll see benchmark test failures in CI indicating that your changes have caused performance to exceed established thresholds.

If this is expected additional overhead:

1. Add a comment to your PR description explaining why the performance change is expected and necessary

2. Update the failing thresholds in .gitlab/benchmarks/bp-runner.microbenchmarks.fail-on-breach.yml following these guidelines:

   For execution time thresholds:

   • Take the new benchmark result from CI
   • Add 2% overhead for variance
   • Round up to a reasonable precision
   • Example: 23.1 ms → 23.1 * 1.02 = 23.562 ms → round to 23.60 ms

   For memory usage thresholds:

   • Take the new benchmark result from CI
   • Add 5% overhead for variance
   • Round up to a reasonable precision
   • Consider unifying similar scenarios to the same threshold (e.g., set all tracer scenarios to < 32.00 MB instead of having slightly different values)

Example threshold update:

- name: span-start
  thresholds:
    - execution_time < 23.60 ms  # was 23.50 ms
    - max_rss_usage < 48.00 MB   # was 47.50 MB

How do I add a new test suite?

We use riot, a Python virtual environment constructor, to run the test suites. It is necessary to create a new Venv instance in riotfile.py if it does not exist already. It can look like this:

Venv(
    name="yaaredis",
    command="pytest {cmdargs} tests/contrib/yaaredis",
    pkgs={
        "pytest-asyncio": "==0.21.1",
        "pytest-randomly": latest,
    },
    venvs=[
        Venv(
            pys=select_pys(min_version="3.8", max_version="3.9"),
            pkgs={"yaaredis": ["~=2.0.0", latest]},
        ),
    ],
),

Once a Venv instance has been created, you will be able to run it as explained in the section below. Next, we will need to add a new CI job to run the newly added test suite. This change can be made in the tests/contrib/suitespec.yml file:

yaaredis:
  parallelism: 1
  paths:
    - '@core'
    - '@bootstrap'
    - '@contrib'
    - '@tracing'
    - '@redis'
    - tests/contrib/yaaredis/*
    - tests/snapshots/tests.contrib.yaaredis.*
  pattern: yaaredis$
  services:
    - redis
  snapshot: true

See tests/README.md for more detail on adding new CI jobs.

How do I update a Riot environment to use the latest version of a package?

Reading through the above example and others in riotfile.py, you may notice that some package versions are specified as the variable latest. When the Riotfile is compiled into the .txt files in the .riot directory, latest tells the compiler to pin the newest version of the package available on PyPI according to semantic versioning.

Because this version resolution happens during Riotfile compilation, latest doesn't always mean "latest" once the compiled requirements files are checked into source control. In order to stay current, these requirements files need to be recompiled periodically.

Assume you have a Venv instance in the Riotfile that uses the latest variable. Note the name field of this environment object.

1. Run scripts/ddtest to enter a shell in the testrunner container
2. export VENV_NAME=<name_you_noted_above>
3. Delete all of the requirements lockfiles for the chosen environment, then regenerate them: for h in `riot list --hash-only "^${VENV_NAME}$"`; do rm .riot/requirements/${h}.txt; done; scripts/compile-and-prune-test-requirements
4. Commit the resulting changes to the .riot directory, and open a pull request against the trunk branch.

Why isn't my lint dependency change taking effect?

If you update tool versions in the [dependency-groups] lint section of pyproject.toml, uv will pick up the change automatically on the next run. To force a clean reinstall of the lint environment, clear the uv cache:

$ uv cache clean

What do I do when my pull request has failing tests unrelated to my changes?

The test suite is not completely reliable. There are usually some tests that can fail without any of their code paths being changed. This slows down development because most tests are required to pass for pull requests to be merged.

The tests/utils module provides the @flaky decorator (link) to enable contributors to handle this situation. As a contributor, when you notice a test failure that is unrelated to the changes you've made, you can add the @flaky decorator to that test. This will cause the test's result not to count as a failure during pre-merge checks.

The decorator requires as a parameter a UNIX timestamp specifying the time at which the decorator will stop skipping the test. A timestamp a few months in the future is a fine default to use.

@flaky is intended to be used liberally by contributors to unblock their work. Add it whenever you notice an apparently flaky test. It is, however, a short-term fix that you should not consider to be a permanent resolution.

Using @flaky comes with the responsibility of maintaining the test suite's coverage over the library. If you're in the habit of using it, periodically set aside some time to grep -R 'flaky' tests and remove some of the decorators. This may require finding and fixing the root cause of the unreliable behavior. Upholding this responsibility is an important way to keep the test suite's coverage meaningfully broad while skipping tests.

How do I enable debug logs for just a specific part of the library?

Enabling debug logs for the whole library with DD_TRACE_DEBUG=1 is often too noisy. Log levels for hierarchies of loggers can be controlled with internal environment variables. For example, to enable debug logs just for ddtrace.debugging, one can set `_DD_DEBUGGING_LOG_LEVEL=DEBUG`. This will set the DEBUG log level for any logger whose name is prefixed with ddtrace.debugging.