Skip to content

Commit 4b70d1c

Browse files
ericholscherclaudestsewd
authored
Optimize CI: caching, uv, parallel jobs, docs-only skip (#12886)
## Summary Overhaul the CircleCI pipeline to reduce wall time and save resources. ### Pipeline architecture ``` Stage 1 (parallel): checks ──────────┐ tests ───────────┼──▶ Stage 2 (parallel): tests-search tests-proxito ───┘ tests-embedapi │ Stage 3: coverage-report ◀┘ ``` - **Stage 1** runs the fast, lightweight jobs in parallel: linting/migration checks, core tests (no ES), and proxito tests - **Stage 2** runs the heavier jobs only after Stage 1 passes: search tests (needs Elasticsearch sidecar) and embed API tests (6 Sphinx versions) — avoiding wasted resources on broken builds - **Stage 3** combines coverage from all test jobs (`coverage combine`) and uploads a single unified report to codecov ### Test marker coverage Every test runs in exactly one job — no gaps, no duplicates: | Marker(s) | Job | Django settings | |---|---|---| | _(unmarked)_ | `tests` | `readthedocs.settings.test` | | `search` only | `tests-search` (1st cmd) | `readthedocs.settings.test` | | `search` + `proxito` | `tests-search` (2nd cmd) | `readthedocs.settings.proxito.test` | | `proxito` only | `tests-proxito` | `readthedocs.settings.proxito.test` | | `embed_api` | `tests-embedapi` | `readthedocs.settings.test` | Notable dual-marked tests handled by `tests-search` with proxito settings: - `search/tests/test_proxied_api.py` (search + proxito) - `search/tests/test_utils.py` (search + proxito) - `search/api/v3/tests/test_api.py:ProxiedSearchAPITest` (inherits search, marked proxito) ### Changes - **Split test suite into parallel jobs**: the single `tests` job previously ran core, proxito, and search tests serially (~18 min). Now they're 3 independent jobs - `tests` — core tests only (no Elasticsearch sidecar needed) - `tests-proxito` — proxito tests with separate Django settings - `tests-search` — search tests with Elasticsearch (Stage 2) - **Combined coverage reporting**: each test job writes to a unique `COVERAGE_FILE`, persists it via CircleCI workspace, and a final `coverage-report` job combines them and uploads a single report to codecov - **Dependency caching**: pip and tox environments cached across runs, keyed on all `requirements/*.txt` files - **`tox-uv` for faster installs**: `tox-uv` plugin uses `uv` for virtualenv creation and dependency resolution inside tox (tox 4+ jobs) - **Docs-only skip**: `skip-if-docs-only` command halts test jobs when only `docs/` files changed (PR branches only, compares against `origin/main`) - **DRY shared commands**: `setup-checkout` and `skip-if-docs-only` reusable commands eliminate duplication ### Expected impact - **Wall time**: ~26 min → ~15 min (parallel jobs, caching) - **Docs-only PRs**: skip all test jobs entirely (~30s per job vs ~10+ min) - **Resource savings**: Elasticsearch only starts for search tests, not all tests ## Test plan - [x] All CircleCI jobs pass (checks, tests, tests-proxito, tests-search, tests-embedapi) - [x] Docs-only PR (#12896) confirms test jobs halt early with success - [ ] Verify coverage-report job combines and uploads correctly - [ ] Verify caching works on subsequent pushes (cache hit in restore_cache step) https://claude.ai/code/session_01PrHk6hpFk9eDTaNujKYLon --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Santos Gallegos <stsewd@proton.me>
1 parent 4d95ad1 commit 4b70d1c

File tree

6 files changed

+273
-100
lines changed

6 files changed

+273
-100
lines changed

‎.circleci/config.yml‎

Lines changed: 173 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,56 @@
1+
# CI Pipeline
2+
#
3+
# Stage 1 (parallel): checks, tests, tests-proxito
4+
# Stage 2 (parallel): tests-search (needs ES), tests-embedapi (6 Sphinx versions)
5+
# Stage 3: coverage-report (combines and uploads to codecov)
6+
#
7+
# Stage 2 runs after tests and tests-proxito pass (but not checks,
8+
# so a formatting error won't block test execution).
9+
#
10+
# Test marker coverage — every test runs in exactly one job:
11+
# tests: -m "not search and not proxito and not embed_api"
12+
# tests-search: -m "search and not proxito" + -m "search and proxito" (proxito settings)
13+
# tests-proxito: -m "proxito and not search"
14+
# tests-embedapi: -m embed_api (multiple Sphinx versions via tox.embedapi.ini)
15+
#
16+
# Caching strategy:
17+
# Each test job caches ~/.local/lib (tox packages) and .tox (virtualenvs
18+
# with all deps). Both save_cache and restore_cache use an exact checksum
19+
# key — on a cache miss (requirements changed), a full install runs and
20+
# a new cache is saved. CircleCI keys are immutable (never overwritten).
21+
# Bump "v1" in key names to bust all caches.
22+
# See: https://circleci.com/docs/guides/optimize/caching/
23+
#
24+
# Other design decisions:
25+
# - tests-search runs two pytest commands because some tests are marked
26+
# both "search" and "proxito" (e.g. test_proxied_api.py). These need
27+
# the ES sidecar AND proxito Django settings, so they run in
28+
# tests-search with --ds=readthedocs.settings.proxito.test.
29+
# - tests-embedapi uses tox<4 (tox.embedapi.ini isn't compatible with
30+
# tox 4), so it can't use tox-uv.
31+
# - Coverage files are copied to unique names (cp .coverage .coverage.tests)
32+
# after each tox run because tox doesn't pass COVERAGE_FILE through to
33+
# the test env. The coverage-report job combines them.
34+
# - skip-if-docs-only compares against origin/main, so it only works for
35+
# PRs targeting main. PRs targeting other branches will always run tests.
36+
137
version: 2.1
238

339
orbs:
440
codecov: codecov/codecov@3.2.2
541
node: circleci/node@5.1.0
642

743
commands:
44+
setup-checkout:
45+
description: Checkout code, initialize submodules, and prepare cache keys
46+
steps:
47+
- checkout
48+
- run: git submodule sync
49+
- run: git submodule update --init
50+
- run:
51+
name: Combine requirements files for cache key
52+
command: cat requirements/*.txt > requirements-cache-key.txt
53+
854
skip-if-docs-only:
955
description: Halt the job if only docs/ files changed
1056
steps:
@@ -19,10 +65,10 @@ commands:
1965
# Compare against the merge base with main to find changed files.
2066
changed_files=$(git diff --name-only $(git merge-base HEAD origin/main)..HEAD)
2167
if [ -z "$changed_files" ]; then
22-
echo "No changed files detected — skipping tests."
23-
circleci-agent step halt
68+
echo "No changed files detected — running tests."
2469
exit 0
2570
fi
71+
# grep -v returns exit code 1 when all lines match, so || true
2672
non_docs=$(echo "$changed_files" | grep -v '^docs/' || true)
2773
if [ -z "$non_docs" ]; then
2874
echo "Only docs changed — skipping tests."
@@ -31,61 +77,153 @@ commands:
3177
3278
jobs:
3379
tests:
80+
# Core tests — excludes search, proxito, and embed_api markers.
81+
# No Elasticsearch sidecar needed.
3482
docker:
3583
- image: 'cimg/python:3.12'
3684
environment:
37-
# Don't skip search tests.
38-
TOX_POSARGS: ''
39-
PYTEST_COVERAGE: --cov-report=xml --cov-config .coveragerc --cov=. --cov-append
85+
PYTEST_COVERAGE: --cov-config .coveragerc --cov
86+
steps:
87+
- setup-checkout
88+
- skip-if-docs-only
89+
- restore_cache:
90+
keys:
91+
- pip-tests-v2-{{ checksum "requirements-cache-key.txt" }}
92+
- run: sudo apt update
93+
- run: sudo apt install -y rclone
94+
# tox-uv makes tox use uv for virtualenv creation and dep resolution.
95+
- run: pip install --user tox tox-uv
96+
- run: tox -e py312
97+
- run: cp .coverage .coverage.tests
98+
- save_cache:
99+
key: pip-tests-v2-{{ checksum "requirements-cache-key.txt" }}
100+
paths:
101+
- ~/.local/lib
102+
- ~/.local/bin
103+
- .tox
104+
- persist_to_workspace:
105+
root: .
106+
paths:
107+
- .coverage.tests
108+
109+
tests-search:
110+
# Search tests — needs Elasticsearch sidecar.
111+
# Runs in Stage 2 to avoid starting ES on broken builds.
112+
docker:
113+
- image: 'cimg/python:3.12'
114+
environment:
115+
PYTEST_COVERAGE: --cov-config .coveragerc --cov
40116
- image: 'docker.elastic.co/elasticsearch/elasticsearch:9.3.1'
41117
name: search
42118
environment:
43119
discovery.type: single-node
44120
ES_JAVA_OPTS: -Xms750m -Xmx750m
45121
ELASTIC_PASSWORD: password
46-
# Disabled SSL for testing.
47122
xpack.security.transport.ssl.enabled: 'false'
48123
steps:
49-
- checkout
50-
- run: git submodule sync
51-
- run: git submodule update --init
124+
- setup-checkout
52125
- skip-if-docs-only
126+
- restore_cache:
127+
keys:
128+
- pip-search-v2-{{ checksum "requirements-cache-key.txt" }}
53129
- run: sudo apt update
54130
- run: sudo apt install -y rclone
55-
- run: pip install --user tox
56-
- run: tox -e py312
57-
- codecov/upload
131+
- run: pip install --user tox tox-uv
132+
- run: tox -e search
133+
- run: cp .coverage .coverage.search
134+
- save_cache:
135+
key: pip-search-v2-{{ checksum "requirements-cache-key.txt" }}
136+
paths:
137+
- ~/.local/lib
138+
- ~/.local/bin
139+
- .tox
140+
- persist_to_workspace:
141+
root: .
142+
paths:
143+
- .coverage.search
144+
145+
tests-proxito:
146+
# Proxito tests — uses readthedocs.settings.proxito.test Django settings.
147+
# No Elasticsearch sidecar needed.
148+
docker:
149+
- image: 'cimg/python:3.12'
150+
environment:
151+
PYTEST_COVERAGE: --cov-config .coveragerc --cov
152+
steps:
153+
- setup-checkout
154+
- skip-if-docs-only
155+
- restore_cache:
156+
keys:
157+
- pip-proxito-v2-{{ checksum "requirements-cache-key.txt" }}
158+
- run: pip install --user tox tox-uv
159+
- run: tox -e proxito
160+
- run: cp .coverage .coverage.proxito
161+
- save_cache:
162+
key: pip-proxito-v2-{{ checksum "requirements-cache-key.txt" }}
163+
paths:
164+
- ~/.local/lib
165+
- ~/.local/bin
166+
- .tox
167+
- persist_to_workspace:
168+
root: .
169+
paths:
170+
- .coverage.proxito
58171

59172
tests-embedapi:
173+
# Embed API tests — tests against 6 Sphinx versions.
174+
# Requires tox<4 (tox.embedapi.ini is not compatible with tox 4).
60175
docker:
61176
- image: 'cimg/python:3.12'
62177
steps:
63-
- checkout
64-
- run: git submodule sync
65-
- run: git submodule update --init
178+
- setup-checkout
66179
- skip-if-docs-only
180+
- restore_cache:
181+
keys:
182+
- pip-embedapi-v2-{{ checksum "requirements-cache-key.txt" }}
67183
- run: pip install --user 'tox<4'
68184
- run: tox --version
69185
- run: tox -c tox.embedapi.ini
186+
- save_cache:
187+
key: pip-embedapi-v2-{{ checksum "requirements-cache-key.txt" }}
188+
paths:
189+
- ~/.local/lib
190+
- ~/.local/bin
191+
- .tox
70192

71-
checks:
193+
coverage-report:
194+
# Combines coverage from tests, tests-search, and tests-proxito,
195+
# then uploads a single unified report to codecov.
72196
docker:
73197
- image: 'cimg/python:3.12'
74198
steps:
75199
- checkout
76-
- run: git submodule sync
77-
- run: git submodule update --init
200+
- attach_workspace:
201+
at: .
202+
- run:
203+
name: Combine coverage and generate report
204+
command: |
205+
pip install --user coverage
206+
coverage combine .coverage.*
207+
coverage xml
208+
coverage report --show-missing
209+
- codecov/upload
210+
211+
checks:
212+
docker:
213+
- image: 'cimg/python:3.12'
214+
steps:
215+
- setup-checkout
78216
# Recipe: https://pre-commit.com/#managing-ci-caches
79217
- run:
80-
name: Combine pre-commit config, python version and testing requirements for caching
218+
name: Combine pre-commit config and requirements for caching
81219
command: |
82220
cp common/pre-commit-config.yaml pre-commit-cache-key.txt
83221
python --version --version >> pre-commit-cache-key.txt
84-
cat requirements/testing.txt >> pre-commit-cache-key.txt
222+
cat requirements-cache-key.txt >> pre-commit-cache-key.txt
85223
- restore_cache:
86224
keys:
87225
- pre-commit-cache-{{ checksum "pre-commit-cache-key.txt" }}
88-
- run: pip install --user tox
226+
- run: pip install --user tox tox-uv
89227
- run: tox -e pre-commit
90228
- run: tox -e migrations
91229
- save_cache:
@@ -97,9 +235,22 @@ workflows:
97235
version: 2
98236
test:
99237
jobs:
238+
# Stage 1: fast, lightweight jobs (parallel)
100239
- checks
101240
- tests
241+
- tests-proxito
242+
# Stage 2: heavier jobs, only after test jobs pass (parallel)
243+
- tests-search:
244+
requires:
245+
- tests
246+
- tests-proxito
102247
- tests-embedapi:
103248
requires:
104-
- checks
105249
- tests
250+
- tests-proxito
251+
# Stage 3: combine coverage and upload to codecov
252+
- coverage-report:
253+
requires:
254+
- tests
255+
- tests-proxito
256+
- tests-search

‎.coveragerc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[run]
22
branch = true
3-
source = .
3+
source = readthedocs
44
omit =
55
*/**/migrations/*
66
*/**/management/commands/*

0 commit comments

Comments
 (0)