Skip to content

chore(deps): upgrade to Python 3.13 #39566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 8 commits into
base: master
Choose a base branch
from
Draft

chore(deps): upgrade to Python 3.13 #39566

wants to merge 8 commits into from
+843 −758

Conversation

webjunkie
Copy link
Contributor

@webjunkie webjunkie commented Oct 13, 2025 •
edited
Loading

Summary

Upgrade from Python 3.12.11 to Python 3.13.7, including deprecation fixes and dependency updates.

Analysis of 83 pinned dependencies found 42 released before Python 3.13.0 (Oct 7, 2024). These have been prioritized into 4 upgrade batches based on criticality and Python 3.13 compatibility.

Progress

Core Python Changes

  • Update pyproject.toml to Python 3.13.7
  • Rebase onto numpy 2.1 upgrade branch
  • Fix AST deprecations (ast.Num → ast.Constant)
  • Fix datetime UTC deprecations (datetime.utcnow/utcfromtimestamp)
  • Upgrade pyodbc: 5.1.0 → 5.2.0
  • Fix hogql-parser packaging for Python 3.13
  • Fix asyncio.get_event_loop() deprecations (deferred - caused massive ruff changes)
  • Review and fix any SSL/TLS changes
  • Update CI/CD configuration

Batch 1: Critical Blockers (Must Upgrade Now)

Criteria: Confirmed/likely build failures on 3.13, missing 3.13 wheels/classifiers, or known ecosystem pivots.

Package Current Target Why Critical Notes
pillow 10.2.0 ≥11.0.0 ⚠️ BLOCKS CI KeyError: '__version__' on 3.13. Pillow 11 is first line with official 3.13 support.
pandas ~=2.2.2 ≥2.2.3 If pinned <2.2.3 2.2.3 is first generally compatible release for Python 3.13 (wheels for normal + free-threaded). 2.3.x OK if want features.
lxml 5.2.1 ≥5.3.1 High 3.13 wheels landed around 5.3 line. Older pins try to build from source and often fail.
celery 5.3.6 ≥5.5.x High Celery 5.5 officially supports Python 3.13. Update together with Kombu & py-amqp.
kombu 5.3.7 ≥5.5.x High (w/ Celery) Kombu 5.5 adds Python 3.13 support (Celery aligned). Keep Celery/Kombu in lockstep.
clickhouse-driver 0.2.9 Evaluate High Upstream does NOT advertise 3.13. Open issue for 3.13 support. If hit wheels/ABI pains, consider clickhouse-connect (official HTTP client).
nh3 0.2.14 ≥0.3.1 Medium-High Rust extension. Current releases ship cp313/cp313t wheels. Upgrade to avoid local builds.
numpy ~=2.1.0 ≥2.1.0 OK Usually OK NumPy 2.1.0 already supports 3.13. 2.2 adds free-threaded builds. If using pandas 2.3, align to NumPy 2.2+.
psutil 6.0.0 ≥6.0.0 OK Medium psutil supports 3.13. Note early 3.13 free-threaded (nogil) had issues—stick to regular 3.13 for now. Prefer ≥7.x.
beautifulsoup4 4.12.3 ≥4.13+ Safe Pure Python; widely built/tested on 3.13 across distros. Upgrade optional but harmless.
conditional-cache 1.2 ≥1.4 Safe Actively maintained; pure Python; 3.13 is fine. Upgrade to pick up fixes.
  • Batch 1 complete

Risk: Medium (ClickHouse component is the only "spicy" one)
Testing: 1-2 days total CI + smoke (API I/O, data export paths, Celery flows, XML/HTML parsing, image ops)
Notes: Keep Celery + Kombu aligned; confirm normal 3.13 (not free-threaded) in prod until stack is ready.

Batch 2: High Priority (Native Deps / Security / Infra)

Criteria: Native code or transitive Rust/C; should be modern before/with the Python jump.

Package Current Target Breaking Changes to Watch
cryptography 39.0.2 ≥43.x/46.x Modern Cryptography uses Rust; ensure manylinux/musllinux wheels in CI base image; occasional backend changes.
paramiko 3.4.0 ≥3.5-4.0 May need newer cryptography; some extras changed (paramiko[invoke] deprecated). Re-run SSH key auth flows.
brotli 1.1.0 ≥1.1.0 OK 3.13 wheels published via google/brotli-wheels; keep at 1.1.x+.
antlr4-python3-runtime 4.13.1 ≥4.13.2 No functional breaks typical; align with grammar toolchain.
dnspython 2.2.1 ≥2.6-2.7 Pure Python; watch resolver timeouts behavior if jump multiple minors.
openpyxl 3.1.2 ≥3.1.5 Pure Python; verify chart/formatting if rely on edge cases.
  • Batch 2 complete

Risk: Low-Medium (crypto/SSH are sensitive)
Testing: 1 day - SSH (Paramiko), TLS handshakes (requests/boto), Excel export/import regressions

Batch 3: Test First (Pure Python, Likely Fine)

Criteria: Pure Python libs that typically "just work" on new CPython.

Package Current Target Notes
boto3 1.28.16 ≥1.40.x Keep aioboto3 in sync.
aioboto3 12.0.0 ≥15.x Tracks aiobotocore; update pairwise w/ boto3.
django-redis 5.2.0 ≥5.4-5.5 Added explicit Python 3.13 support; ensure redis-py ≥5.x.
django-prometheus 2.2.0 Latest 2.x Metrics only; low risk.
django-axes 5.9.0 Latest 6.x Check middleware order & signals.
django-picklefield 3.0.1 ≥3.3.0 Supports Django 5.1/5.2; drops old Py versions.
django-structlog 2.1.3 ≥9.1.1 Track LTS combos of Python/Django/Celery.
dj-database-url 0.5.0 ≥3.0.1 Big age gap; simple API; safe to bump.
django-statsd 2.5.2 ≥2.7.0 Minor.
django-loginas 0.3.11 Latest Admin-only; test staff login.
djangorestframework-csv ~3.0.2 Latest 3.x Serialization only; low risk.
markdown-it-py ~3.0.0 Latest 3.x Pure Python; plugin compat.
natsort 8.4.0 Latest 8.x Pure Python; safe.
jsonref 1.1.0 Stay or bump 1.1.0 is already 3.13-rebuilt in distros; fine.
phonenumberslite 8.13.6 Latest 8.13.x Regular monthly data updates; pure Python.
geoip2 4.6.0 ≥5.x Pure Python; keep DB reader & models in sync.
drf-exceptions-hog 0.4.0 Latest Project-local; pure Python; run API error paths.
more-itertools 9.0.0 Latest 10/11.x Pure Python utilities.
celery-redbeat 2.1.1 Sync w/ Celery 5.5 Keep aligned with Celery.
  • Batch 3 complete

Risk: Low
Testing: 1 day - app smoke tests; focus on integrations (AWS, Redis, DRF CSV, admin impersonation)

Batch 4: Defer (Ancient but Stable / Niche)

Criteria: Old but low-surface or internal-only; upgrade if tests reveal issues.

Package Current Recommendation Notes
gunicorn 20.1.0 Bump to 22.x/23.x Mostly runtime flags & worker classes; validate CLI flags. When convenient.
lzstring 1.0.4 Keep Pure Python; tiny surface.
nanoid 2.0.0 Keep Pure Python ID gen; validate only if rely on crypto randomness.
django-deprecate-fields 0.1.1 Keep Migration helper; pure Python.
clickhouse-pool 0.5.3 Re-evaluate Tightly coupled to clickhouse-driver; if move to clickhouse-connect, may replace pooling logic.
mimesis 5.2.1 Keep or bump Test data generation; pure Python.
  • Batch 4 complete

Risk: Very low
Testing: Ad hoc only when choose to bump

Build & Deploy

  • Verify uv sync completes successfully
  • Build succeeds
  • Docker image builds
  • Deployment smoke test

Dependency Relationships & Coupling

  • Celery stack: Upgrade Celery 5.5.x + Kombu 5.5.x together (and ensure compatible py-amqp). This is where Python 3.13 support lands.
  • AWS SDK: Keep boto3 in sync with aioboto3/aiobotocore (update in one PR, rerun S3/SQS/SNS paths).
  • ClickHouse: If clickhouse-driver becomes a blocker on 3.13 build images, migrate to clickhouse-connect (official, HTTP). Adjust pooling (urllib3 pools built-in).
  • Django ecosystem: Ensure Django core version supports 3.13 (Django 5.1+). Most django-* libs above have kept pace.
  • Crypto: Paramiko depends on cryptography/bcrypt/pynacl versions; update together.

Breaking Changes to Watch (High Level)

  • Pillow 11.x: Drops older Python; some deprecated APIs removed—validate image save/resize/format plugins.
  • Celery 5.5/Kombu 5.5: Transport defaults & quotas have changes; check ETA/queues, especially if use RabbitMQ quorum queues.
  • cryptography: Newer versions rely on Rust; ensure CI runners have compatible manylinux/musllinux bases (or just use prebuilt wheels).
  • lxml 5.3+: Wheel availability fixes most issues; if try free-threaded 3.13t, expect more variability.
  • Paramiko 4.x: Tighten on dependencies/extras; re-test key exchanges and host-key policies.

Recommended Execution Order

  1. Batch 1 (Critical) — Unblock 3.13: Pillow → pandas (≥2.2.3) → lxml → Celery/Kombu (together) → ClickHouse choice → nh3 → validate NumPy/psutil
  2. Batch 2 (High Priority) — Infra/security/native: cryptography → paramiko → dnspython/openpyxl
  3. Batch 3 (Test First) — Sweep through pure-Python libs with CI
  4. Batch 4 (Defer) — Bump opportunistically

Blockers & Concerns

  • ClickHouse: clickhouse-driver does NOT list Python 3.13 in classifiers yet; if hit build/runtime issues, switch to clickhouse-connect to avoid blockers.
  • Free-threaded 3.13 (nogil, cp313t): Some extensions still have rough edges (e.g., psutil on early betas). Use standard 3.13 in production unless explicitly need free-threading.
  • Django core: Make sure on Django 5.1+ before cutting over to 3.13; 4.2 won't work.

Estimated Testing Effort

  • Batch 1: 1-2 days total CI + smoke (task queue, ClickHouse paths, ingest/export, HTML/XML sanitize, image ops)
  • Batch 2: ~1 day (crypto/TLS + SSH flows + Excel I/O)
  • Batch 3: ~1 day broad regression
  • Batch 4: Opportunistic

Key Python 3.13 Changes

Reference: https://docs.python.org/3/whatsnew/3.13.html

  • PEP 667: datetime.utcnow/utcfromtimestamp removed ✅ Fixed
  • asyncio.get_event_loop() behavior changes (deferred)
  • ast.Num/ast.Str/ast.Bytes deprecated ✅ Fixed
  • SSL/TLS minimum version changes
  • C API changes affecting extensions

Final Sanity Checks Before Merge

  • Confirm Django 5.1+ in base requirements
  • Stick to standard CPython 3.13 (not free-threaded) unless ready to validate every C/Rust extension
  • For Linux wheels, use manylinux/musllinux-compatible base images to pull prebuilt wheels (cryptography, lxml, psutil, nh3)
  • For ClickHouse consider switching to clickhouse-connect if clickhouse-driver becomes brittle on runners

@webjunkie
chore(deps): upgrade numpy to 2.1.x for Python 3.13 support
1a3c992 
Update numpy from 1.26.4 to ~=2.1.0 (resolved to 2.1.3)
Update pandas from 2.2.0 to ~=2.2.2 (resolved to 2.2.3) - required for numpy 2.x compatibility
Update scikit-learn to use ~=1.5.0 specifier for consistency

Numpy 2.0+ is required for Python 3.13 support. Verified all breaking changes:
- No deprecated functions used in codebase (checked np.tostring, np.alltrue, etc.)
- Ruff NPY201 checks passed
- All experiment statistics tests (41 tests) passed
- Basic numpy operations verified working

Pandas 2.2.2+ required for numpy 2.x support per upstream compatibility matrix.
@webjunkie
fix: revert ruff in lockfile
98717c4
@webjunkie
chore(python): upgrade to Python 3.13.7
4a3ce1b 
Upgrade Python from 3.12.11 to 3.13.7 across all environments.

Includes uv upgrade from 0.7.8 to 0.8.19 to enable Python 3.13.7 downloads. As documented in uv Python versions guide: "The available Python versions are frozen for each uv release. To install new Python versions, you may need upgrade uv."

Python 3.13.7 chosen over 3.13.8 because uv 0.8.19 (latest in flox) was released Sept 19, 2025, before python-build-standalone added 3.13.8 on Oct 7, 2025.

Changes:
- Update pyproject.toml requires-python to 3.13.7
- Update Dockerfiles to python:3.13.7-slim-bookworm
- Update all GitHub workflows to Python 3.13.7 and uv 0.8.19
- Update flox manifest to uv 0.8.19
- Update mypy cache key to 3.13
- Update test snapshots for Python 3.13 paths

Breaking changes will be addressed in follow-up PRs as needed.
@webjunkie webjunkie changed the base branch from master to chore/upgrade-numpy-2.1 October 13, 2025 14:03
@posthog-bot
Copy link
Contributor

It looks like the code of hogql-parser has changed since last push, but its version stayed the same at 1.2.0. 👀
Make sure to resolve this in hogql_parser/setup.py before merging!

@webjunkie
fix: replace deprecated ast.Num with ast.Constant
70ab1c9 
ast.Num is deprecated as of Python 3.8 and will be removed in Python 3.14.
Use ast.Constant instead, which is the unified node type for all constants.
@webjunkie
fix: replace deprecated datetime.utcfromtimestamp
bee9611 
Replace datetime.utcfromtimestamp() with datetime.fromtimestamp(, datetime.UTC).
The utc* methods are deprecated in Python 3.12+ and will be removed in a future version.

Partial fix - more datetime deprecations remain in other files.
@webjunkie
fix: replace all deprecated datetime UTC methods
82eb69a 
Replace deprecated UTC datetime methods with Python 3.13-compatible equivalents:
- datetime.utcnow() → datetime.now(datetime.UTC)
- datetime.utcfromtimestamp(ts) → datetime.fromtimestamp(ts, datetime.UTC)

These methods are deprecated in Python 3.12+ and will be removed in a future version.

Affected files:
- common/hogvm/python/stl/__init__.py (5 occurrences)
- ee/api/sentry_stats.py
- ee/tasks/auto_rollback_feature_flag.py
- ee/tasks/subscriptions/__init__.py
- posthog/temporal/subscriptions/subscription_scheduling_workflow.py
- posthog/session_recordings/queries/test/session_replay_sql.py
- posthog/session_recordings/queries/test/listing_recordings/test_session_recording_list_from_query.py
- posthog/session_recordings/queries/sub_queries/person_ids_subquery.py
- posthog/management/commands/analyze_behavioral_cohorts.py
@webjunkie
chore: upgrade pyodbc to 5.2.0 for Python 3.13 support
0cbc12e 
pyodbc 5.1.0 fails to build on Python 3.13 due to a setup.py issue.
Upgrade to 5.2.0 which has Python 3.13 support.
@webjunkie
fix(hogql-parser): update setup.py for Python 3.13 compatibility
df1971e 
Remove packages declaration for hogql_parser-stubs and rely on MANIFEST.in
for including stub files. Add Python 3.13 classifier.

Note: Building still requires --no-build-isolation-package flag with uv
until all dependency build issues are resolved.
@webjunkie webjunkie force-pushed the chore/upgrade-python-313 branch from 0c88272 to df1971e Compare October 13, 2025 14:26
@posthog-bot
Copy link
Contributor

It looks like the code of hogql-parser has changed since last push, but its version stayed the same at 1.2.0. 👀
Make sure to resolve this in hogql_parser/setup.py before merging!

Base automatically changed from chore/upgrade-numpy-2.1 to master October 13, 2025 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@webjunkie @posthog-bot