Evaluate depending on ty/ruff crates for Python semantic analysis

[joshuadavidthomas]

Summary

Decide whether and how to leverage ty (ruff's type checker) for Python module
resolution, import tracking, MRO, and cross-file symbol resolution — rather
than building these capabilities from scratch in djls.

Context

As djls adds features that require understanding Python code beyond
single-file AST extraction (model graph qualified keying, MRO for inheritance,
cross-file import resolution), we face a build-vs-depend decision. We already
use ruff_python_parser and ruff_python_ast as git dependencies. The ty
ecosystem provides substantially more:

• ty_module_resolver (~9K lines) — Full Python module resolution:
  sys.path search, .pth files for editable installs, namespace packages,
  typeshed, stub files
• ty_python_semantic (~116K lines) — Complete semantic analysis:
  per-file symbol tables, scope analysis, use-def chains, cross-file import
  resolution, MRO (C3 linearization), type inference, IDE support
  (SemanticModel with completions, go-to-definition, etc.)
• ty_site_packages (~2.7K lines) — Python environment discovery: venvs,
  conda, platform differences, pyvenv.cfg parsing
• ruff_db (~16K lines) — Foundation: File type, System trait
  (filesystem abstraction), VendoredFileSystem (bundled typeshed), diagnostics

Why This Matters Now

We already have a bug from rolling our own resolution: any project using a src/ layout gets incorrect module paths. 33 corpus snapshots across 4 repos are affected:

• InvenTree — src.backend.InvenTree.order.models (should be InvenTree.order.models, source root is src/backend/)
• Sentry — src.sentry.monitors.models (should be sentry.monitors.models, source root is src/)
• pretix — src.pretix.api.models (should be pretix.api.models, source root is src/)
• nomnom — src.nomnom.nominate.models (should be nomnom.nominate.models, source root is src/)

Our module_path_from_relative strips the project root and converts slashes to dots, but for src/ layouts the source root is a subdirectory, not the project root. This is exactly the kind of edge case ty_module_resolver handles via configurable src_roots. More will follow (editable installs, namespace packages, platform differences).

The immediate need is qualified keying for the model graph (#451 / task f9f8a7zj): resolving ForeignKey(User) to django.contrib.auth.models.User. This requires per-file import resolution at minimum.

Options

Option A: Side-car database

Run ty's ProjectDatabase alongside DjangoDatabase. Bridge at the query
level — when Django layer needs Python resolution, call into the ty database.

Pros:

• Lower risk, incremental adoption
• Can try for just qualified keying first
• Django-specific Salsa database stays untouched
• Clean separation: djls owns Django intelligence, ty owns Python intelligence

Cons:

• Two Salsa runtimes (different versions: djls uses salsa 0.25.2 from
  crates.io, ruff uses a git pin)
• Two file registries, two filesystem abstractions running in parallel
• Bridging overhead and complexity
• Need to keep ty's database populated/in-sync

Option B: Adopt ruff_db as foundation

Replace djls-source and djls-workspace's filesystem layer with ruff_db.
Implement ruff_db::Db on DjangoDatabase. Then ty_module_resolver::Db and
ty_python_semantic::Db layer on top naturally.

Pros:

• Single Salsa database, single runtime
• Full access to ty's semantic model as native queries
• Elegant long-term architecture
• One filesystem abstraction

Cons:

• Invasive refactor (replace djls-source, djls-workspace filesystem layer)
• Must adopt ruff's Salsa git pin (and track their updates)
• ruff_db is ~16K lines with a large System trait (~20 methods)
• Must bundle VendoredFileSystem (typeshed zip, ~5MB) even though djls
  doesn't need typeshed
• Locks into ruff's infrastructure decisions
• Build time impact (~140K+ lines of additional Rust)

Option C: Custom resolution inspired by ty (no dependency)

Write our own module resolution, informed by ty's algorithms and edge case
handling, but using our existing infrastructure (djls-source::File,
djls-workspace::FileSystem, our Salsa database). Use ty's code as a
reference implementation, not a dependency.

What we'd take from ty (knowledge, not code):

• Search path precedence — ordering logic from
  ty_module_resolver/resolve.rs (project root → extra paths → site-packages),
  including .pth file handling for editable installs
• Source root detection — understanding that src/ is often a source
  root, not a package. ty handles this via configured src_roots in
  SearchPathSettings. This is the fix for the src-layout bug:
  module_path_from_relative should strip the source root prefix, not just
  the project root
• pyvenv.cfg parsing — from ty_site_packages, robust venv metadata
  extraction (Python version, home path, system site-packages flag)
• Platform-aware site-packages — lib/pythonX.Y/site-packages vs
  Lib/site-packages, Debian dist-packages, free-threaded Python builds
• Module resolution rules — try foo.py then foo/__init__.py, namespace
  packages

What we'd skip (don't need for Django):

• Typeshed / stub file (.pyi) resolution
• VendoredFileSystem
• The full System trait (~20 methods)
• Type inference, lint rules, diagnostics infrastructure
• Generic/protocol/TypedDict handling in MRO
• KnownModule enum for stdlib special-casing

What we'd build (~1,000-1,500 lines estimated):

1. Source root discovery — detect src/ layouts, respect config, fall back to
   project root
2. Search path construction — project source roots → PYTHONPATH →
   site-packages (with .pth support)
3. Module resolution — dotted name → file path, handling __init__.py and
   namespace packages
4. Per-file import map — walk a file's import statements, resolve each name to
   a qualified path (can use ruff_python_ast import nodes directly since we
   already depend on it)
5. Robust site-packages discovery — pyvenv.cfg parsing, platform-aware paths

Pros:

• No new crate dependencies beyond what we already have
• Uses our existing Salsa database, File type, FileSystem trait
• No Salsa version conflict
• No build time impact
• Scoped to exactly what Django features need
• Can be built incrementally (fix src-layout bug first, add import map later)
• ty's code serves as a well-tested reference for edge cases

Cons:

• We write and maintain the code ourselves
• Risk of missing edge cases ty handles (mitigated by using their code as
  reference)
• If djls eventually needs full Python semantics, this becomes throwaway work
• Must actively check ty's code for new edge case handling as Python evolves

Key Technical Details

Salsa version mismatch: djls uses salsa = "0.25.2" (crates.io), ruff
uses salsa = { git = "...", rev = "..." }. Two different Salsa versions
means two runtimes — a single DjangoDatabase cannot implement trait
hierarchies from both. Options A requires bridging around this; Option B
requires adopting ruff's pin. Option C avoids it entirely.

ruff_db::System trait: ~20 methods including path_metadata,
canonicalize_path, read_to_string, read_directory, walk_directory,
glob, env_var, case_sensitivity, etc. ruff provides OsSystem and
TestSystem implementations. Required for Options A and B.

ruff_db::Db requires VendoredFileSystem: Even if djls doesn't use
typeshed, the trait requires it. Required for Options A and B.

ty_python_semantic::SemanticModel: The primary LSP interface — provides
resolve_module(), attribute_completions(), import_completions(),
scoped_completions(), members_in_scope_at(). Rich IDE infrastructure that
could directly power djls features. Available in Options A and B.

The src-layout bug: module_path_from_relative in
djls-project/resolve.rs strips the project root and converts to dots. For
projects with a src/ layout, the source root is e.g. <project>/src/backend/
(InvenTree), <project>/src/ (Sentry, pretix, nomnom) — not <project>/. 33
snapshots across 4 repos are affected. ty solves this with configurable
src_roots. Option C would fix this immediately; Options A/B would fix it as
part of integration.

Decision Criteria

• How much Python resolution complexity will djls realistically need in the
  next 6-12 months?
• Is the integration cost (Options A/B) worth it vs. building scoped
  resolution (Option C)?
• How stable are ty's APIs? (ty is pre-1.0, actively evolving)
• Build time impact on development workflow?
• Does depending on ty create a maintenance burden (tracking their git pin,
  adapting to API changes)?
• Could Astral eventually publish these crates to crates.io, making
  integration cleaner?
• Is Option C's ~1,500 line estimate realistic, or will scope creep push it
  toward 5K+?

Related

• Task f9f8a7zj / Python import resolution for model graph qualified keying #451: Python import resolution for model graph qualified
  keying (immediate need)
• Task m0bz37s9 / Introduce djls-engine crate and clarify crate boundaries #453: djls-engine crate and crate boundary cleanup
  (architectural context)
• Task jv8xldfv / Django model graph extraction #449: Django model graph extraction (current PR)