epic: deprecate Python ORM core; Rust workspace as SSOT

[ds1sqe]

Summary

Promote the Rust workspace (type-bridge-core/) to single source of truth for all ORM mechanism. Re-implement the Python public API as a thin PyO3 facade over the Rust orm crate. Preserve Pydantic v2 semantics at the Python layer.

Invariant: do not break userspace

The user-facing import surface in type_bridge/__init__.py is frozen for the duration of this epic.

from type_bridge import Entity, Relation, String, Flag, ... continues to work, with identical Pydantic semantics, at every phase boundary. All change happens below the import line.

This is non-negotiable. Any sub-issue that cannot honor this invariant is closed without merging.

Why

The repo carries two parallel ORM implementations:

Layer	LOC	Role today
type_bridge/ (Python)	~24k	Default user surface (PyPI). Pydantic-integrated, metaclass-driven.
crates/core (Rust)	11.7k	SSOT for TypeQL AST, parser, compiler, schema, validation, coercion.
crates/orm (Rust)	8.7k	Full async ORM mirror — not exposed to Python.
crates/python (Rust)	1.7k	PyO3 bindings: core only.

Python ↔ Rust call sites today: 6 (one hard, five optional). Two implementations of the same concept drift; the cost compounds with every TypeDB version bump.

Target architecture

┌─────────────────────────────────────────────────────────────┐
│ Python user surface (UNCHANGED imports)                     │
│   class Person(Entity): ...    (Pydantic BaseModel)         │
└──────────────────┬──────────────────────────────────────────┘
                   ▼
┌─────────────────────────────────────────────────────────────┐
│ PyO3 facade (type_bridge package, ~thin)                    │
└──────────────────┬──────────────────────────────────────────┘
                   ▼
┌─────────────────────────────────────────────────────────────┐
│ crates/python (expanded): bindings for crates/orm           │
└──────────────────┬──────────────────────────────────────────┘
                   ▼
┌─────────────────────────────────────────────────────────────┐
│ crates/orm + crates/core + typedb-driver (Rust) — SSOT      │
└─────────────────────────────────────────────────────────────┘

Pydantic owns: validation, dumps, schema. Rust owns: persistence, queries, IO.

Pydantic seam — gate decision

Seam	Shape	Verdict
B-i	Pure Pydantic BaseModel; Rust-backed manager takes/returns Pydantic instances. Serde hop per CRUD.	Recommended.
B-ii	Person is a #[pyclass] implementing __get_pydantic_core_schema__.	Rejected. PyO3 #[pyclass] cannot host Pydantic's model_validator(mode='wrap'|'before') chain (_preserve_iid, _wrap_raw_values in type_bridge/attribute/base.py) without re-implementing __get_pydantic_core_schema__ in Rust to replicate validator ordering — fragile across Pydantic version bumps. Additionally, ModelMetaclass.__new__ cannot be applied to #[pyclass] types, making Pydantic's metaclass-driven field collection incompatible at class-define time.
B-iii	Pydantic class with Rust handle in __pydantic_private__.	Rejected. Hybrid ownership of one logical state is a drift trap.

B-i pays a serde hop per CRUD call (microbenchmark concern). B-ii pays "two Rust validators forever" tax. B-i wins.

Schema-binding model

Runtime schema descriptor, not static codegen:

1. Class-define time: Python metaclass collects entity name, attributes, flags, role definitions.
2. Registers descriptor with the Rust extension (PyEntityManager::register(descriptor)).
3. Rust ORM operates against (descriptor, row_dict).

No build-time Rust compilation per consumer project. Static codegen is out of scope.

Phase 1 design constraints

Three policy decisions must be settled in Phase 1 design, before any binding is shipped:

• Descriptor registration is a standalone public API, not buried inside PyEntityManager. This keeps a future opt-in path open for typed Rust models (e.g. crates/orm-derive::include_schema! consumers sharing the same registration primitive). Phase 1 is not complete until the registration primitive is callable from a context with no active manager instance.
• Canonical serde dict shape: unwrapped primitives via the entity's to_dict() path. Rust never receives wrapped Attribute instances; Python never re-wraps after a Rust-returned dict without going through model_validate.
• _iid write path Rust → Python: an explicit PyO3 setter method on the entity instance, not ad-hoc injection into __pydantic_private__. Rust must not assume direct access to Pydantic's private storage. The setter must be defined in crates/python, not synthesized by Python-side metaclass code.

Sub-issues

Seq	Phase	Title	Blocked By
#TBD	Gate	Pydantic seam decision recorded (B-i / B-ii / B-iii)	—
#TBD	0	Spike: Person CRUD via Rust manager, all Pydantic semantics green	Gate
#TBD	1-design	Phase 1 design constraints settled — descriptor registration primitive callable with no active manager; canonical serde dict shape locked; _iid setter contract in crates/python typed	Phase 0
#TBD	1	Expose crates/orm via PyO3 (PyDatabase, PyTransaction, PyEntityManager, PyRelationManager, PyFilter, PyHookRunner, PySchemaManager)	Phase 1-design
#TBD	2	Dual-backend feature flag (TYPE_BRIDGE_BACKEND=rust|python); full Python test suite passes on both (dev-time only; Phase 3 flips default for release)	Phase 1
#TBD	3	Drop Python typedb-driver dep; flip default backend to Rust	Phase 2
#TBD	4	Retire Python fallbacks in core paths; drop RUST_AVAILABLE flag; remove lark grammar	Phase 3
#TBD	5	Codegen split — TQL parser → Rust; Jinja2 renderer stays Python	Phase 4
#TBD	6	Port migration/ runtime to Rust	Phase 4

Phase 5 and Phase 6 can run in parallel after Phase 4.

Phase 0 kill criteria

If any of these fire during the spike, the epic halts:

• validate_assignment=True semantics cannot be preserved without monkey-patching pydantic-core.
• Per-CRUD serde overhead >50× the Python baseline at small N.
• Wheel build for spike branch fails on Linux or macOS.
• Metaclass drift causes downstream Entity subclasses to fail attribute resolution or flag inheritance.
• Entity.manager(db) FFI latency >500µs (breaks REPL-style workflows).
• Descriptor registration contract cannot be expressed as a stable typed interface in crates/python without unsafe blocks or per-version Pydantic ABI assumptions.

Dependency diagram

Seam decision
    │
    ▼
Phase 0 (spike) ─── kill-criteria gate
    │
    ▼
Phase 1 design constraints settled
    │
    ▼
Phase 1 (PyO3 surface for orm)
    │
    ▼
Phase 2 (dual-backend flag)
    │
    ▼
Phase 3 (drop python typedb-driver, flip default)
    │
    ▼
Phase 4 (retire fallbacks)
    │
    ├──────────────┐
    ▼              ▼
Phase 5         Phase 6
(codegen)       (migration/)

Decisions

• Release policy: no intermediate PyPI release. The epic completes before the next public version bump. Phase boundaries are internal milestones; Phase 2's dual-backend flag is dev-time only, never user-visible.
• Async-Python: v2. v1 is sync-only via block_on. v1 does not need to leave stubs reserving space for async.
• migration/: port to Rust. Phase 6 is a port, not a decision.

Verification gates

Before Phase 3 flips the default backend, the following test categories must exist and pass on the Rust backend:

• Downstream-shape compat test — a mock external subclass of Entity exercises metaclass-synthesized attributes, __init_subclass__ hooks, and flag inheritance. Catches metaclass drift without needing downstream repos on hand.
• Pydantic-semantic differential — for a fixed schema, model_dump_json() is byte-identical across Python-backend and Rust-backend over a corpus that includes validate_assignment flows, @model_validator(mode='before'/'wrap'), __pydantic_private__ mutation, Field(alias=...), Field(validation_alias=...), and model_json_schema() stability.
• Cross-runtime transaction consistency — within a single transaction, interleaved backend selection (via TYPE_BRIDGE_BACKEND swap) preserves isolation and consistency. Establishes that the dual-backend flag has no hidden transaction-scope bugs.
• Relation group-by integration — dedicated coverage for RelationGroupByQuery on relations with 3+ roles, multi-value roles, and hooks. Current depth is too thin to gate Phase 1 bindings on.
• Coverage parity gate — Rust-backend test depth ≥ Python-backend test depth before the default-flip ships. Concretely: ≥95% line coverage measured by cargo llvm-cov on the Rust side and coverage.py / pytest-cov on the Python side, against a shared test corpus that exercises every public ORM entrypoint.
• Property-based serde round-trip (deferred to Phase 1 hardening, not a gate) — Entity → to_dict() → model_validate() → to_dict() byte-identicality over randomized inputs. Tracked as an explicit acceptance criterion on the Phase 1 sub-issue; not a Phase 0 kill criterion. Named here so it is not discovered mid-Phase 2.

Non-goals

• Replacing Pydantic. Python user surface remains Pydantic v2.
• Replacing the TypeDB driver. Rust ORM continues to use the official typedb-driver Rust crate.
• Adding new ORM features. This is re-platforming, not feature expansion.
• Removing the codegen renderer. Jinja2 Python-output generation stays Python-hosted.
• Computed fields are out of scope for the Rust binding. Python-only @computed_field models are not a target for Rust-backed insertion.

Related

• Part of epic: single Rust core, thin multi-language bindings #110 — epic: single Rust core, thin multi-language bindings (this epic is the
  Python / PyO3 binding + SSOT consolidation pillar)
• Rust core integration epic: RFC: Rust Core Integration for type_bridge #95