Manage proxy_db lifetimes with reference counting instead of a gc hook

[Korijn]

Motivation

observ's whole cleanup story hinged on a garbage-collector hook: proxy_db held a strong reference to every wrapped target, keyed on id(target), and a gc.callbacks hook combined with sys.getrefcount() inspection (in the hook and in Proxy.__del__) decided when entries could be dropped. Consequences:

• Entries for targets still referenced elsewhere were only released on a full gen-2 collection, which is rare — so memory could grow substantially in between.
• With gc.disable() (not uncommon in latency-sensitive apps), those entries leaked forever.
• Every full collection paid for a Python-land scan of the whole db.
• Correctness rested on fragile refcount arithmetic ("if 2: we are the last to hold a reference!").

Design

Invert the ownership so that plain reference counting does all cleanup, and the registry never needs to be scanned:

• The per-target entry is now a TargetDep — the Dep for the container itself, which also owns the target, the per-key deps, and (weakly) the proxies that wrap it. Owning the target is what keeps id(target) keying safe: an id can only be recycled after its TargetDep is gone and has removed itself from the registry.
• Every Proxy holds a strong reference to its TargetDep (__dep__).
• Per-key deps are KeyDep instances that hold their owning TargetDep. Watchers already hold strong references to the deps they depend on, so an entry stays alive for exactly as long as a proxy exists or a subscribed watcher needs its dep identity — the transient-proxy notification path keeps working (covered by a new test).
• proxy_db.db maps id(target) → weakref to the TargetDep; entries remove themselves via weakref callbacks (with an identity guard against stale callbacks).

No gc hook, no refcount inspection, no Proxy.__del__, and deliberately no reference cycles among TargetDep/KeyDep/proxies — cleanup is deterministic and immediate, and works with the garbage collector fully disabled. Keydeps become weakly registered: a keydep without subscribers has nobody to notify, so it can be recreated freely on the next tracked read (this also removes the vestigial keydep creation in the dict write trap).

Performance

Traps now reach their deps through a slot (self.__dep__) instead of a global-db id() dict lookup, entry creation no longer constructs WeakValueDictionarys (keydeps are materialized lazily, proxies are a plain dict of weakrefs), and the gen-2 gc callback overhead is gone entirely. Spot check (min of 5 timeit repeats, Python 3.11):

op	master	this PR
keyed/nested reads	6.5 µs	5.0 µs
keyed write (watched)	4.6 µs	4.2 µs
reactive() creation	4.1 µs	1.9 µs
deep-watched list write	144 µs	134 µs

Behavioral changes

• A registry entry is now released as soon as the last proxy dies and no watcher depends on the target — previously it was kept while the raw target was alive. Since dep identity only matters to subscribers, this is not observable, but white-box lifecycle tests were updated accordingly.
• proxy_db.attrs(proxy) is replaced by direct proxy.__dep__ / proxy.__dep__.keydeps access.
• The clear_proxy_db conftest fixture is no longer needed and was removed.
• Verified: 10k create/mutate/destroy churn iterations with gc.disable() leak zero registry entries, and gc.callbacks stays empty.

Full test suite passes (including qt group, run offscreen), plus ruff check/format.

🤖 Generated with Claude Code

https://claude.ai/code/session_01T4cFbKSCxjGabtJbB9e5H2

---

Generated by Claude Code

[berendkleinhaneveld]

Love it!

[Korijn]

Heads-up: benchmark gate regression on test_watcher_creation

The Benchmarks job fails with two regressions vs master:

• test_watcher_creation: +57% — the real problem
• test_read_list_iterate[reactive]: +5.5% — right at the threshold, looks like variance (lists have no keydeps)

Root cause

test_watcher_creation repeatedly creates and discards a watcher over the same reactive({"count": 0}). On master, the keydep for "count" was a plain Dep held strongly in the entry's keydep dict, so it was created once and reused every round.

In this PR keydeps live in a WeakValueDictionary and are only kept alive by a subscribed watcher. Between rounds the previous watcher is dropped, so its keydep dies and is recreated — a fresh KeyDep + weakref-with-callback every single round. Instrumented: 2000 create/destroy cycles → 2000 KeyDep allocations (should be ~1). Pinning the keydep so it's reused makes creation 1.77× faster, which accounts for the whole regression.

This weakness is deliberate — it's what lets an entry be reclaimed by pure reference counting (no cycles, works even with gc.disable()). Fixing the churn means keydeps have to persist and be reused, and that reintroduces a design fork:

Options

1. Strong keydeps + keep KeyDep.owner (simple). ~3 lines. Restores perf exactly. Introduces an ordinary TargetDep ↔ KeyDep cycle reclaimed by CPython's standard cyclic collector. Still removes the gc.callbacks hook entirely (the actual goal), but drops the "no cycles / gc-disable-safe" bonus — I'd update the PR text to match.

2. Strong plain keydeps + watcher-side owner retention. Keeps full reference-counting cleanup (no cycles, gc-disable-safe) and every current claim. More invasive: the Watcher separately retains each owning TargetDep, touching its dep bookkeeping, cleanup, and depend() propagation.

3. Leave keydeps weak, claw the 57% back elsewhere in the creation path. Investigating now — if Watcher.__init__ overhead can be trimmed enough to bring the net under the 5% gate, we keep the current lifetime model unchanged.

Leaning toward exploring (3) first since it preserves the cleanest semantics; will report back. Happy to take (1) if you'd rather keep it minimal.

---

Generated by Claude Code

[berendkleinhaneveld]

I propose to merge and let this test_watcher_creation regression be. Makes sense there is a bit of overhead in this specific case but I much prefer this architecture.

[Korijn]

I agree! 💯