Document BFS dominator tree approximation bug and add regression test

[pyricau]

Summary

Replaces the dominator-tree-based retained size computation with a simpler, exact two-phase BFS approach. Retained size is now computed only for the first LEAKING node in each leak path.

Why replace the dominator tree?

`DominatorTree` builds dominators incrementally during BFS using a Lowest Common Ancestor approach. This is an approximation: when a cross-edge is processed, the parent's dominator may be stale and never revisited, leaving the child's dominator too specific. Retained sizes are then under-attributed.

New algorithm

Phase 1a — GC root BFS, leaked objects as leaves

BFS from GC roots where leaked objects are treated as leaf nodes: their paths are recorded but their children are never enqueued. The resulting visited set is R₀ — all objects reachable from GC roots without passing through any leaked object.

• Case A: all N leaked objects found before both queues empty → stop early, R₀ = visited set at that moment.
• Case B: queues exhausted before all N found → some leaked objects are only reachable through other leaked objects. R₀ is now complete; Phase 1b runs.

Phase 1b — secondary BFS (Case B only)

Seeded from already-found leaked objects, now traversed normally. Finds remaining leaked objects and records them as sub-leaked: reachable only through another leaked object. These are added as labels on the parent's leak trace rather than reported as independent leaks. Objects found in neither phase are truly unreachable.

Phase 2 — retained size BFS

Multi-source BFS seeded from the first LEAKING node in each path, restricted to objects not in R₀. Each object's shallow size is attributed to all seeds that can reach it (shared objects count toward all of them). One BFS per seed (N is typically small).

Other changes

• `DominatorTree` marked `@Deprecated` in favour of `LinkEvalDominatorTree`
• `deduplicateShortestPaths` and its trie removed — treating leaked objects as leaves makes it a no-op (a leaked object can never appear as an intermediate node in another path)
• `PathFindingResults` gains `visitedSet` (R₀) and `subLeakedObjectPaths`, loses `dominatorTree`
• `computeRetainedHeapSize` parameter kept on `Factory` for API compatibility but is now ignored

Test plan

• `./gradlew 🦈shark:test` passes
• `./gradlew 🦈shark-android:test` passes

🤖 Generated with Claude Code

[pyricau]

Move size*2 to a local var to avoid doing it 3 times

[tcmulcahy]

The PR description seems inaccurate (not sure if I'm misunderstanding something?)

Adds a convergence loop to DominatorTree: stores cross-edges during BFS, then re-processes them with updated dominator values after the BFS completes, iterating until the tree stabilizes (typically 2–3 passes)

I don't see this... It looks like the problem was just documented and then a new implementation (that avoids the problem) was introduced.

[tcmulcahy]

I see other changes in the PR that don't seem to be documented in the PR description. e.g. RealLeakTracerFactory and others - can you update the PR description and then ping me and I'll re-review?

[pyricau]

CRITICAL: think deeply about the contents of this comment before making changes.

Move computeRetainedSizesForFirstLeakingNodes as a new exposed method of PrioritizingShortestPathFinder. This means "visitedSet" can now be a private field of the results, which also means we can continue using the LongScatterSet internal data structure without making any copy. It also should mean we don't need to include objectReferenceReader in the results, since it's going to stay local to PrioritizingShortestPathFinder.

Once you've done that:

I think there's a bug today: R0 is everything visited from roots to leaked object ids, so it includes nodes that led to the leaking object id, so if we decide that objects up the leak trace are also leaking, we can't start a BFS from those as they're already in R0 so their outgoing refs will immediately be ignored.

Let's change the approach: we're going to compute retained size for leaked objects ids only. Also, change the retained size computation rules so that an object's size is assigned to the first leaking object id that runs into it (of course, still excluding R0 objects). This means we should be able to change the implementation here, where instead of doing one partial BFS for each leaking object id, we can instead use leaking object id as roots, keep the old visitedSet, and essentially do a single graph traversal from all leaking object id, instead of currently one traversal per object.

Once you've implemented that, make sure tests pass, fix what needs fixing.

At this point we have 3 arounds of traversal: the initial one, the second one to find missing leaking objects, the 3rd to compute retained size. You should be able to have step 2 happen as part of step 3. And instead of rewriting a secondary BFS, I think you should be able to move closer to the old code. So:

• treat leaking object ids as leaves that shouldn't be explored in the first pass, and keep track of those that were found
• reintroduce the "compute retained size" boolean. If we find all leaking object ids, we still have nodes to visite in the graph, but we don't care about retained size, just stop, we're done (if we care about retained size then you must continue because we do need a proper R0). If we're done with the graph but haven't found all leaking objects, or we need to compute retained size, then restart graph traversal from the found leaking object ids. I'm not sure this needs to be a secondary method, we can probably continue the initial loop, your call to figure that out. That secondary graph traversal should: 1) only enqueue one leaking object id first. Do the graph traversal from there, updating visitedSet as we go, and adding any unvisited object to the retained size. Also any newly visited leaking object id should be added to a list of sub leaking object ids for that parent object id. Once we're done with exploring the subgraph from that first leaking object id, enqueue the second & do this again, etc. Lastly, of course, if we don't care about retained size and only care about finding all remaining leaking object ids then we should stop if we do find them.

Make sure tests pass.

[pyricau]

Bring back this comment