Read the heap dump's domination from the classes up

shark/shark-explorer/AGENTS.md

@@ -1,7 +1,8 @@
 # Shark Explorer — agent guide
 
 A desktop app that renders a heap dump's dominator tree as a navigable treemap, as rings around a
-centre, or as a stack of rows the way a profiler draws a call tree. The long term goal is a YourKit-style
+centre, or as a stack of rows the way a profiler draws a call tree — and that same domination read from
+the classes up, which is the stack of rows the other way round. The long term goal is a YourKit-style
 heap explorer; these are the first surfaces.
 
 This file is scoped to `shark/shark-explorer/`. It only records things an agent would get wrong by
@@ -31,6 +32,20 @@ under-attributed. Don't build on it.
 `notes/dominator-tree.md` for its memory profile and for the reference reader behaviour that makes a
 treemap read strangely until you know about it.
 
+## One heap dump, two trees, one shared root
+
+`HeapExplorer.tree` is that domination read from the roots down, and `tree.reverseTree` is the same
+domination read from the classes up, for the classes view — built on first use, because it costs a pass
+over every object of the dump. Both implement `HeapTree`, which is what a layout and a presentation take.
+
+**The whole heap dump is a node of both, with the same id**, so that a path zoomed into either tree
+starts at the same place. Every other node is told apart by its id alone
+(`ReverseDominatorTree.isReverseNode`), and **that one shared node is the trap**: what a clicked cell is
+cannot be worked out from its id, so `selectionOf` in `HeapDumpExplorer.kt` takes the shape being drawn.
+Asking one tree about the other's node is what a `require` there reports; asking the reverse tree for a
+node it doesn't have costs a full pass over the heap dump first. `notes/treemap-rendering.md` has the
+rest of it.
+
 ## The heap dump is read off the UI thread
 
 A `HeapGraph` is read only and safe to read from several threads at once, so the reason everything that

shark/shark-explorer/notes/dominator-tree.md

@@ -80,6 +80,40 @@ Both are why `ReferenceStrengthReader.foldedObjectIdsOf` exists: **whatever Shar
 tracked explicitly**, because the object is in no walk and in no tree, and anything counting objects or
 bytes over the whole dump would otherwise either miss it or count it twice. See the reachability section.
 
+## The same domination, read from the classes up
+
+`ReverseDominatorTree` reads the one tree the other way: every object on the row of its class, and above
+each row the classes of the objects dominating it. It needs no second dominator computation — only
+`immediateDominatorOf`, one object at a time, plus `nodes` for shallow sizes and to know which objects
+were folded. So it costs nothing until the classes view is opened, and the numbers say what opening it
+costs. Measured on `large-dump.hprof` (39 MB, 387,971 objects), which takes 2.4 s to open:
+
+| | |
+| --- | --- |
+| Gathering every object onto its class row — one pass over `graph.objects` | 0.20 s, 8,679 rows |
+| The level above the widest row — one `immediateDominatorOf` per object it gathers | 0.01 s, 44 rows |
+
+**The pass is over `graph.objects` and not over `nodes`**, because an object's class comes off the index
+it's read from, and looking it up per object again is what made the first version of this take minutes.
+The objects with no node are the folded ones, whose bytes are counted in the object holding them:
+311,601 of the dump's 387,971 objects have a node, and those are what the rows hold.
+
+**A row's weight is the shallow bytes of the objects at the bottom of its column.** Which is what makes
+the reverse root weigh exactly what the dominator tree's root does — 30,764,843 bytes on that dump,
+asserted in `ReverseDominatorTreeTest`. Note that this is *not* `HeapSizes.totalByteCount` (30,753,181
+there): that one is the strength legend's arithmetic over the reachability walks, and the 11 KB between
+them is not something to make a test assert.
+
+What it holds between reads is **at most one entry per object of the heap dump**, however many levels are
+open, because expanding a row hands its entries to the rows above it and drops its own — the live entries
+therefore stand for disjoint sets of the objects at the bottom of the columns.
+
+And what it is for, on that dump, in one column: `16,730 × byte[]` is 10.5 MB, a third of the heap; above
+it `151 × Bitmap` accounts for 8.1 MB of those bytes and `14,874 × Class` for 2.2 MB; above the bitmaps,
+two `LinkedHashMap$LinkedHashMapEntry`, one `LinkedHashMap` and one `LruCache`, each holding 3.9 MB of
+them. Six rows of one column, in the picture the view opens on, with no chain walked and nothing clicked —
+where the treemap has that same fact spread over 151 rectangles in different corners of itself.
+
 ## Why big objects sit flat under the root
 
 The first thing a production dump shows is a crowd of rectangles directly under the whole heap dump — on an

shark/shark-explorer/notes/treemap-rendering.md

@@ -2,21 +2,23 @@
 
 Implemented in `shark-explorer-core`: `Squarify.kt` (row layout), `TreemapLayout.kt` (adaptive depth
 and hit testing), `RadialLayout.kt` (the same, as rings), `StackLayout.kt` (the same, as a row per
-level), `TreemapRect.kt`, `LayoutCell.kt` (what the three layouts have in common),
-`HeapDominatorTreemap.kt` (the dominator tree as a `TreemapTree`, and `present()`, which labels and
-colours the cells a layout produced), `TreemapPresentation.kt` (a presentation per shape, each with an
-`of()` pairing a layout with that). Drawn by `TreemapView`, `RadialView` and `StackView` in
-`shark-explorer-app`.
+level, either way up), `TreemapRect.kt`, `LayoutCell.kt` (what the three layouts have in common),
+`HeapTree.kt` (what a layout can be built from: the two readings of the heap dump both implement it),
+`HeapDominatorTreemap.kt` (the dominator tree as a `HeapTree`, and `present()`, which labels and
+colours the cells a layout produced), `ReverseDominatorTree.kt` (the same domination read from the
+classes up), `TreemapPresentation.kt` (a presentation per shape, each with an `of()` pairing a layout
+with a tree). Drawn by `TreemapView`, `RadialView` and `StackView` in `shark-explorer-app`.
 
 **`of()` is a presentation's own, not a method per shape on `HeapDominatorTreemap`.** It used to be the
 other way round, and adding a third shape is what moved it: that class is a 1,200 line heap dump reader
 which detekt allows 50 functions, and `presentStack` was the fiftieth. Rather than split it somewhere
 arbitrary, the per-shape method came out of it — which is also the better line, since which shapes exist
 is no business of a heap dump reader. `present(cells)` is all that stayed behind: it reads a name and a
-strength off a `CellSubject`, and every shape's cells are those. **So a fourth shape needs nothing in
-`HeapDominatorTreemap` at all.**
+strength off a `CellSubject`, and every shape's cells are those. **Which is what let the fourth view be
+a second tree rather than a second reader**: `of()` takes a `HeapTree`, so a view of the classes is the
+stack layout over `ReverseDominatorTree`, and nothing in `HeapDominatorTreemap` knows about it.
 
-## Three shapes, one cut of the tree
+## Four views, three shapes, two readings
 
 A cell is a `LayoutCell`: a `CellSubject` — one node, or the children a node didn't draw — plus a
 depth, a weight and whatever geometry its layout adds. `TreemapCell` adds a rectangle, `RadialCell` an
@@ -27,6 +29,11 @@ being a second copy of all of it, and the third shape is what confirmed the pric
 and a `StackPresentation` with its `of()`. Nothing about colouring, labelling, selection, hit resolution
 or navigation moved.
 
+The fourth view is cheaper still, because it is a *reading* rather than a shape: the classes view is
+`StackLayout` with `rowsGoUp`, over the other tree. What it cost was one `HeapTree` implementation, one
+`Boolean` in the layout, `reverseScrolling` in the view, and the panels learning to describe a row — no
+new canvas, no new layout, no new hit testing.
+
 The three layouts make the same decisions — largest cell subdivided first, children too small to see
 grouped, a cell budget, truncation counted — differing only in what "too small" measures. A treemap
 compares areas; the radial view compares arc lengths along the middle of a ring, because a sector of
@@ -69,6 +76,53 @@ It skips one thing the treemap does: **a row doesn't draw its bitmap**. A row is
 and a bitmap fitted into 18 dp is a smear — the picture is the treemap's contribution, and asking for it
 here would cost a heap dump read and a decode per row for nothing.
 
+## The classes view: the same stack, read from the leaves
+
+`ReverseDominatorTree` is the heap dump's domination read the other way: every object on the row of its
+class, and above each row the classes of the objects dominating it. Drawn by `StackLayout(rowsGoUp =
+true)`, so the whole heap dump is the row across the bottom and a column grows up from it — an icicle
+chart the way a profiler draws a *reverse* call tree. What a column says is "these bytes are `byte[]`,
+held by `Bitmap`, held by `ImageView`", and how wide it is says how much of the heap's `byte[]` bytes
+that accounts for.
+
+Three things about it are not obvious from the other view:
+
+- **A row is weighed by the objects at the bottom of its column, in their own bytes.** Retained size is
+  the wrong weight here and not by a little: it would count an object's bytes again on every row above
+  it, so the rows would add up to several times the heap and "a fifth of the dump" would mean nothing.
+  Shallow bytes add up to the dump exactly once, which is why the reverse root weighs exactly what the
+  dominator tree's root weighs (asserted in `ReverseDominatorTreeTest`) and why a row's children cover
+  it to the byte.
+- **Every cell of it is a pile of objects, including the root.** Which is why a row is *not* drawn like
+  the treemap's class piles: washing out every cell and dashing every edge would spend the whole picture
+  saying something no cell of it contradicts, and lose the hues that make a column readable. The count
+  in the label (`1,204 × byte[]`) is what says "pile" instead. The two rows that stop a column keep
+  their own look: `Nothing in particular` is slate and dashed, uncollected garbage stays purple.
+- **It is built as it is read, and bounded to one entry per object of the heap dump.** The class rows
+  cost one pass over every object; a row above one costs one dominator read per object that row gathers.
+  Expanding a row hands its entries to the rows above it and drops its own, so however many levels are
+  open, the live entries stand for disjoint sets of the objects at the bottom of the columns.
+
+Two decisions in the wiring, both to keep it a view rather than a screen:
+
+- **The two trees share the root node id**, so a path zoomed into either of them starts at the same
+  place, and switching shape substitutes the root *in the view request* rather than in the history —
+  which is what leaves the path into the tree being left there to come back to. Every other node is told
+  apart by its id: `ReverseDominatorTree.isReverseNode` is a range check at the far end of the range
+  `HeapDominatorTreemap` takes its pile ids from.
+- **What a cell is, is read off the view it was clicked in, not off its node.** Because of that shared
+  root: the whole heap dump is a row of this view and the root of the other, and a pile of siblings that
+  didn't fit is named after the cell they were left out of, which on a dump with more classes than one
+  row can draw is that very node. So `selectionOf` takes the shape. The one thing it must *not* do is
+  re-read the selection when the shape changes, which would be asking one tree about the other's node —
+  hence the effect keyed on the request alone.
+
+Going up is done to the finished layout rather than threaded through it: `StackLayout` places every row
+from the top as usual and flips it once `rowCount` is known, because a row's distance from the *bottom*
+isn't known until the last row has been placed. `contentHeight` is clamped to the viewport when growing
+up, so the root row sits on the bottom edge of the view rather than of the rows, and `StackView` scrolls
+it with `reverseScrolling` so that `scrollTo(0)` still means "where it opens".
+
 ## Depth is area-driven, not a fixed level
 
 A heap dump's dominator tree has ~1 M nodes; a treemap can usefully show a few thousand rectangles.
@@ -196,7 +250,9 @@ The tree's nodes are object ids, and the piles it invents — the uncollected ga
 groups — need ids of their own. **`nodeId < 0` is not the test for one**, and taking it for one is a bug that
 looks like nothing: an object id is a heap address, a 32 bit dump records it in 4 bytes, and shark widens
 those by sign, so **every object above the 2 GB mark of such a dump has a negative id**. `isPileId` is a range
-check against `Int.MIN_VALUE` instead, and the pile ids start at `Long.MIN_VALUE`.
+check against `Int.MIN_VALUE` instead, and the pile ids start at `Long.MIN_VALUE`. The classes view's rows
+are ids of their own out of that same range, counting *down* from just below `Int.MIN_VALUE`, so
+`isReverseNode` is the same kind of range check and the two halves of the range can't collide.
 
 What the sign test cost, before it was a range check: on `large-dump.hprof`, **44 of the 4,616 rectangles of
 the opening view** had `contains()` say the tree didn't hold them, so pointing at one selected nothing, the

shark/shark-explorer/shark-explorer-app/src/main/java/shark/explorer/app/CellColors.kt

@@ -16,6 +16,7 @@ import shark.explorer.ReachabilityStrength.STRONG
 import shark.explorer.ReachabilityStrength.THREAD_LOCAL
 import shark.explorer.ReachabilityStrength.UNREACHABLE
 import shark.explorer.ReachabilityStrength.WEAK
+import shark.explorer.ReverseNodeKind
 
 /**
  * How the rectangles are coloured. Pick one above the view.
@@ -78,11 +79,17 @@ internal class CellColors private constructor(
   val label: Color get() = LABEL
 
   /** Dark enough on a pile of objects for the dashes of [outlineOf] to read as dashes. */
-  fun borderOf(presented: PresentedCell<*>): Color = when (presented.content) {
-    is CellContent.Object -> if (coloring.scheme == DAISY_SCHEME) DAISY_BORDER else BORDER
+  fun borderOf(presented: PresentedCell<*>): Color = when (val content = presented.content) {
+    is CellContent.Object -> objectBorder
+    // A row of the classes view is drawn like an object, borders included — bar the one row that reads as
+    // a pile there too, whose edge is dashed. See [colorOf].
+    is CellContent.ObjectRow ->
+      if (content.kind == ReverseNodeKind.NO_OWNER) PILE_BORDER else objectBorder
     else -> PILE_BORDER
   }
 
+  private val objectBorder: Color get() = if (coloring.scheme == DAISY_SCHEME) DAISY_BORDER else BORDER
+
   fun colorOf(presented: PresentedCell<*>): Color {
     val depth = presented.cell.depth
     val strength = presented.strength
@@ -98,6 +105,15 @@ internal class CellColors private constructor(
       } else {
         objectColor(strength, depth, hueIndexOf(presented))
       }
+      // A row of the classes view is a pile as well, and coloured like an object all the same: every cell
+      // there is a pile, so the slate would be the whole picture and a column would be one flat block. Its
+      // hue is what makes a column read as one thing, the way nesting does in a treemap.
+      is CellContent.ObjectRow -> if (content.kind == ReverseNodeKind.NO_OWNER) {
+        // Except the row that is objects with nothing in common, which is the one that reads as a pile.
+        pileColor(strength, depth)
+      } else {
+        objectColor(strength, depth, hueIndexOf(presented))
+      }
       is CellContent.Object -> objectColor(strength, depth, hueIndexOf(presented))
     }
   }

shark/shark-explorer/shark-explorer-app/src/main/java/shark/explorer/app/CellView.kt

@@ -29,11 +29,14 @@ import androidx.compose.ui.unit.sp
 import kotlin.math.roundToInt
 import shark.explorer.CellContent
 import shark.explorer.CellSubject
+import shark.explorer.HeapDominatorTreemap
 import shark.explorer.LayoutCell
 import shark.explorer.ObjectGroupKind
+import shark.explorer.ReverseDominatorTree
+import shark.explorer.ReverseNodeKind
 import shark.explorer.TreemapPoint
 
-/** Which shape the dominator tree is drawn as. Pick one above the view. */
+/** Which shape a heap dump's domination is drawn as. Pick one above the view. */
 internal enum class ViewShape(val displayName: String) {
 
   /** Nested rectangles: area is retained size, nesting is domination. */
@@ -52,7 +55,27 @@ internal enum class ViewShape(val displayName: String) {
    * doesn't spend area on nesting, so the deep end of a chain is drawn and named at full size — and
    * therefore the one shape taller than the window, which is why it scrolls.
    */
-  STACK("Stack")
+  STACK("Stack"),
+
+  /**
+   * The same stack of rows the other way up, of the other of the heap dump's two trees: every object of
+   * the dump on the row of its class along the bottom, and what dominates them stacked above, class by
+   * class. See [shark.explorer.ReverseDominatorTree].
+   *
+   * So a row here is a pile of objects rather than one object, and reading up a column answers "what
+   * holds all the `byte[]`, and what holds that" — which [STACK] can only answer one object at a time.
+   */
+  CLASSES("Classes");
+
+  /**
+   * Whether this shape draws [nodeId] at all, which is what a shape being switched leaves behind.
+   *
+   * The heap dump's two trees share their root and no other node — see [shark.explorer.HeapTree] — so a
+   * path zoomed into one of them is nothing to the other, and which shape is drawn is what says which of
+   * the two a node is expected to be in.
+   */
+  fun draws(nodeId: Long): Boolean = nodeId == HeapDominatorTreemap.ROOT_OBJECT_ID ||
+    ReverseDominatorTree.isReverseNode(nodeId) == (this == CLASSES)
 }
 
 /**
@@ -108,18 +131,26 @@ internal fun BoxScope.NotExpandedBadge(nodeCount: Int) {
 internal fun Offset.toTreemapPoint() = TreemapPoint(x.toDouble(), y.toDouble())
 
 /**
- * How a cell is outlined: dashed for every instance of one class, dotted for the siblings that didn't
- * fit, solid for an object and for the two halves of the heap dump.
+ * How a cell is outlined: dashed for every instance of one class and for the objects nothing in
+ * particular holds, dotted for the siblings that didn't fit, solid for an object, for the two halves of
+ * the heap dump and for a row of the classes view.
  *
- * A pile of objects shouldn't have the same edge as one object, in either shape. Along with the washed
+ * A pile of objects drawn among objects shouldn't have the same edge as one object. Along with the washed
  * out fill and the label, it's the third thing saying this cell isn't something you can inspect the
- * fields of.
+ * fields of. Every cell of the classes view is a pile, so there a dashed edge would mark nothing out and
+ * make the rows hard to tell apart: see [CellContent.ObjectRow].
  */
 internal fun outlineOf(content: CellContent): Stroke = when {
   content is CellContent.ObjectGroup && content.kind == ObjectGroupKind.CLASS -> Stroke(
     width = PILE_BORDER_WIDTH,
     pathEffect = PathEffect.dashPathEffect(CLASS_GROUP_DASH_INTERVALS)
   )
+  // The one row of that view that isn't objects gathered by something: they have nothing in common but
+  // that nothing in particular holds them, so the edge says so the way a pile's does.
+  content is CellContent.ObjectRow && content.kind == ReverseNodeKind.NO_OWNER -> Stroke(
+    width = PILE_BORDER_WIDTH,
+    pathEffect = PathEffect.dashPathEffect(CLASS_GROUP_DASH_INTERVALS)
+  )
   content is CellContent.Leftover -> Stroke(
     width = PILE_BORDER_WIDTH,
     pathEffect = PathEffect.dashPathEffect(LEFTOVER_DOT_INTERVALS)