Intern analysis values while deserializing

sbt keeps one Analysis resident per subproject for the whole session, and
structurally-equal values are shared neither across those analyses nor,
for api tree nodes, within a single one. On a 582-analysis monorepo
corpus, holding every analysis co-resident costs 2233 MB; canonicalizing
values as they are read brings that to 1460 MB (-35%).

- global weak string pool, interned in BinaryDeserializer.string()
- 8 weak-valued name -> UsedName pools, one per UseScope combination,
  probed by the already-canonical name so a pool hit allocates nothing
- per-read HashMap dedups value-equality xsbti.api nodes (~95% of Type
  duplication is intra-analysis); it dies with the read, so it cannot leak
- share the 8 possible scope EnumSets instead of copying one per UsedName
- cache enum values(), which clones its array on every call, on per-node
  read paths; read used-name sets with a direct loop
- intern on the fresh-compilation path too (AnalysisCallback.usedName)

Canonical values are held weakly, so they are released once no analysis
references them. NameHash is deliberately not interned: it has zero
within-analysis duplication and a 6.7M distinct population, so a weak pool
costs more than it saves (+12 MB retained and 15.6% of read CPU when
measured directly).

The analysis format version is unchanged and api hashes are preserved, so
existing analysis files stay readable and incremental invalidation is
unaffected. Read cost is +12% on AnalysisFormatBenchmark and +5% loading
the corpus with 8 threads.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: hoangmaihuy

build.sbt

@@ -349,7 +349,8 @@ lazy val zincCore = (projectMatrix in internalPath / "zinc-core")
       exclude[MissingClassProblem]("xsbti.*"),
     ),
     libraryDependencies ++= List(
-      "org.scala-lang.modules" %% "scala-parallel-collections" % "1.2.0"
+      "org.scala-lang.modules" %% "scala-parallel-collections" % "1.2.0",
+      "com.google.guava" % "guava" % "33.6.0-jre"
     ),
   )
   .jvmPlatform(scalaVersions = scala3_only)

internal/zinc-core/src/main/scala/sbt/internal/inc/AnalysisInterner.scala

@@ -0,0 +1,92 @@
+/*
+ * Zinc - The incremental compiler for Scala.
+ * Copyright Scala Center, Lightbend, and Mark Harrah
+ *
+ * Licensed under Apache License 2.0
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * See the NOTICE file distributed with this work for
+ * additional information regarding copyright ownership.
+ */
+
+package sbt.internal.inc
+
+import java.util.{ EnumSet => JEnumSet }
+import java.util.concurrent.ConcurrentMap
+
+import com.google.common.collect.{ Interner, Interners, MapMaker }
+import xsbti.UseScope
+
+/**
+ * The process-wide canonicalizer for strings and `UsedName`s: fresh
+ * (compiler-produced) and persisted (deserialized) analyses both route through
+ * it, so co-resident analyses share a single instance per distinct value.
+ *
+ * Strings are canonicalized through a Guava weak interner: canonical instances
+ * are held only through weak references, so once every analysis referencing a
+ * string is released the canonical instance becomes eligible for GC.
+ *
+ * `UsedName`s are canonicalized without constructing a candidate first: a
+ * `UsedName` is fully determined by its (already canonical) name string and one
+ * of the eight possible `UseScope` combinations, so each combination gets a
+ * weak-valued name-keyed map. A pool entry is removed once the canonical
+ * `UsedName` is no longer referenced by any analysis (the value also holds the
+ * only strong reference to its key while alive). On a pool hit no `UsedName`
+ * is constructed.
+ *
+ * Both pools make interning leak-free across the lifetime of a long-running
+ * build server. Benchmarks that need an uninterned baseline compare against a
+ * stock zinc checkout; production code always interns.
+ */
+object AnalysisInterner {
+
+  private val stringPool: Interner[String] = Interners.newWeakInterner()
+
+  // One weak-valued pool per UseScope combination, keyed by the canonical name.
+  private val usedNamePools: Array[ConcurrentMap[String, UsedName]] =
+    Array.fill(8)(new MapMaker().weakValues().makeMap[String, UsedName]())
+
+  def internString(s: String): String = stringPool.intern(s)
+
+  /**
+   * The canonical `UsedName` for `name` used in the scope combination
+   * `scopeBits` (bit 0: Default, bit 1: Implicit, bit 2: PatMatTarget).
+   * Constructs a `UsedName` only when the value is not pooled yet.
+   */
+  def usedName(name: String, scopeBits: Int): UsedName = {
+    // UsedName stores the escaped name, so the pool must be probed with it too;
+    // escaping is a no-op unless the name contains control characters.
+    val escaped = UsedName.escapeControlChars(name)
+    val pool = usedNamePools(scopeBits)
+    val existing = pool.get(escaped)
+    if (existing != null) existing
+    else {
+      val fresh = UsedName.make(escaped, scopeSets(scopeBits))
+      val prev = pool.putIfAbsent(fresh.name, fresh)
+      if (prev == null) fresh else prev
+    }
+  }
+
+  /**
+   * The eight possible scope sets, indexed by scope bits. Shared by every
+   * UsedName and treated as immutable -- they must never be mutated. The bit
+   * scheme must stay in sync with `ConsistentAnalysisFormat.writeUsedNameSet`
+   * (zinc-persist), which encodes the same bits on the write side.
+   */
+  private[inc] val scopeSets: Array[JEnumSet[UseScope]] =
+    Array.tabulate(8) { i =>
+      val e = JEnumSet.noneOf(classOf[UseScope])
+      if ((i & 1) != 0) e.add(UseScope.Default)
+      if ((i & 2) != 0) e.add(UseScope.Implicit)
+      if ((i & 4) != 0) e.add(UseScope.PatMatTarget)
+      e
+    }
+
+  private[inc] def scopeBits(scopes: JEnumSet[UseScope]): Int = {
+    var i = 0
+    if (scopes.contains(UseScope.Default)) i |= 1
+    if (scopes.contains(UseScope.Implicit)) i |= 2
+    if (scopes.contains(UseScope.PatMatTarget)) i |= 4
+    i
+  }
+}

internal/zinc-core/src/main/scala/sbt/internal/inc/Incremental.scala

@@ -920,9 +920,18 @@ private final class AnalysisCallback(
   }
 
   def usedName(className: String, name: String, useScopes: EnumSet[UseScope]) = {
+    // Canonicalize freshly-produced used names and their strings into the
+    // process-wide pools, so a just-compiled analysis shares them with every
+    // other resident analysis. (Api tree nodes are deduped only when an
+    // analysis is deserialized, not on this fresh-compile path.)
     usedNames
       .getOrElseUpdate(className, ConcurrentHashMap.newKeySet[UsedName].asScala)
-      .add(UsedName.make(name, useScopes))
+      .add(
+        AnalysisInterner.usedName(
+          AnalysisInterner.internString(name),
+          AnalysisInterner.scopeBits(useScopes)
+        )
+      )
     ()
   }
 

internal/zinc-core/src/main/scala/sbt/internal/inc/UsedName.scala

@@ -13,12 +13,23 @@ package sbt.internal.inc
 
 import java.{ util => ju }
 import scala.{ collection => sc }
+import scala.util.hashing.MurmurHash3
 import xsbti.compile.{ UsedName => XUsedName }
 import xsbti.UseScope
 
+/**
+ * `scopes` must never be mutated after construction: instances may share one
+ * scope set (`make` uses the set it is given, and interning aliases equal
+ * instances), and the hash below is computed once.
+ */
 case class UsedName private (name: String, scopes: ju.EnumSet[UseScope]) extends XUsedName {
   override def getName: String = name
   override def getScopes: ju.EnumSet[UseScope] = scopes
+
+  // A canonical (interned) instance is inserted into one name-set per class that
+  // uses it, and EnumSet.hashCode iterates its elements; caching makes each
+  // re-hash a field read. Fits in the object's existing alignment padding.
+  override val hashCode: Int = MurmurHash3.caseClassHash(this)
 }
 
 object UsedName {
@@ -33,7 +44,7 @@ object UsedName {
     new UsedName(escapedName, useScopes)
   }
 
-  private def escapeControlChars(name: String) = {
+  private[inc] def escapeControlChars(name: String): String = {
     if (name.indexOf('\n') > 0) // optimize for common case to regex overhead
       name.replace("\n", "\u26680A")
     else

internal/zinc-core/src/test/scala/sbt/internal/inc/AnalysisInternerSpec.scala

@@ -0,0 +1,101 @@
+/*
+ * Zinc - The incremental compiler for Scala.
+ * Copyright Scala Center, Lightbend, and Mark Harrah
+ *
+ * Licensed under Apache License 2.0
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * See the NOTICE file distributed with this work for
+ * additional information regarding copyright ownership.
+ */
+
+package sbt.internal.inc
+
+import java.lang.ref.WeakReference
+import java.util.concurrent.{ Callable, CountDownLatch, Executors, TimeUnit }
+import xsbti.UseScope
+
+class AnalysisInternerSpec extends UnitSpec {
+
+  behavior of "AnalysisInterner"
+
+  it should "canonicalize equal strings to one instance" in {
+    val s1 = new String("com.example.Foo") // force distinct heap instances
+    val s2 = new String("com.example.Foo")
+    assert(s1 `ne` s2)
+    assert(AnalysisInterner.internString(s1) `eq` AnalysisInterner.internString(s2))
+  }
+
+  it should "return one UsedName instance per (name, scope) without construction on hit" in {
+    val a = AnalysisInterner.usedName("map", 1)
+    val b = AnalysisInterner.usedName(new String("map"), 1)
+    assert(a `eq` b)
+    assert(a.name == "map")
+    assert(a.scopes.contains(UseScope.Default))
+    assert(!a.scopes.contains(UseScope.Implicit))
+  }
+
+  it should "distinguish scope combinations for the same name" in {
+    val default = AnalysisInterner.usedName("map", 1)
+    val implicitScope = AnalysisInterner.usedName("map", 2)
+    val patMat = AnalysisInterner.usedName("map", 4)
+    assert(default `ne` implicitScope)
+    assert(default `ne` patMat)
+    assert(implicitScope.scopes.contains(UseScope.Implicit))
+    assert(patMat.scopes.contains(UseScope.PatMatTarget))
+  }
+
+  it should "pool names containing control characters under their escaped form" in {
+    val a = AnalysisInterner.usedName("weird\nname", 1)
+    val b = AnalysisInterner.usedName("weird\nname", 1)
+    assert(a `eq` b)
+    assert(a.name == "weird♨0Aname") // stored escaped, exactly as UsedName.make does
+  }
+
+  it should "converge on one canonical instance under concurrent interning" in {
+    // Parallel compilation interns from many threads at once. Every thread must
+    // observe the same canonical reference, and the pools must never deadlock.
+    val threads = 16
+    val start = new CountDownLatch(1)
+    val pool = Executors.newFixedThreadPool(threads)
+    try {
+      val futures = (1 to threads).map { _ =>
+        pool.submit(new Callable[(String, UsedName)] {
+          def call(): (String, UsedName) = {
+            start.await() // release all threads together to maximize contention
+            (
+              AnalysisInterner.internString(new String("concurrent.value")),
+              AnalysisInterner.usedName(new String("concurrent.value"), 5)
+            )
+          }
+        })
+      }
+      start.countDown()
+      val results = futures.map(_.get(30, TimeUnit.SECONDS)).toList // deadlock => timeout
+      assert(results.forall(_._1 `eq` results.head._1))
+      assert(results.forall(_._2 `eq` results.head._2))
+    } finally pool.shutdownNow()
+  }
+
+  it should "release canonical instances once no analysis references them" in {
+    // The leak-free requirement: canonical instances are held only weakly, so
+    // once every analysis referencing a value is dropped it becomes GC-eligible.
+    val stringRef =
+      new WeakReference[String](AnalysisInterner.internString(new String("ephemeral.str")))
+    val usedNameRef =
+      new WeakReference[UsedName](AnalysisInterner.usedName("ephemeral.used", 1))
+    val released = (1 to 100).exists { _ =>
+      System.gc()
+      Thread.sleep(10)
+      stringRef.get() == null && usedNameRef.get() == null
+    }
+    assert(released, "weak pools must not retain values after they become unreachable")
+  }
+
+  it should "keep pooled values while an analysis still references them" in {
+    val held = AnalysisInterner.usedName("held", 1)
+    System.gc()
+    Thread.sleep(10)
+    assert(AnalysisInterner.usedName("held", 1) `eq` held)
+  }
+}