[2.0.x] Intern analysis values while deserializing - #1755
Merged
Conversation
sbt keeps one Analysis resident per subproject for the whole session, and structurally-equal values are shared neither across those analyses nor, for api tree nodes, within a single one. On a 582-analysis monorepo corpus, holding every analysis co-resident retains 1906 MB after GC; canonicalizing values as they are read brings that to 1237 MB (-35%). - global weak string pool, interned in BinaryDeserializer.string() - 8 weak-valued name -> UsedName pools, one per UseScope combination, probed by the already-canonical name so a pool hit allocates nothing - both pool types are hand-rolled on ConcurrentHashMap + ReferenceQueue in WeakPools.scala, so this adds no dependency - per-read HashMap dedups value-equality xsbti.api nodes (~95% of Type duplication is intra-analysis); it dies with the read, so it cannot leak - share the 8 possible scope EnumSets instead of copying one per UsedName - cache enum values(), which clones its array on every call, on per-node read paths - intern on the fresh-compilation path too (AnalysisCallback.usedName) Canonical values are held weakly, so they are released once no analysis references them, and draining the reference queue releases the pool entry with them. NameHash is deliberately not interned: it has zero within-analysis duplication and a 6.7M distinct population, so a weak pool costs more than it saves (+12 MB retained and 15.6% of read CPU when measured directly). The analysis format version is unchanged and api hashes are preserved, so existing analysis files stay readable and incremental invalidation is unaffected. Read cost is +3.7% on AnalysisFormatBenchmark and +17% loading the corpus, unchanged from 1 to 8 threads: the pools add per-call work, not contention. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a 2.0.x backport of #1754
sbt keeps one Analysis resident per subproject for the whole session, and structurally-equal values are shared neither across those analyses nor, for api tree nodes, within a single one. On a 582-analysis monorepo corpus, holding every analysis co-resident retains 1906 MB after GC; canonicalizing values as they are read brings that to 1237 MB (-35%).
Canonical values are held weakly, so they are released once no analysis references them, and draining the reference queue releases the pool entry with them. NameHash is deliberately not interned: it has zero within-analysis duplication and a 6.7M distinct population, so a weak pool costs more than it saves (+12 MB retained and 15.6% of read CPU when measured directly).
The analysis format version is unchanged and api hashes are preserved, so existing analysis files stay readable and incremental invalidation is unaffected. Read cost is +3.7% on AnalysisFormatBenchmark and +17% loading the corpus, unchanged from 1 to 8 threads: the pools add per-call work, not contention.