Perf: Index collections by tag instead of re-sorting and re-deriving tags per lookup - #4332
Open
j9t wants to merge 1 commit into
Open
Perf: Index collections by tag instead of re-sorting and re-deriving tags per lookup#4332j9t wants to merge 1 commit into
j9t wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As mentioned in prior conversation with @zachleat, a major performance improvement, and an AI-assisted result of performance optimizations on one of my Eleventy projects. I took the liberty to use AI (Claude Code) for the following summary, too.
I vouch for doing due diligence and already, successfully using the same optimizations in my own projects (via patch-package)—a massive speed-up.
Problem
TemplateCollection.getFilteredByTag()is called once per collection (tag) per build, fromTemplateMap.getTaggedCollection(). Each call did two things that scale badly:getAllSorted()re-sorted the entire collection (Sortable.sort()has no cache), so a site withntemplates andttags paidt × O(n log n)comparisons.TemplateData.getIncludedTagNames(item.data)for every item, which rebuilds (and deduplicates) the tag array from scratch—t × narray allocations.On a large site this dominates the build. On a 22,000-template site with ~1,100 tags that is roughly
24 million sort comparisons and 23 million tag-array rebuilds per build.
Change
TemplateCollectionnow keeps two caches, both invalidated inadd()—the only entry point thatmutates
items, and the same signal the existing_filteredByGlobsCacherelies on via_dirty:Mapof tag name to items (built by walking the sorted list once), sogetFilteredByTag()is aMap.get()instead of a full scan.Because the index is built from the sorted list, each tag’s items are already in the same order the
filter produced, so results are unchanged.
getAllSorted(),getFilteredByTag()andgetFilteredByTags()all keep returning fresh arrays, so callers can still sort or reverse theresult in place (#352).
getFilteredByTags()(plural) now intersects the first tag’s index entry with the remaining tagsinstead of scanning the whole collection.
Results
Synthetic benchmark, one
getFilteredByTag()call per unique tag (as a build does), 6 tags per item:On a real 22,000-template site, this plus the 4.0
inputPathMapchange forgetMapEntryForInputPath()took a production build from ~120–150s to ~48s, with byte-identicaloutput.
Testing
test/TemplateCollectionTest.js: cache invalidation when templates are added after a lookup, mutability/independence of returned arrays, and the no-tag-name / unknown-tag /"all"cases.buildawesomeExcludeFromCollectionsastrue/string/array, lookups interleaved withadd()) produced identical results.