🪢 refactor: Export the Summarization Primitives - #483
Merged
Conversation
The summarize node keeps these to itself, so a caller that compacts a conversation outside a run has to reimplement them and then drift from them: LibreChat carries its own copy of the wrapper overhead constant, the checkpoint prompts, the parameter split, and the instruction builder. Move them to a module the barrel re-exports, leaving the node importing what it used to define. buildSummarizationInstruction now trims the prior summary so a blank one asks for a fresh checkpoint instead of a consolidation of nothing.
SUMMARY_WRAPPER_OVERHEAD_TOKENS claimed 33 tokens for a carrier that measures 48 on o200k_base and more on Anthropic, and its JSDoc described wrapper text that has never existed in the tree. Nothing could catch either, because the number lived in the summarize node while the string it was measuring lived in AgentContext. Replace it with buildSummaryCarrierText, which both sides now call. The node sizes a summary by measuring the message it will actually inject, and AgentContext builds that message from the same function, so the two can no longer disagree. A caller with no tokenizer, which is one of the conditions overflow recovery summarizes under, estimates from the same string rather than reserving nothing. Stop consulting the provider's output_tokens for the stored count. On a reasoning summarizer it includes hidden thinking that never reaches the checkpoint, so every later context calculation reserved room for tokens that are never sent. Provider usage stays a billing input. Narrow the barrel to the three names a caller compacting outside a run actually needs. The two default prompts stay internal: LibreChat's manual flow deliberately words its own, so exporting them would publish an API with no consumer. The langfuse routing fixture sizes a message by its character length and read the summary as a SystemMessage, which that counter flattens to 1, so a correctly measured carrier no longer fits its 120 token budget. Its ceiling now sits between the carrier and the transcript.
The no-counter branch estimated the carrier at four characters per token. That is a mean for English prose, not an upper bound, and this branch is the one that matters: shouldSummarizeOverflow fires precisely when there is no counter, so the count it produces is persisted and then reserved by instructionTokens on the overflow retry. Undercounting there is what makes the retry overflow again. Measured against o200k_base and Anthropic's tokenizer, four characters per token understates base64 by 1.5x and Korean by 4.6x, while coefficients large enough to cover those overestimate English prose by roughly 4x. No character heuristic is both safe and useful, so drop it and fall back to the tokenizer this package already bundles, picking the encoding from the summarizer's model. Provider output_tokens stays out of it. On a reasoning summarizer it includes hidden thinking that never reaches the checkpoint, so using it as a floor would persist a reservation many times the summary's real size.
The fallback picked its encoding from summarizationConfig.model, which is
the summarizer, not the model the carrier is re-injected into. That model
is undefined for ordinary self-summarization, so an Anthropic agent fell
through to encodingForModel('') and measured itself with o200k_base, and
a dedicated cheap summarizer could point the encoding at a different
provider entirely.
The count is spent by AgentContext.instructionTokens against the agent's
own context window, so it has to be denominated in the agent's tokenizer.
Anthropic counts run well above o200k_base on the same text: a Korean
summary measures 163 tokens against claude and 100 against o200k_base, so
the wrong encoding under-reserved the overflow retry by 39 percent.
Take the model from the agent's clientOptions, mirroring how Run.create
picks the encoding for the counter this stands in for, so a Claude model
reached through Bedrock or OpenRouter still resolves correctly. Fall back
to the provider when no model was recorded, treating ANTHROPIC as Claude
but not BEDROCK, which also serves Llama, Titan and Mistral.
Run.create derives one token counter from agents[0] and StandardGraph hands that same counter to every AgentContext, so a Claude agent behind a GPT first agent measured its summary carrier in o200k_base and persisted a count 1.6x below what the retry reserves. Counters built by createTokenCounter now carry their encoding, and the summary carrier takes the bundled tokenizer whenever that encoding disagrees with the receiving agent's. A host-supplied counter is unstamped and stays authoritative, since its units are the ones the host's own token map is denominated in.
LangChain accepts modelName as an alias for model and this repository configures agents through both, so reading only model reported an unconfigured model and fell through to the provider: a Claude-backed OpenRouter agent got o200k_base and undercounted its checkpoint carrier by 1.6x on CJK. Both the summary carrier's tokenizer and the run-wide counter Run.create derives now resolve either key. Two private copies of that resolution already existed in invoke and SubagentExecutor; all four sites now share resolveClientOptionsModel.
A host can register its own provider with family: 'anthropic', which serves Claude under whatever name and deployment alias the host chose. An exact match on the ANTHROPIC enum missed those, so a custom provider with no configured model measured its checkpoint carrier in o200k_base and under-reserved it by 1.6x on CJK. The provider fallback now reads the family, the way isThinkingEnabled already does, with the enum check still ahead of it because the registry is only populated once @/llm/providers is imported.
A model name without `claude` in it is the absence of a signal, not evidence of a non-Claude model, so an opaque deployment alias on a host-registered `family: 'anthropic'` provider skipped the family lookup and measured the carrier with `o200k_base`, undercounting a Korean checkpoint by 1.63x.
Resolves the conflicts between the summarization-primitives extraction and main's compaction semantic index / Bedrock prompt-cache work. - src/summarization/node.ts: kept both sides' imports (the moved primitives from ./shared alongside main's Bedrock cache helpers and renderCompactionSemanticIndex). buildSummarizationInstruction stays moved out of node.ts. - src/summarization/shared.ts: carried main's semanticIndexAppendix parameter onto the moved buildSummarizationInstruction, so the appendix still leads the instruction and the prior summary still trails it, on top of the blank-prior-summary trim. - Test files: unioned the two sides' imports and new describe blocks. - src/summarization/__tests__/shared.test.ts: added coverage for the appendix ordering now that the builder lives in shared.ts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Tbgs8cH8TASmdwt9LfQ3C
danny-avila
changed the title
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This supersedes #450, which sits on a fork branch I cannot push to and has gone conflicted against
main. It carries every commit from that PR plus a merge commit resolving the conflicts, so #450 can be closed once this lands. Credit for the original work goes to @berry-13.The underlying change is unchanged from #450:
createSummarizeNodeis a LangGraph node fused to a live run, so anything wanting to compact a conversation outside a run cannot call it, and the primitives it uses are private tonode.ts. LibreChat's manual/compactwork ended up reimplementing the summary carrier, both checkpoint prompts, themaxSummaryTokenssplit, and the instruction builder — four places where the two compaction paths can silently disagree while writing to the same summary boundary. Those move intosrc/summarization/shared.ts, which the module barrel re-exports.Conflicts came from
mainlanding the compaction semantic index (#476) and the Bedrock prompt-cache work on the same lines the extraction touched.origin/main(d9ddb58) into the PR head (7cb14f1) as a merge commit, keeping the original authorship intact.src/summarization/node.tsby keeping both sides: the primitives moved to./sharedalongsideaddBedrockTailCacheControl/resolveBedrockPromptCacheTtlandrenderCompactionSemanticIndex.main'ssemanticIndexAppendixparameter ontobuildSummarizationInstructionin its new home inshared.ts, so the rendered index still leads the instruction and the prior summary still trails it — on top of ♻️ refactor: Export the Summarization Primitives #450's blank-prior-summary trim.describeblocks inAgentContext.test.ts,specs/summarization.test.ts, andsummarization/__tests__/node.test.ts.src/summarization/__tests__/shared.test.tscovering the appendix ordering, since that behavior now lives inshared.tsand ♻️ refactor: Export the Summarization Primitives #450's test file predates it.Newly exported from the package root:
buildSummaryCarrierText,separateSummarizationParameters, andbuildSummarizationInstruction.Change Type
Testing
npx tsc -p tsconfig.json --noEmit: clean.npx jest src/summarization src/specs/summarization src/specs/summarize-prune src/specs/multi-agent-summarization src/agents/__tests__/AgentContext.test.ts: 11 suites, 305 passed, 11 skipped.llm/anthropic,llm/google, andllm/vertexaihave no API credentials ("Unable to detect a Project Id"), andspecs/durability-checkpoint.integrationcannot download the MongoMemoryServer binary. None touch summarization.npm run lint: no errors insrc/. The two reported errors are pre-existingtest/stubs/*.tstsconfig-include failures already present onmain.origin/mainin exactly the 15 files ♻️ refactor: Export the Summarization Primitives #450 touches, so the merge introduced no collateral changes.scripts/sort-imports.tsand Prettier is clean on every file touched during the resolution.Test Configuration:
Node 20,
npm ciagainst the lockfile on this branch. LLM provider specs and the Mongo durability spec need credentials and network egress respectively, neither available in this environment.Checklist