Auto-compact conversations

[barryvdh]

The Claude SDK now has auto-compacting; see https://platform.claude.com/docs/en/build-with-claude/context-editing#client-side-compaction-sdk

When the limit approaches the context limit, it create a summary. I'm currently doing this manually:

if (static::shouldCompact($messages)) {
    $messages[] = new UserMessage(config('ai.summary_prompt'));

    $summary = Prism::text()
        ->using(config('ai.provider'), config('ai.model'))
        ->withMessages($messages)
        ->usingTemperature(0)
        ->withMaxTokens(config('ai.summary_max_tokens'))
        ->asText();

    $summaryMessage = new AssistantMessage($summary->text);

    $aiMessage = AiConversationMessage::create([
        'ai_conversation_id' => $conversation->id,
        'role' => 'assistant',
        'content' => $summaryMessage->content,
        'is_compacted' => true,
        'prompt_tokens' => $summary->usage->promptTokens ?? null
    ]);

    // Reset messages
    $messages = [$summaryMessage];
}

$messages[] = new UserMessage($inputPrompt, $additionalContent);

That way, we can just start the conversation with the summary. Would it be useful to do auto-compacting in Prism as well?

Not sure if it's best done before or after the chat, but I don't think we have the input tokens per message already, so currently I'm just estimating them.

We could do something like this to enabled compaction

 return Prism::text()
    ->usingCompaction({enabled: true, summaryPrompt: $myPrompt, model: $model, contextTokenThreshold: 100000})

The resulting message would be just another message in the onComplete messages. Claude asks to wrap the summary in <summary> tags, so perhaps they just detect that and discard any previous messages. Although is would be cleaner to have a 'is_summary' or 'is_compaction' property on the message (or a new message type), so you can easily filter it.

We do have the promptTokens after usage, so you could just run compaction after the message when threshold is reached, but that would give a delay. When streaming, you could already process the result and then just compact.

[seomikewaltman]

Sounds good. One thing I liked about the Assistant API that is being deprecated was it did a lot of this overhead management for you. That way you could just write functional application code and not get stuck in the weeds of system design