Sleeptime message grouping causes role confusion

[temujin9]

Describe the bug
Sleeptime trigger evaluates after each capture call, not after a complete request/response cycle is finished. There's no atomicity guarantee that all parts of a multi-part assistant response are processed together before sleeptime evaluation.

1. Frontend sends multi-part assistant response (e.g., thinking block + response text) via /messages/capture
2. Each part creates a separate Message in Letta
3. Sleeptime frequency counter increments per message batch
4. If counter hits trigger threshold between assistant message parts, sleeptime fires mid-response
5. Sleeptime agent receives incomplete context: thinking block without response text, or response without full conversation
6. Missing context causes sleeptime to misinterpret its role - attempts conversational response instead of memory processing.

Please describe your setup

• How are you running Letta?
  • Using Letta Cloud, but this bug is visible in OSS version as well.
• Describe your setup
  • Linux, accessing Letta Cloud api via custom rust and letta-python.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

• I'm using Opus with very fresh context window (freshly created agents) in sleeptime.
• I can semi-reliably reproduce this both on first turn sleeptime (another oddity, for another bug report) and further in.

Agent File (optional)
Please attach your .af file, as this helps with reproducing issues.

[temujin9]

I can implement a fix for this repo (which would presumably land in Letta Cloud). To do so, either a) letta stop honoring the number of turns for sleeptime so precisely, or b) we change what a turn means in a way that expands how much each one contains (1 user request + all responses).

[temujin9]

I don't have an agent file showing the issue, as I already deleted the agent, but I do have one showing a workaround / fix: grouping all agent responses before submitting. This was hackily done with a custom client, and would need a different approach natively.

test - Sleeptime.af.json

(PS - Github doesn't like .af extension, I had to add the .json to get it to upload. You might want to reconsider filename formats, if you want people to upload the results.)

[carenthomas]

Thanks for starting the discussion! The /messages/capture endpoint is experimental right now, and agree that the current sleeptime abstraction isn't the most ideal for it. I'm actively thinking about what might be a better way for sleeptime to ingest this data.

I'm surprised it performs so poorly, given that the previous messages should still be passed into the history, to gain a more complete understanding on follow-up turns. I'll tune the prompting here, but agree that it's better for sleeptime to get it right the first time rather than having to backtrack.

a) letta stop honoring the number of turns for sleeptime so precisely

I think this might be complicated to implement dynamically, because the nature of sleeptime is that it runs in the background rather than based on a specific trigger. The main way to implement it would be to rely on sleeptime to decline processing if it seems like the assistant is mid turn. Would achieve the dynamic aspect, but could be costly in terms of additional inference.

A naive solution might be to increase the default frequency for the capture endpoint (and add additional prompting) to maximize the change that a set of messages includes a full turn. Would love to see some examples if you have it available to see what low hanging fruit there might be in the immediate term.

b) we change what a turn means in a way that expands how much each one contains (1 user request + all responses).

In this case, are you dropping the remaining interleaved user messages and only keeping the first one? Or are you keeping all of the messages, but grouping all the user messages and assistant responses together respectively? Also curious what your use case is - coding, or something else?

[temujin9]

Unfortunately, that naive solution won't work reliably: I've seen reproduction on a cleave in the middle of much later context. Sadly, I didn't save that particular failure, it was instructive. Prompt tuning alone probably won't help: I tried several attempts at that first, though I may not be as good at it.

For (b), I was proposing keeping all messages, grouping user message with all subsequent assistant responses, and submitting only after the next user message comes in, to clearly delimit the end of the prior batch. (It's what I'm doing for my current backfill on the capture endpoint, and it seems to work without issue.) That would make five turns "five user messages and all agent responses to that", which seems logical enough to explain in documentation.