Allow in-place Chat history editing

[ncoghlan]

lmstudio-js supports in-place Chat history editing via:

• offering both mutable and immutable chat history and chat message instances via an internal mutability flag (using the MaybeMutable base class)
• the .replaceText() method on saved messages (there are also .appendFile() and appendText() methods on messages)
• the .replaceSystemPrompt() method to remove all previously added system prompt messages and insert the given prompt at the start of the chat history
• the .filterInPlace() method to process all currently saved messages with the given filtering callback
• the .pop() method
• supporting direct iteration over and indexed access to the messages saved in the chat history

The Python SDK should offer similar capabilities (although the exact spelling may differ).

[ncoghlan]

I was going to mark this as "Good first issue", since the TypeScript SDK offers a starting point for the Python SDK API design, but there are some open design questions that need to be answered before it would really qualify for that marker.

There's no specific ETA on resolving those questions (folks that genuinely need this capability aren't blocked by the absence of a supported API, they just need to tinker with the ._messages field directly, which they understandably would prefer not to be doing).

[NullMagic2]

The problem with modifying ._messages is that you need to know how the object is structured internally, and there's no guarantee the implementation will remain constant. If you want to allow _messages to be modified, at the very least the internal structure would have to be well-documented in the API / tutorial.

[yagil]

@NullMagic2 what about creating a an empty Chat object, iterating over the old one and appending / modifying as needed?

[NullMagic2]

@NullMagic2 what about creating a an empty Chat object, iterating over the old one and appending / modifying as needed?

For my own solution, I'm creating my own chat object and then converting to the proper format using Chat.from_history() but that's much slower and less than ideal when compared to simple in-place editing.

The reason I do that as opposed to just editing ._messages is because Chat.from_history() is a public documented API, and if I have my own structure, at least I know it'll behave a particular way.

[ArEnSc]

I can probably take this one next

[ncoghlan]

In that case, I should write down those open design questions that I mentioned:

1. Having regular Chat instances be nominally immutable append-only (edit: they're not immutable) has some advantages, so I'm inclined to make a _ChatBase class that Chat and MutableChat both inherit from.
2. Having mutable and immutable versions of the messages themselves would be a pain, since the nominally immutable ones are automatically generated from the SDK schema. Adding any methods at all to the individual messages is inconvenient for the same reason. Accordingly, a MutableChat.replace_message(index, *, text=None, append_text=None, append_file=None) API that derives a replacement message from the original and overwrites it would be preferable to a directly supported mechanism for mutating messages in place
3. Chat should implement the Sequence ABC (provided read-only access to _messages), while MutableChat should implement the MutableSequence ABC
4. Due to internal cache coherency problems, MutableChat is likely incompatible with the not-yet-and-may-never-be-public API for implicitly converting files to their embeddings when the chat history is sent to the server (so the latter would presumably be part of Chat rather than _ChatBase, or else should be ditched entirely on the grounds of "If the implementation is hard to explain, it's a bad idea") Scratch that one, I already removed even the private APIs for implicit file caching back in 5aeb367 Belatedly filed a feature request to replace that with something more explicit: Feature: add a per-client file handle caching helper #151

The reason I consider these "open" questions is because I'm not sure those initial ideas will hold up to actual implementation.

[ArEnSc]

Shooting from the hip with this one.
I see we're using a traditional imperative style with the current chat history in history.py. How about we try an immutable fluent API? It could look something like this using a builder-like pattern.
This would eliminate the complexity of maintaining both mutable and immutable structures, removing any possible confusion between the two.

# Every method returns a new Chat
chat = Chat()
    .add_user_message("Hello")
    .add_assistant_message("Hi!")

Since each operation returns an immutable copy, it makes experimentation and branching straightforward:
pythonbase = Chat().add_system_message("Be helpful")
formal = base.add_user_message("Write formally")
casual = base.add_user_message("Write casually")

This approach addresses the open design questions:

No _ChatBase needed - We'd have a single Chat class that's always immutable, eliminating the inheritance complexity entirely.
Message replacement stays simple - The replace_message(index, *, text=None, append_text=None, append_file=None) API you described works perfectly in an immutable context—it just returns a new Chat instance with the derived replacement message.

No need to worry about mutating SDK-generated message objects.

Just implement Sequence ABC - Chat only needs to implement the read-only Sequence interface. No need for MutableSequence since there's no MutableChat.

What do you think? @ncoghlan and I am curious about what the typescript / js team thinks.

[ncoghlan]

We wouldn't be able to use the current Chat for that API for compatibility reasons. That includes compatibility with prediction callbacks, where on_message=chat.append wouldn't work properly if chat.append returned a new instance instead of mutating the method's bound instance in-place.

In saying that, I also realised that describing the current Chat as immutable isn't right - it's append-only rather than actually being immutable.

Adding that idea as a ChatBuilder API that intrinsically supported checkpointing could potentially be interesting, though.

Considering the mutation APIs from the initial post:

• Message level .replaceText -> ChatBuilder.replace_message(...)
• MutableChat.replaceSystemPrompt -> ChatBuilder.replace_system_prompt(...)
• MutableChat.filterInPlace -> ChatBuilder.filter(...) (wraps a new modified chat rather than filtering in-place)
• MutableChat.pop -> ChatBuilder.pop() (wraps a new modified chat with the last value removed)

And then the various existing methods that append to the chat would also become copying operations.

Such a type probably shouldn't be using a regular list as its data storage, though (given the amount of pointer copying it would end up doing). The interaction with the prediction callbacks would also still be awkward - I'm not seeing a way around either switching back to a regular append-in-place chat instance, or else sending in custom callbacks that rebind a ChatBuilder reference instead of updating a regular Chat instance in place.

[ncoghlan]

A simplification relative to my initial open questions: since the distinction is "append only" vs "arbitrary mutation" rather than "immutable" vs "mutable", I now think it would be fine to have MutableChat inherit from Chat, rather than being sibling classes of a common base.