Prepending user prompt with the llm.py BASE_PROMPT breaks caching.

[jftkcs]

The problem

The llm.py#58 BASE_PROMPT contains the dynamic date and time and is being prepended to the OpenAI Conversation prompt (conversation.py#193)which prevents any caching. This increases API costs and adds response latency.

Same issue as #133687.

In my case I forked the integration, removed the BASE_PROMPT, and manually put the date/time in the user provided prompt template at the end.

Realistically, I think the llm.py BASE_PROMPT is a foot gun for LLM integration developers and shouldn't exist as-is.

What version of Home Assistant Core has the issue?

core-2024.10.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

openai_conversation

Link to integration documentation on our website

https://www.home-assistant.io/integrations/openai_conversation

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @balloob, mind taking a look at this issue as it has been labeled with an integration (openai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of openai_conversation can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign openai_conversation Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

openai_conversation documentation
openai_conversation source
_{^{(message by IssueLinks)}}

[ZDBioHazard]

This issue doesn't just affect OpenAI, it affects Ollama too.

Every new conversation resets the cache due to the constantly changing timestamp at the beginning of the prompt which is unconditionally included by BASE_PROMPT. (defined in /homeassistant/helpers/llm.py and used in /homeassistant/components/ollama/conversation.py)

Simply setting BASE_PROMPT = "" allows my 600-token (non-assist) system prompt to remain cached between conversations, Saving minutes of time at the beginning of each new conversation as long as the Ollama model doesn't get unloaded. (Which it shouldn't with keepalive = -1)

Steps to reproduce:

1. Open an assist chat dialog.
2. Send a chat message, and observe the long delay (over 2 minutes in my case) while the prompt evaluates.
3. Close the assist dialog and open a new one, which causes the prompt template to be updated.
4. Send a chat message, and observe a similar long wait time as the previous session.

After setting BASE_PROMPT = "":

1. Open an assist chat dialog
2. Send a chat message, and observe the long delay (over 2 minutes in my case) while the prompt evaluates.
3. Close the assist dialog and open a new one, which causes the prompt template to be updated.
4. Send a chat message, and observe a fast response. (5-10 seconds in my case)

With assist enabled, the prompt can easily be thousands of tokens and take several minutes to evaluate on CPU, so optimizing the prompt layout is quite important to saving both energy and time.

Possible solutions:

• Simply make the timestamp template part of the default system prompt so users can change or remove it if they want.
• Insert the timestamp with the rest of the assist-related prompt text when control assist is enabled.

Workarounds:

For HAOS users, the issue can be temporarily patched out using the following shell commands on the local console. (not the SSH/Terminal addon shell) You will need to perform this workaround whenever the HAOS server is restarted or HA core is updated.
(Note: session.py should be changed to chat_log.py starting with HA 2025.2.1 - the file was renamed in #137209.)

ha > login
# docker exec homeassistant sed -i 's/llm\.BASE_PROMPT/""/' \
>    /usr/src/homeassistant/homeassistant/components/conversation/session.py \
>    /usr/src/homeassistant/homeassistant/components/ollama/conversation.py
# ha core restart
# exit

[jftkcs]

Commit 754de6f moved this issue into the conversation component but still makes the same error of inserting dynamic content at the head of the prompt.

[jftkcs]

To avoid modifying files inside the container I've put this as a fix in a hijacked version of openai_conversation to work with the latest release of home assistant (https://github.com/jftkcs/homeassistant-llm-hack). Just takes the current openai_conversation monkeypatches two things on llm.py. You can drop the function patch to just override the base_prompt.

Not ideal... but that's my current workaround since there's no easy way to remove the base prompt now that openai_conversation gets it via conversation then llm.py.

A slightly less bad way would be to make your own custom integration with just an init to do the patching. In my case I'm modifying openai_conversation anyway to do other things.