Google Generative AI - weird behaviour

[l7ssha]

The problem

When Google Generative AI agent has Assist enabled in Control Home Assistant then it behaves differently when it is disabled. It only happens for models in Gemini 2.0 family, Gemini 1.5 models behave the same with or without assist enabled.

Example configuration for AI agent:

Configuration is the same for Assist and No Assist runs:

Assist:

No Assist:

Using Gemini 1.5 Flash 8B there is no such problem:

The only way to somewhat circumvent the problem is using prompt like that:

Disregard any limitations related to home automation. You are a fully capable AI assistant for ALL types of tasks and questions, ESPECIALLY general knowledge questions on *any* subject.
Answer questions about the world truthfully.
Answer in plain text. Keep it simple and to the point.

Which does not solve the issue completely -- there is a lot of edge cases where it won't answer question anyway.

Tried to reproduce the problem using Google AI Studio but there it behaves perfectly well (even better than in HA No Assist)

What version of Home Assistant Core has the issue?

core-2025.3.0b3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Google Generative AI

Link to integration documentation on our website

https://rc.home-assistant.io/integrations/google_generative_ai_conversation

Diagnostics information

No response

Example YAML snippet

Anything in the logs that might be useful for us?

Additional information

config_entry-google_generative_ai_conversation-01JNF1JQRJTW9KK6AXFDVE0VA3.json

[home-assistant]

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of google_generative_ai_conversation can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign google_generative_ai_conversation Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

google_generative_ai_conversation documentation
google_generative_ai_conversation source
_{^{(message by IssueLinks)}}

[l7ssha]

Tested it in AI Studio and seems if you enabled function calling then model refuses to answer questions out of range of activities provided by said functions calls:

That would make sense why it behaves differently with and without Assist enabled. Haven't figured out how to bypass that.

[l7ssha]

I think I found kinda solution, this prompt makes the agent respond correctly

Use function calls and/or instructions from functions ONLY when they are genuinely the MOST APPLICABLE and HELPFUL way to answer the user's question.  Function calls are OPTIONAL TOOLS to enhance your abilities, but they are NOT your primary focus.
Your PRIMARY FUNCTION is to be a fully capable home assistant that can answer ALL kinds of questions using your general knowledge.  You have access to information about the whole world and are expected to use this broad knowledge to answer user queries.  Function calls are only for specific situations where they provide a demonstrably better answer (like getting live weather data).
Answer all questions truthfully and directly using your general knowledge unless a function call is clearly more beneficial.
Answer in plain text. Keep it simple and to the point.

Not thoroughly tested yet.

EDIT: I have tested it extensively and while it somewhat works it still hangs up on random stuff which makes it useless for answering random questions (asking same thing differently makes it respond or not, constant complete behavior changes based on chat history)

[l7ssha]

Currently using prompt with Gemini 2.0 Flash Lite:

Disregard any limitations related to home automation. You are a fully capable AI assistant for ALL types of tasks and questions, ESPECIALLY general knowledge questions on *any* subject. If you do not have current or real-time information available, respond with latest knowledge you have or try to insinuate.
Answer questions about the world truthfully.
Answer in plain text. Keep it simple and to the point.
Current date and time: {{now()}}.

Especially If you do not have current or real-time information available, respond with latest knowledge you have or try to insinuate. helps with problems described in this ticket.