google generative ai: inconsistent when controlling entities

[arpitgupta]

The problem

I am not sure when this started happening, I have not been actively using this integration. When I first set it up it worked well.

Today I am running into an issue where it is able to read the state of the entity but when I ask it to turn it on or off it fails to do it.

For example I asked it about the status of sun blind and it correctly determined that but could not close it

2024-12-20 20:24:02.932 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Input: 'is sun blind open?' with history: []
2024-12-20 20:24:03.476 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "Yes, the sun blind in the Living Room is open.\n"
]
2024-12-20 20:24:12.585 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Input: 'close it please' with history: [parts {
  text: "is sun blind open?"
}
role: "user"
, parts {
  text: "Yes, the sun blind in the Living Room is open.\n"
}
role: "model"
]
2024-12-20 20:24:13.321 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [function_call {
  name: "HassTurnOff"
  args {
    fields {
      key: "name"
      value {
        string_value: "Living Room Blind 3 , sun blind"
      }
    }
    fields {
      key: "domain"
      value {
        list_value {
          values {
            string_value: "cover"
          }
        }
      }
    }
  }
}
]
2024-12-20 20:24:13.321 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool call: HassTurnOff({'domain': ['cover'], 'name': 'Living Room Blind 3 , sun blind'})
2024-12-20 20:24:13.322 INFO (MainThread) [homeassistant.helpers.intent] Triggering intent handler <OnOffIntentHandler - HassTurnOff>
2024-12-20 20:24:13.322 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool response: {'error': 'MatchFailedError', 'error_text': "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Living Room Blind 3 , sun blind', area_name=None, floor_name=None, domains={'cover'}, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}
2024-12-20 20:24:13.888 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "OK. I couldn\'t find the Living Room Blind 3, sun blind.  Is there another name for it?\n"

I tried the same with a dimmer and saw same behavior

role: "model"
, parts {
  text: "is dimmer in living room on?"
}
role: "user"
, parts {
  text: "Yes, the living room dimmer is on.\n"
}
role: "model"
]
2024-12-20 20:25:38.015 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [function_call {
  name: "HassTurnOff"
  args {
    fields {
      key: "name"
      value {
        string_value: "Dimmer, living room dimmer"
      }
    }
    fields {
      key: "area"
      value {
        string_value: "Living Room"
      }
    }
  }
}
]
2024-12-20 20:25:38.016 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool call: HassTurnOff({'area': 'Living Room', 'name': 'Dimmer, living room dimmer'})
2024-12-20 20:25:38.016 INFO (MainThread) [homeassistant.helpers.intent] Triggering intent handler <OnOffIntentHandler - HassTurnOff>
2024-12-20 20:25:38.025 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool response: {'error': 'MatchFailedError', 'error_text': "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Dimmer, living room dimmer', area_name='Living Room', floor_name=None, domains=None, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}
2024-12-20 20:25:38.572 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "OK. I couldn\'t find the dimmer in the living room. Is there another name for it?\n"

I have also seen times today where both the above calls actually worked. So it randomly works and then stops working. So I am not sure what's going on.

What version of Home Assistant Core has the issue?

core-2024.12.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

Google Generative AI

Link to integration documentation on our website

https://www.home-assistant.io/integrations/google_generative_ai_conversation

Diagnostics information

2024-12-20 20:24:02.932 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Input: 'is sun blind open?' with history: []
2024-12-20 20:24:03.476 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "Yes, the sun blind in the Living Room is open.\n"
]
2024-12-20 20:24:12.585 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Input: 'close it please' with history: [parts {
  text: "is sun blind open?"
}
role: "user"
, parts {
  text: "Yes, the sun blind in the Living Room is open.\n"
}
role: "model"
]
2024-12-20 20:24:13.321 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [function_call {
  name: "HassTurnOff"
  args {
    fields {
      key: "name"
      value {
        string_value: "Living Room Blind 3 , sun blind"
      }
    }
    fields {
      key: "domain"
      value {
        list_value {
          values {
            string_value: "cover"
          }
        }
      }
    }
  }
}
]
2024-12-20 20:24:13.321 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool call: HassTurnOff({'domain': ['cover'], 'name': 'Living Room Blind 3 , sun blind'})
2024-12-20 20:24:13.322 INFO (MainThread) [homeassistant.helpers.intent] Triggering intent handler <OnOffIntentHandler - HassTurnOff>
2024-12-20 20:24:13.322 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool response: {'error': 'MatchFailedError', 'error_text': "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Living Room Blind 3 , sun blind', area_name=None, floor_name=None, domains={'cover'}, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}
2024-12-20 20:24:13.888 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "OK. I couldn\'t find the Living Room Blind 3, sun blind.  Is there another name for it?\n"







role: "model"
, parts {
  text: "is dimmer in living room on?"
}
role: "user"
, parts {
  text: "Yes, the living room dimmer is on.\n"
}
role: "model"
]
2024-12-20 20:25:38.015 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [function_call {
  name: "HassTurnOff"
  args {
    fields {
      key: "name"
      value {
        string_value: "Dimmer, living room dimmer"
      }
    }
    fields {
      key: "area"
      value {
        string_value: "Living Room"
      }
    }
  }
}
]
2024-12-20 20:25:38.016 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool call: HassTurnOff({'area': 'Living Room', 'name': 'Dimmer, living room dimmer'})
2024-12-20 20:25:38.016 INFO (MainThread) [homeassistant.helpers.intent] Triggering intent handler <OnOffIntentHandler - HassTurnOff>
2024-12-20 20:25:38.025 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool response: {'error': 'MatchFailedError', 'error_text': "<MatchFailedError result=MatchTargetsResult(is_match=False, no_match_reason=<MatchFailedReason.NAME: 1>, states=[], no_match_name=None, areas=[], floors=[]), constraints=MatchTargetsConstraints(name='Dimmer, living room dimmer', area_name='Living Room', floor_name=None, domains=None, device_classes=None, features=None, states=None, assistant='conversation', allow_duplicate_names=False), preferences=MatchTargetsPreferences(area_id=None, floor_id=None)>"}
2024-12-20 20:25:38.572 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "OK. I couldn\'t find the dimmer in the living room. Is there another name for it?\n"

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant]

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of google_generative_ai_conversation can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign google_generative_ai_conversation Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

google_generative_ai_conversation documentation
google_generative_ai_conversation source
_{^{(message by IssueLinks)}}

[arpitgupta]

For example I restarted HA container and now it's working fine for a little bit and then became in consistent again.

[tronikos]

What's the entity name? It tried to control: "Living Room Blind 3 , sun blind" Do you have a comma in the name?
Anyway, there is likely little we can do on the integration side. See if you have better luck with Gemini 2.0.

[arpitgupta]

entity id is: cover.living_room_blind_3 and I did not set any explicit name for it. It's name shows up as Living Room Blind 3 and I added an alias for it sun blind.

It tried to control: "Living Room Blind 3 , sun blind"

This is the part that confused me as well, as to why its merging the entity name and alias when trying to control the device. But is able to determine the state when I ask it if its open.

Trying out 2.0 experimental and see how it goes.

[tronikos]

I personally don't use aliases. Can you add to the prompt in settings something like: "When controlling devices, either use the entity name or alias, don't combine them."

[arpitgupta]

I just add this text to the instructions section when I configure the service?

Also Gemini 2.0 experimental is working better. However it required me to change the top k value to 40.

[tronikos]

Select the configure button in the settings > integrations > Google Generative AI Conversation
The default instructions is:

"You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Answer in plain text. Keep it simple and to the point."

Add at the end of it: "Don't use aliases when controlling devices."

[rozza-m]

I have this too, even when I have a single device exposed that doesn't have any aliases.