Skip to content

Add opt-in Agent Mode action publishing (GA-AUD)#93

Merged
Chance-Konstruktion merged 1 commit into
mainfrom
claude/amazing-davinci-K2HYX
May 24, 2026
Merged

Add opt-in Agent Mode action publishing (GA-AUD)#93
Chance-Konstruktion merged 1 commit into
mainfrom
claude/amazing-davinci-K2HYX

Conversation

@Chance-Konstruktion
Copy link
Copy Markdown
Owner

Summary

Implements GA-AUD — the Home Assistant side of Agent Mode. The integration can now publish controller actions to the executor shipped in #92 (GA-109), so Agent Mode works end-to-end.

When enabled, each analyzed frame additionally:

  1. asks the LLM for one controller action (PromptBuilder.build_action),
  2. validates it against the action schema + a button whitelist (parse_action),
  3. publishes it to gaming_assistant/{client_id}/action for the executor.

The normal tip pipeline is unchanged — this is an additional inference per frame (the chosen "tip + action" behavior).

Safety model

  • Opt-in master switch CONF_AGENT_MODE (default False), exposed via a new gaming_assistant.set_agent_mode service and an Agent Mode switch entity.
  • Runtime-only / resets to OFF on every restart — the coordinator is built from entry.data only, so I deliberately did not persist agent mode. The AI never controls inputs unless deliberately re-enabled.
  • Isolated — action generation runs behind asyncio.wait_for + a broad try/except, so a bad model reply / timeout can never disrupt the tip pipeline.
  • Whitelist enforced both in the prompt and in parse_action; no_op/invalid output is dropped (never published).
  • The executor still enforces its own whitelist, --dry-run, and emergency stop. Recommended first run: executor in --dry-run.

Changes

  • const: CONF_AGENT_MODE, DEFAULT_AGENT_MODE, MQTT_ACTION_TOPIC, AGENT_VALID_BUTTONS.
  • coordinator: agent_mode flag + set_agent_mode() setter, async_publish_action(), and the isolated _maybe_publish_agent_action() hook after a successful tip.
  • image_processor: generate_action()build_action → LLM → parse_action; returns None on invalid/no_op output, never raises.
  • service set_agent_mode (+ services.yaml), Agent Mode switch entity, and agent_mode strings in EN/DE/ES/FR/IT.
  • tests: tests/test_agent_mode.py — behavioural tests for generate_action + static wiring contracts (the coordinator can't be imported without a real HA env, mirroring test_coordinator.py).
  • Docs: CHANGELOG, README (enable example + safety note), ROADMAP (GA-AUD → 🟡, per-action confirmation still open).

Test plan

  • python -m unittest discover -s tests313 tests, all green (+15 new).
  • CI coverage gate locally: pytest --cov=custom_components/gaming_assistant --cov-fail-under=4548.30%.
  • py_compile + JSON validation for all touched files.
  • Live end-to-end on real hardware (vgamepad/ViGEmBus) — not possible in CI.

Follow-up

Per-action confirmation UI (the remaining half of GA-AUD) is intentionally out of scope.

https://claude.ai/code/session_01VaYnzPqHF5kxB3stM73g5i


Generated by Claude Code

Wires the Home Assistant side of Agent Mode so the integration can publish
controller actions to the executor shipped in GA-109. When enabled, each
analyzed frame additionally asks the LLM for one action
(PromptBuilder.build_action), validates it against the schema + a button
whitelist (parse_action), and publishes it to
gaming_assistant/{client_id}/action.

Strictly opt-in and safe by design: a runtime-only master switch that
always resets to OFF on restart (CONF_AGENT_MODE, default False), exposed
via a new set_agent_mode service and an Agent Mode switch entity. Action
generation lives behind asyncio.wait_for and a broad try/except so it can
never disrupt the normal tip pipeline. The executor still enforces its own
whitelist, dry-run and emergency stop.

- const: CONF_AGENT_MODE, DEFAULT_AGENT_MODE, MQTT_ACTION_TOPIC, AGENT_VALID_BUTTONS
- coordinator: agent_mode flag + setter, async_publish_action, isolated
  _maybe_publish_agent_action hook after a successful tip
- image_processor: generate_action() (build_action -> LLM -> parse_action),
  returns None on invalid/no_op output (never raises)
- service set_agent_mode + Agent Mode switch + EN/DE/ES/FR/IT strings
- tests/test_agent_mode.py: generate_action behaviour + wiring contracts
@Chance-Konstruktion Chance-Konstruktion merged commit f0ba2d3 into main May 24, 2026
3 checks passed
@Chance-Konstruktion Chance-Konstruktion deleted the claude/amazing-davinci-K2HYX branch May 24, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants