Skip to content

Gate Agent for Generation Quality Control#14

Merged
Lum1104 merged 7 commits intomainfrom
gate_agent
Nov 24, 2025
Merged

Gate Agent for Generation Quality Control#14
Lum1104 merged 7 commits intomainfrom
gate_agent

Conversation

@Lum1104
Copy link
Copy Markdown
Owner

@Lum1104 Lum1104 commented Nov 21, 2025

📝 Description

Gate Agent for Quality Control

Summary

Introduces an optional Gate Agent node into the MER pipeline. This agent acts as a quality control layer, evaluating intermediate analysis results (Audio, Video, Peak Frame) and requesting refinements via dynamic prompting if the quality is insufficient or if modalities conflict.

Key Changes

  • New Node: GateAgent class (mer_factory/nodes/gate_agent.py) that uses a ReAct loop to evaluate outputs.
  • CLI Option: Added --use-gate-agent flag (default: False) to enable this feature.
  • Graph Routing: Updated graph.py to conditionally route through the Gate Agent.
  • Retry Logic: Implemented feedback loops where the Gate Agent generates specific refinement prompts for failed modalities.
  • Tools: Added analyze_media_metrics and analyze_video_motion to assist the Gate Agent in verification.
  • Optimization: Added skip logic in async_nodes.py to avoid re-processing modalities that have already passed evaluation.

Motivation

To address the "garbage in, garbage out" problem in multimodal analysis. By filtering and refining weak or conflicting signals before the final synthesis, we aim to improve the overall accuracy and reliability of the generated emotional reasoning.


Note

Adds an optional Gate Agent to the MER pipeline for quality control with dynamic re-prompts and retries, enabled via --use-gate-agent, updating graph/state/nodes, and documenting usage.

  • Pipeline/Graph:
    • Add Gate Agent routing (mer_factory/graph.py): conditional node gate_agent, route_gate_agent, and edges to retry audio/video/peak_frame or proceed to synthesize_summary.
  • Nodes:
    • New GateAgent (mer_factory/nodes/gate_agent.py): ReAct-style evaluation, tool calls, JSON decisions, feedback history, retry caps, refined prompt generation, and state updates.
    • Enhance async nodes (mer_factory/nodes/async_nodes.py): support dynamic prompts per modality, skip re-processing when already passed, integrate caching loads.
  • Tools:
    • New verification utilities (mer_factory/tools.py): analyze_media_metrics, analyze_video_motion, extract_subtitles, run_terminal_command, run_python_code.
  • State/Config/CLI:
    • Extend state with gate fields (mer_factory/state.py): gate_decision, retry_target, gate_feedback, retry_counts, dynamic_prompts.
    • Add config/flag (utils/config.py, main.py): use_gate_agent option; wire flag to create_graph(use_gate_agent=...) and Typer CLI (--use-gate-agent/-uga).
  • Docs:
    • Document Gate Agent and CLI flag in README.md and README_zh.md; update roadmap image.

Written by Cursor Bugbot for commit 564e374. This will update automatically on new commits. Configure here.

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is being reviewed by Cursor Bugbot

Details

Your team is on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle for each member of your team.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

Comment thread mer_factory/tools.py
Comment thread mer_factory/tools.py
except subprocess.TimeoutExpired:
return "Error: Command timed out."
except Exception as e:
return f"Error executing command: {str(e)}"
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Command injection vulnerability in whitelist validation

The run_terminal_command function attempts to secure execution by validating only the first token of the command against a whitelist. However, by setting shell=True, it allows attackers to chain commands (e.g., ffmpeg -i x; rm -rf /). The validation logic checks ffmpeg, passes, and then the shell executes the malicious chained command.

Fix in Cursor Fix in Web

Comment thread mer_factory/nodes/async_nodes.py Outdated
Comment thread mer_factory/nodes/gate_agent.py
prompt = prompts.get_audio_prompt(has_label)
if has_label:
prompt = prompt.format(label=ground_truth_label)
# Check for dynamic prompt from Gate Agent
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Cache logic blocks Gate Agent refinement retries

When cache is enabled, the function checks for and loads existing results before checking for dynamic_prompts. This causes the node to return the cached (rejected) output instead of generating a new analysis with the refined prompt, effectively disabling the Gate Agent's retry mechanism when caching is active.

Fix in Cursor Fix in Web

Comment thread mer_factory/tools.py
Comment thread mer_factory/tools.py
# (as per user request for "gate agent write the code is fine")
# We pass 'pd' so they can use pandas easily
exec_globals = {"pd": pd, "print": print}
exec(code, exec_globals)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Unrestricted execution of arbitrary code

The run_python_code method uses exec() with a globals dictionary that does not strictly explicitly disable __builtins__. This allows the code generated by the agent (or injected via prompts) to import modules like os and execute arbitrary system commands (RCE), posing a significant security risk.

Fix in Cursor Fix in Web

Comment thread mer_factory/tools.py
# 1. Get Duration and Audio Volume
# ffmpeg -i input -filter:a volumedetect -f null /dev/null
cmd = f'ffmpeg -i "{path}" -filter:a volumedetect -f null -'
result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Command injection vulnerability in media analysis tools

The analyze_media_metrics and analyze_video_motion functions use shell=True with file paths that come from LLM-generated tool arguments. This creates a command injection vulnerability where a malicious or compromised LLM could execute arbitrary shell commands by crafting special file paths. The run_terminal_command function correctly uses shell=False and shlex.split for security, but these media analysis functions don't follow the same pattern. File paths should be passed as list arguments with shell=False instead of string interpolation with shell=True.

Additional Locations (1)

Fix in Cursor Fix in Web

Comment thread mer_factory/nodes/gate_agent.py
- Implement two-stage verification: plan generation → execution
- Track feedback history to avoid repeated rejections
- Save refined prompts to files for future usage
- Increase max verification turns from 5 to 15
- Fix Windows path handling in ffmpeg commands
- Add gate_decision and retry_target to state tracking
Comment thread mer_factory/nodes/gate_agent.py Outdated
@Lum1104 Lum1104 merged commit 6cc7263 into main Nov 24, 2025
3 checks passed
@Lum1104 Lum1104 deleted the gate_agent branch November 24, 2025 09:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant