feat: Add Agent QA Pipeline template #6053
Open
Samir-atra wants to merge 1 commit intoaden-hive:mainfrom
Open
Conversation
This commit adds the Agent QA Pipeline template which which performs quality assessment on other Hive agents through: - Static analysis (topology, patterns, edge consistency) - Functional testing (spec-level reasoning) - Resilience testing (error handling patterns) - Security auditing (OWASP LLM Top 10) The agent demonstrates the following framework features: (at 0% template coverage before this PR): - Fan-out/fan-in pattern: 3 parallel test runners - ON_FAILURE edge: load-agent -> generate-report - HITL pause_nodes: review-test-plan - Conditional routing: judge -> PASS/FAIL vs CONDITIONAL - Feedback loop with max_node_visits: request-fixes -> load-agent (max 3) - nullable_output_keys: load_errors, test_preferences, fix_suggestions Phase 1 (works today): - Static analysis + spec-level LLM reasoning for all 3 test categories - Demonstrates all graph patterns - Useful as-is for structural validation Phase 2 (needs Proposal 1 - sub-graph execution) - Runtime testing of target agents Phase 3 (needs Proposals 2+3) - Tool interception for resilience testing - Snapshot comparison for regression testing Files: - examples/templates/agent_qa_pipeline/agent.py - Main agent class - examples/templates/agent_qa_pipeline/agent.json - Declarative agent spec - examples/templates/agent_qa_pipeline/config.py - Configuration - examples/templates/agent_qa_pipeline/nodes/__init__.py - Node definitions - examples/templates/agent_qa_pipeline/tests/test_structure.py - Structure tests Resolves aden-hive#4286
PR Requirements WarningThis PR does not meet the contribution requirements. PR Author: @Samir-atra To fix:
Exception: To bypass this requirement, you can:
Micro-fix requirements (must meet ALL):
Why is this required? See #472 for details. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR implements the Agent QA Pipeline template as proposed in Issue #4286 - a goal-driven agent that performs quality assessment on other Hive agents.
What it does
The Agent QA Pipeline performs comprehensive quality assessment on other Hive agents through:
The pipeline produces a PASS / CONDITIONAL / FAIL verdict with a score (0-100) and letter grade (A-F), along with actionable fix suggestions and iterative re-test cycles.
Graph Architecture
Framework Features Demonstrated
This template is the first to demonstrate these features (all at 0% template coverage before this PR):
on_failureedgepause_nodesmax_node_visitsnullable_output_keysType of Change
Related Issues
Resolves #4286
Changes Made
Testing
All 27 tests pass:
Checklist
Implementation Notes
Phase 1 (Works Today)
Future Enhancements (Framework Proposals)
Each phase is independently valuable and can be implemented incrementally.