HEAL Pattern Fix Workflow - TODO

High Priority

Bootstrap Workflow Validation

Test Full Agentic Workflow End-to-End from Scratch - Critical, High Impact
- Run complete autonomous pipeline on fresh checkout:
  1. Extract JIRA tickets with multi-agent verification
  2. Review and refine extracted answers autonomously
  3. Re-extract failures flagged by review agent
  4. Discover patterns from quality Q&A
  5. Convert to lightspeed-evaluation format
  6. Run full evaluation
- Validate autonomous quality loop works (Review Agent → Linux Expert refinement)
- Measure success rate: % tickets passing review without human intervention
- Document any manual interventions needed (should be zero for agentic POC)
- Files involved:
  - src/heal/bootstrap/extract_jira_tickets.py
  - src/heal/bootstrap/refine_extracted_tickets.py
  - scripts/discover_patterns.py
  - scripts/convert_to_eval_format.py
- Impact: Validates fully autonomous multi-agent workflow works end-to-end
- Why: Current testing done mid-process with pre-existing data; need clean-slate validation
- Priority: CRITICAL - proves POC concept before scaling
- Status: Pending - refinement improvements not yet tested fresh

Improve Progress Visibility and Timing

Add Historical Timing Data for Better Estimates - High Impact, Low Effort
- Store historical timing data in .diagnostics/timing_history.json
  - Track completion time (ms) per pattern and evaluation type
  - Store: pattern_id, eval_type (full/retrieval), num_runs, duration_ms, timestamp
- Calculate estimates from historical averages
  - Use moving average of last N runs for same pattern/config
  - Fall back to default estimate if no historical data
  - Format: historical_avg_ms if historical_avg_ms else runs * 2 * 60000
- Update progress messages to use milliseconds instead of seconds
  - Capture: start_ms = int(time.perf_counter() * 1000)
  - Report: print(f"Completed in {elapsed_ms}ms ({elapsed_ms/60000:.1f} min)")
- Files to modify:
  - src/heal/agents/okp_mcp_agent.py:
    - diagnose_full() - add timing persistence
    - diagnose_retrieval_only() - add timing persistence
  - src/heal/runners/run_pattern_fix_poc.py:
    - Add timing history loader/saver
    - Update estimate calculations
- Impact: Users get accurate time estimates based on actual historical performance
- Why: Current estimates (runs * 2 minutes) are hardcoded and often inaccurate
- Priority: High - improves user experience during long-running batch operations

Medium Priority

Pattern Fix Loop Enhancements

Performance and Quality

Future Enhancements

Jailbreak Protection Investigation (Separate from HEAL)

Investigate Guardrail Configuration for CLA Jailbreak Protection - Medium Impact, Medium Effort
- Context: Open JIRA tickets (e.g., RSPEED-2219) document jailbreak vulnerabilities
- Current status: Jailbreaks likely work (no Llama Guard deployed locally)
- Investigation steps:
  1. Find where guardrails are configured:
    - CLA: Check for input validation, safety filters
    - lightspeed-stack: Check middleware, request handlers
    - llamastack: Check inference safety settings
  2. Search for existing protections:
```
# Configuration files
find . -name "*config*.yaml" -o -name "*safety*.yaml"

# Code patterns
grep -r "safety\|guard\|filter\|sanitize" --include="*.py"
grep -r "jailbreak\|prompt.injection" --include="*.py"
```
  3. Understand current state:
    - What protections exist (if any)?
    - Where do they live (CLA/lightspeed-stack/llamastack)?
    - Why aren't they blocking jailbreaks?
- Potential fixes (after investigation):
  - Deploy Llama Guard (if available in llamastack)
  - Add pattern-based input filters (regex for common jailbreaks)
  - Strengthen system prompts (though this alone is weak)
  - Add response validation (detect prompt leakage)
- Scope for HEAL:
  - Jailbreak tickets marked OUT_OF_SCOPE (correctly)
  - Not RHEL technical questions - security tests
  - Separate concern from fixing incorrect RHEL answers
- Impact: Addresses security vulnerabilities documented in open tickets
- Priority: Medium - separate from core HEAL workflow
- Status: Requires investigation before implementation
- Note: Different problem domain than HEAL (security vs correctness)

Product-Based Intent System (Separate Interface)

Agentic Product Coverage and Intent Discovery - High Impact, High Effort
- Concept: Test-driven intent system that ensures CLA coverage of all Red Hat products
- Workflow:
  1. Product Enumeration Agent - LLM discovers all Red Hat products
    - RHEL (versions 6-10), Satellite, OpenShift, Ansible Automation, etc.
    - Query Solr/OKP for product list or use RH product catalog
  2. Test Discovery/Generation - For each product:
    - Find existing tickets: find_tickets(product="Satellite")
    - If gaps exist: Generate synthetic tests for untested areas
  3. Failure Detection - Test CLA knowledge:
    - If tests pass: Note in comments "CLA has good {product} coverage"
    - If tests fail: Identify knowledge gaps
  4. Intent Rule Builder - Test-driven rule creation:
    - Iterate on intent rules until tests pass
    - Rules guide how to answer product-specific questions
- Integration with HEAL:
  - Intent rules feed into Pattern Discovery (product-aware grouping)
  - Proactive (ensure coverage) complements reactive (fix bugs)
  - Could use same agents: Linux Expert, Solr Expert, Review Agent
- Example:
```
# Discover products
products = ["RHEL", "Satellite", "OpenShift", ...]

# Find/create tests
satellite_tests = find_or_create_tests(product="Satellite")

# Test CLA
results = test_cla(satellite_tests)  # 2/3 fail on uninstall

# Build intent rules (test-driven)
while not all_tests_pass():
    intent_rules["satellite_uninstall"] = refine_rule()
```
- Architecture:
  - Separate interface from current HEAL workflow
  - Orthogonal: HEAL fixes existing bugs, Intent System prevents future gaps
  - Complementary: Intent rules improve pattern discovery accuracy
- Impact: Systematic product coverage, prevents blind spots, broadens agentic scope
- Credit: Proposed by user's friend
- Priority: Future - after core HEAL workflow validated
- Status: Design idea captured, pending HEAL core completion

Low Priority

Code Quality and Maintenance

Completed ✅

Notes

Recent Improvements (2026-04-15)

Problem: Batch pattern fix workflow sat silently with no progress output, making it impossible to tell if running or stuck.

Root Causes:

Python stdout buffering when output piped through tee
Progress monitoring watching wrong files (results.csv instead of evaluation_*_summary.json)
Auth conflicts between Anthropic SDK and Claude Agent SDK (ADC issues)
No cleanup on Ctrl+C - left on pattern branches instead of returning to main

Solutions Applied:

Added sys.stdout.reconfigure(line_buffering=True) + PYTHONUNBUFFERED=1
Fixed file pattern to run_*/evaluation_*_summary.json
Switched all LLM calls to use Claude Agent SDK's query() function
Added signal handlers (SIGINT/SIGTERM) + atexit cleanup
Added start/end timestamps and progress updates with explicit flush()
Enhanced output to show phase summaries instead of raw evaluation logs

User Preference: Milliseconds for all timing (not seconds), coming from nanosecond-level systems programming background

Next Steps

Waiting on user review for:

Library API design (docs/LIBRARY_API_DESIGN.md)
Performance metrics design (docs/PERFORMANCE_METRICS_DESIGN.md)

Once approved, can proceed with implementation.

FilesExpand file tree

TODO.md

Latest commit

History