Skip to content

SCALAR_VECTOR_FIXES.md

Latest commit

 

History

History
199 lines (155 loc) · 6.44 KB

SCALAR_VECTOR_FIXES.md

File metadata and controls

199 lines (155 loc) · 6.44 KB
Raw

Scalar/Vector Bug Fixes - Pattern Evaluation

Summary

Fixed critical bugs in pattern-level evaluation that were causing:

  1. ❌ False "documentation gap" errors (0.00 pattern averages despite 0.15 URL F1)
  2. ❌ False RAG bypass detection (missing metadata)
  3. ❌ Incorrect problem analysis (couldn't detect retrieval vs answer problems)

Root Cause

Key Mismatch Bug: _build_evaluation_result_from_runs() was looking for metric keys that didn't exist:

# ❌ WRONG - looking for full metric identifiers
url_f1=averages.get("custom:url_retrieval_eval")  # Returns None!
answer_correctness=averages.get("custom:answer_correctness")  # Returns None!
faithfulness=averages.get("ragas:faithfulness")  # Returns None!

But parse_results_per_ticket() created dicts with simplified keys:

# ✅ Actual keys in the dict
run_metrics["url_f1"] = score
run_metrics["answer_correctness"] = score  
run_metrics["faithfulness"] = score

Result: All .get() calls returned None, causing pattern-level aggregates to be 0.00 even when per-run scores were correct.

Fixes Applied

Fix 1: Correct Key Names in _build_evaluation_result_from_runs

File: src/heal/agents/okp_mcp_agent.py:3366-3379

# ✅ FIXED - use simplified keys that actually exist
result = EvaluationResult(
    ticket_id=ticket_id,
    url_f1=averages.get("url_f1"),  # ✅ Correct
    mrr=averages.get("mrr"),  # ✅ Correct
    context_relevance=averages.get("context_relevance"),  # ✅ Correct
    context_precision=averages.get("context_precision"),  # ✅ Correct
    faithfulness=averages.get("faithfulness"),  # ✅ Correct
    answer_correctness=averages.get("answer_correctness"),  # ✅ Correct
    response_relevancy=averages.get("response_relevancy"),  # ✅ Correct
    # ... other fields
)

Fix 2: Extract Metadata in parse_results_per_ticket

File: src/heal/agents/okp_mcp_agent.py:2812-2930

Problem: parse_results_per_ticket() only extracted metric scores, not metadata like:

  • rag_used, docs_retrieved (needed for RAG bypass detection)
  • expected_urls, retrieved_urls (needed for URL comparison)
  • tool_calls, contexts (needed for diagnostics)

Solution: Extract metadata from first row of each ticket and include it in results:

# Old structure: Dict[ticket_id, List[run_metrics]]
# New structure: Dict[ticket_id, {"runs": [...], "metadata": {...}}]

results[ticket_id] = {
    "runs": ticket_runs,  # List of per-run metric dicts
    "metadata": {
        "tool_calls": tool_calls,
        "contexts": contexts,
        "expected_urls": expected_urls,  # From test config YAML
        "retrieved_urls": retrieved_urls,  # From tool_calls JSON
        "rag_used": rag_used,  # Was search tool called?
        "docs_retrieved": docs_retrieved,  # Were docs returned?
    }
}

Fix 3: Pass Metadata to _build_evaluation_result_from_runs

File: src/heal/agents/okp_mcp_agent.py:3586-3591, 3677-3682

# ✅ FIXED - pass metadata to builder
for tid, ticket_data in per_ticket_data.items():
    ticket_result = self._build_evaluation_result_from_runs(
        tid, 
        ticket_data["runs"],  # Per-run metrics
        ticket_data["metadata"]  # Ticket-level metadata
    )
    per_ticket_results[tid] = ticket_result

Fix 4: Populate Metadata Fields in EvaluationResult

File: src/heal/agents/okp_mcp_agent.py:3373-3399

result = EvaluationResult(
    # ... averaged metrics ...
    # Metadata from CSV/config (same across all runs)
    tool_calls=metadata.get("tool_calls"),
    contexts=metadata.get("contexts"),
    expected_urls=metadata.get("expected_urls", []),
    retrieved_urls=metadata.get("retrieved_urls", []),
    rag_used=metadata.get("rag_used", False),
    docs_retrieved=metadata.get("docs_retrieved", False),
)

Impact

Before Fixes:

📊 PER-RUN SCORES:
   Run 1: Answer=0.57, Faith=0.63, URL_F1=0.17
   Run 2: Answer=0.57, Faith=0.53, URL_F1=0.11
   Run 3: Answer=0.73, Faith=0.26, URL_F1=0.17

📊 SCORES (avg):
   ❌ Answer Correctness: 0.62
   ⚠️ Faithfulness:       0.48
   ✅ URL F1:             0.15
   ✅ Context Relevance:  0.78  ← Proves docs retrieved!

📊 BASELINE METRICS:
   URL F1 (avg):             0.00  ← BUG! Should be 0.15
   Answer Correctness (avg): 0.00  ← BUG! Should be 0.62
   Faithfulness (avg):       0.00  ← BUG! Should be 0.48

❌ DOCUMENTATION GAP DETECTED  ← FALSE POSITIVE!
⚠️ RAG Bypass: 3 tickets  ← FALSE POSITIVE!

After Fixes:

📊 PER-RUN SCORES:
   Run 1: Answer=0.57, Faith=0.63, URL_F1=0.17
   Run 2: Answer=0.57, Faith=0.53, URL_F1=0.11
   Run 3: Answer=0.73, Faith=0.26, URL_F1=0.17

📊 SCORES (avg):
   ❌ Answer Correctness: 0.62
   ⚠️ Faithfulness:       0.48
   ✅ URL F1:             0.15
   ✅ Context Relevance:  0.78

📊 BASELINE METRICS:
   URL F1 (avg):             0.15  ← ✅ Correct!
   Answer Correctness (avg): 0.62  ← ✅ Correct!
   Faithfulness (avg):       0.48  ← ✅ Correct!
   Success Rate:             0%

🎯 RETRIEVAL PROBLEM DETECTED  ← ✅ Correct diagnosis!
   → Low URL F1, MRR, but docs ARE being retrieved
   → Multi-agent Solr optimization should help

What Now Works

✅ Pattern-level aggregates match per-run averages
✅ RAG bypass detection uses actual metadata (not false positives)
✅ Problem analysis correctly identifies retrieval vs answer issues
✅ URL comparison works (has expected_urls and retrieved_urls)
✅ Multi-agent optimization will trigger (not skipped due to false documentation gap)
✅ MRR is extracted and used in problem detection

Files Modified

  1. src/heal/agents/okp_mcp_agent.py:
    • parse_results_per_ticket() - Extract metadata + metrics
    • _build_evaluation_result_from_runs() - Use correct keys + metadata
    • Both callers updated (diagnose_retrieval_only, diagnose)

Testing

Run the pattern fix loop to verify:

cd /home/emackey/Work/rhel-lightspeed/HEAL
./runners/fix.sh BOOTLOADER_GRUB_ISSUES

Expected behavior:

  • ✅ Pattern averages match per-run scores
  • ✅ No false "documentation gap" errors
  • ✅ Correct RAG bypass detection
  • ✅ Multi-agent Solr optimization runs
  • ✅ Per-ticket diagnostics show correct URLs

Related Issues

This bug was related to but different from:

  • List vs scalar: This was a key mismatch, not type mismatch
  • Missing metadata: Second bug - parse_results_per_ticket wasn't extracting metadata

Both are now fixed!