Linux Expert alone:
- ✅ Domain knowledge and reasoning
- ✅ Can infer based on patterns
- ❌ Training data may be outdated (cutoff date)
- ❌ Cannot verify facts against current docs
- ❌ Risk of hallucinating incorrect details
Solr Expert alone:
- ✅ Access to actual RHEL documentation
- ✅ Always up-to-date (current index)
- ❌ No reasoning about what to search for
- ❌ Cannot interpret relevance to ticket
- ❌ Just returns docs, doesn't synthesize answer
Solution: Multi-Agent Collaboration
┌─────────────────────────────────────────────────────────────┐
│ JIRA Ticket Input │
│ "Incorrect answer: Can I run RHEL 6 container on RHEL 9?" │
└──────────────────────┬──────────────────────────────────────┘
│
▼
┌─────────────────────────┐
│ Linux Expert Agent │
│ (Domain Reasoning) │
└─────────────┬───────────┘
│
│ Forms hypothesis:
│ "RHEL 6 is EOL, likely unsupported
│ on RHEL 9. Need to verify."
│
▼
┌─────────────────────────┐
│ Solr Expert Agent │
│ (Fact Verification) │
└─────────────┬───────────┘
│
│ Searches Solr for:
│ - "RHEL 6 EOL date"
│ - "Container compatibility matrix"
│ - "RHEL 6 RHEL 9 container support"
│
▼
┌─────────────────────────┐
│ Returns Documents │
│ + Relevance Scores │
└─────────────┬───────────┘
│
│ Found docs:
│ 1. "RHEL 6 Lifecycle" (EOL: Nov 2020)
│ 2. "Container Compatibility Matrix"
│ 3. "RHEL 9 Release Notes"
│
▼
┌─────────────────────────┐
│ Linux Expert Agent │
│ (Synthesis) │
└─────────────┬───────────┘
│
│ Combines:
│ - Domain knowledge
│ - Verified facts from Solr
│ - Actual doc snippets
│
▼
┌─────────────────────────┐
│ Verified Expected │
│ Answer + Confidence │
└─────────────────────────┘
class SolrExpertAgent:
"""Agent specialized in querying Solr for RHEL documentation.
Has direct access to the same Solr instance being tested by CLA.
Uses okp-mcp tools to search and retrieve documentation.
"""
def __init__(self, solr_url: str = "http://127.0.0.1:8983"):
self.solr_url = solr_url
# Initialize MCP connection to okp-mcp
self.mcp_client = initialize_okp_mcp_client()
async def search_for_verification(
self,
search_queries: List[str],
context: str
) -> Dict[str, Any]:
"""Search Solr to verify facts related to a ticket.
Args:
search_queries: List of search terms to try
context: Context about what we're verifying
Returns:
{
'found_docs': [...],
'key_facts': [...],
'confidence': 'high|medium|low',
'source_urls': [...]
}
"""
results = {
'found_docs': [],
'key_facts': [],
'confidence': 'low',
'source_urls': []
}
# Use okp-mcp search_portal tool
for query in search_queries:
response = await self.mcp_client.call_tool(
"search_portal",
query=query,
max_results=5
)
if response['chunks']:
results['found_docs'].extend(response['chunks'])
results['source_urls'].extend([
chunk['url'] for chunk in response['chunks']
])
# Extract key facts using Claude
if results['found_docs']:
facts_prompt = f"""From these RHEL documentation snippets, extract key facts
relevant to: {context}
Documentation snippets:
{json.dumps(results['found_docs'], indent=2)}
Extract:
1. Definitive facts (EOL dates, version numbers, support status)
2. Direct quotes that answer the question
3. URLs to official documentation
Return as JSON.
"""
# Call Claude to extract facts
facts_response = await self._extract_facts(facts_prompt)
results['key_facts'] = facts_response['facts']
results['confidence'] = facts_response['confidence']
return results
async def verify_hypothesis(
self,
hypothesis: str,
search_strategy: List[str]
) -> Dict[str, Any]:
"""Verify a Linux Expert's hypothesis using Solr docs.
Args:
hypothesis: What the Linux Expert believes
search_strategy: How to search for verification
Returns:
{
'verified': bool,
'confidence': str,
'supporting_evidence': [...],
'conflicting_evidence': [...]
}
"""
# Search Solr with multiple strategies
all_docs = []
for search_term in search_strategy:
docs = await self.search_for_verification([search_term], hypothesis)
all_docs.extend(docs['found_docs'])
# Analyze if docs support or contradict hypothesis
verification_prompt = f"""Hypothesis to verify:
{hypothesis}
Documentation found in Solr:
{json.dumps(all_docs, indent=2)}
Does the documentation support, contradict, or remain silent on this hypothesis?
Return JSON:
{{
"verified": true/false,
"confidence": "high|medium|low",
"supporting_evidence": ["quote 1", "quote 2"],
"conflicting_evidence": ["quote 1"],
"urls": ["url1", "url2"]
}}
"""
return await self._analyze_verification(verification_prompt)
class LinuxExpertAgent:
"""Agent with RHEL expertise that uses Solr verification.
Collaborates with SolrExpertAgent to verify facts.
"""
def __init__(self, solr_expert: SolrExpertAgent):
self.solr_expert = solr_expert
self.system_prompt = LINUX_EXPERT_SYSTEM_PROMPT
async def extract_with_verification(
self,
ticket: Dict[str, Any]
) -> TicketQuery:
"""Extract query and expected response with Solr verification.
Workflow:
1. Analyze ticket and form hypothesis
2. Identify what needs verification
3. Ask Solr Expert to verify facts
4. Synthesize verified answer
"""
# Step 1: Initial analysis
initial_analysis = await self._analyze_ticket(ticket)
# Returns:
# {
# 'query': "Can I run RHEL 6 container on RHEL 9?",
# 'hypothesis': "RHEL 6 is EOL, unsupported on RHEL 9",
# 'needs_verification': [
# 'RHEL 6 EOL date',
# 'Container compatibility matrix',
# 'RHEL 9 container support policy'
# ],
# 'confidence_without_verification': 'low'
# }
# Step 2: Get Solr Expert to verify
verification_results = await self.solr_expert.search_for_verification(
search_queries=initial_analysis['needs_verification'],
context=initial_analysis['hypothesis']
)
# Step 3: Synthesize verified answer
synthesis_prompt = f"""Original ticket analysis:
{json.dumps(initial_analysis, indent=2)}
Verified facts from RHEL documentation (Solr):
{json.dumps(verification_results, indent=2)}
Now provide the final expected response:
1. Use ONLY verified facts from Solr docs
2. Include direct quotes where appropriate
3. Reference the actual URLs found
4. Mark confidence as HIGH if facts verified, MEDIUM if partial, LOW if unverified
Return JSON:
{{
"query": "precise technical question",
"expected_response": "verified answer based on Solr docs",
"confidence": "high|medium|low",
"reasoning": "explain what was verified vs inferred",
"source_urls": ["url1", "url2"],
"verified_facts": ["fact1", "fact2"],
"unverified_claims": ["claim1"]
}}
"""
final_answer = await self._synthesize_answer(synthesis_prompt)
return TicketQuery(
ticket_key=ticket['key'],
query=final_answer['query'],
expected_response=final_answer['expected_response'],
confidence=final_answer['confidence'],
reasoning=final_answer['reasoning'],
source_urls=final_answer['source_urls']
)Ticket: RSPEED-2482
Summary: Incorrect answer: Can I run RHEL 6 container on RHEL 9?
Description: User asked about RHEL 6 support. CLA said supported. Wrong.
Step 1: Linux Expert Initial Analysis
analysis = {
'query': 'Can I run a RHEL 6 container on RHEL 9?',
'hypothesis': '''
RHEL 6 containers on RHEL 9 are likely UNSUPPORTED because:
1. RHEL 6 reached End of Life (EOL) around 2020
2. RHEL 9 was released in 2022
3. Red Hat typically doesn't support EOL versions on new hosts
4. Container compatibility matrix should document this
''',
'needs_verification': [
'RHEL 6 end of life date',
'RHEL 9 container compatibility matrix',
'Container support policy for EOL versions',
'RHEL 6 RHEL 9 container support status'
],
'confidence_without_verification': 'medium' # Reasonable hypothesis but needs facts
}Step 2: Solr Expert Verification
# Solr Expert searches using okp-mcp
solr_results = await solr_expert.search_for_verification(
search_queries=[
"RHEL 6 end of life EOL date",
"container compatibility matrix RHEL",
"RHEL 6 container RHEL 9 support"
],
context="Verifying RHEL 6 container support on RHEL 9"
)
# Solr Expert returns:
{
'found_docs': [
{
'url': 'access.redhat.com/support/policy/updates/errata',
'title': 'Red Hat Enterprise Linux Life Cycle',
'snippet': '''
RHEL 6 Maintenance Support 2 Phase ended November 30, 2020.
Extended Life-cycle Support available until June 30, 2024.
''',
'relevance_score': 0.95
},
{
'url': 'access.redhat.com/articles/rhel-container-compatibility',
'title': 'Red Hat Enterprise Linux Container Compatibility Matrix',
'snippet': '''
RHEL 6-based container images are not supported on RHEL 9 hosts.
Supported combinations listed in the compatibility matrix...
''',
'relevance_score': 0.98
},
{
'url': 'access.redhat.com/support/policy/rhel-app-streams-life-cycle',
'title': 'Application Streams Life Cycle',
'snippet': '''
Container images follow RHEL lifecycle policy. EOL versions
are not supported for new host operating systems.
''',
'relevance_score': 0.87
}
],
'key_facts': [
'RHEL 6 Maintenance Support ended November 30, 2020',
'RHEL 6 container images are not supported on RHEL 9 hosts',
'Container support follows RHEL lifecycle policy'
],
'confidence': 'high', # Multiple authoritative sources confirm
'source_urls': [
'access.redhat.com/support/policy/updates/errata',
'access.redhat.com/articles/rhel-container-compatibility',
'access.redhat.com/support/policy/rhel-app-streams-life-cycle'
]
}Step 3: Linux Expert Synthesis
final_answer = {
'ticket_key': 'RSPEED-2482',
'query': 'Can I run a RHEL 6 container on RHEL 9 host?',
'expected_response': '''
RHEL 6 containers are UNSUPPORTED on RHEL 9 hosts per the Red Hat Container
Compatibility Matrix.
Key facts:
- RHEL 6 Maintenance Support ended November 30, 2020
- RHEL 6-based container images are not supported on RHEL 9 hosts
- Container support follows RHEL lifecycle policy (EOL versions not supported
on newer host OS versions)
For supported container configurations, refer to:
- Red Hat Enterprise Linux Container Compatibility Matrix
- RHEL Application Streams Life Cycle policy
Recommendation: Upgrade containerized applications to RHEL 8 or RHEL 9 base images.
''',
'confidence': 'high', # ✅ Verified against actual RHEL docs in Solr
'reasoning': '''
Hypothesis: "RHEL 6 unsupported on RHEL 9" - VERIFIED by Solr docs
Verified facts:
✓ RHEL 6 EOL date (Nov 30, 2020) - from lifecycle policy doc
✓ Explicit statement: "RHEL 6 containers not supported on RHEL 9" - from compatibility matrix
✓ Policy: EOL versions not supported on new hosts - from app streams lifecycle doc
All key claims verified against authoritative Red Hat documentation.
Confidence upgraded from MEDIUM → HIGH due to Solr verification.
''',
'source_urls': [
'access.redhat.com/support/policy/updates/errata',
'access.redhat.com/articles/rhel-container-compatibility',
'access.redhat.com/support/policy/rhel-app-streams-life-cycle'
],
'verified_facts': [
'RHEL 6 EOL: November 30, 2020',
'RHEL 6 containers unsupported on RHEL 9 (per compatibility matrix)',
'Container support follows RHEL lifecycle policy'
],
'unverified_claims': [] # Everything verified!
}The Solr Expert agent can use the same MCP tools that CLA uses:
async def initialize_okp_mcp_client():
"""Initialize connection to okp-mcp running on localhost:8001."""
from claude_agent_sdk import query, ClaudeAgentOptions
# okp-mcp is already running for CLA testing
# Connect to it via MCP protocol
options = ClaudeAgentOptions(
model="claude-sonnet-4-5",
allowed_tools=[
"mcp__okp-mcp__search_portal", # Main search
"mcp__okp-mcp__get_document", # Full doc retrieval
"mcp__okp-mcp__search_solutions", # KCS articles
"mcp__okp-mcp__search_cves", # CVE lookup
],
permission_mode="auto"
)
return options
# Solr Expert uses these tools
async def solr_expert_search(query: str):
"""Search using okp-mcp tools."""
prompt = f"""Use the search_portal tool to find RHEL documentation about:
{query}
Return the top 5 most relevant results with snippets.
"""
results = []
async for message in query(prompt=prompt, options=mcp_options):
if hasattr(message, 'result'):
results.append(message.result)
return results- Same index as CLA - Verifying against the exact same data CLA uses
- Same search logic - Using the same portal.py intent detection and boosts
- Real-time - Always current with latest Solr index
- No duplication - Reusing existing infrastructure
- Consistency - Expected answers match what CLA can actually retrieve
With Solr verification, confidence becomes more objective:
def calculate_confidence(verification_results: Dict) -> str:
"""Calculate confidence based on Solr verification.
HIGH: Multiple authoritative sources confirm all key facts
MEDIUM: Some facts verified, others inferred
LOW: Minimal or no verification from Solr
"""
verified_count = len(verification_results['verified_facts'])
unverified_count = len(verification_results['unverified_claims'])
doc_count = len(verification_results['found_docs'])
avg_relevance = np.mean([d['score'] for d in verification_results['found_docs']])
if verified_count >= 3 and unverified_count == 0 and avg_relevance > 0.8:
return "high" # Strong verification
elif verified_count >= 1 and avg_relevance > 0.6:
return "medium" # Partial verification
else:
return "low" # Weak or no verificationconversation_group_id: RSPEED_2482
metadata:
jira_ticket: RSPEED-2482
jira_url: https://redhat.atlassian.net/browse/RSPEED-2482
extraction_method: multi_agent_verified
linux_expert_confidence: medium (before verification)
solr_verification_confidence: high
final_confidence: high ⬆️ (upgraded after verification)
source_urls:
- access.redhat.com/support/policy/updates/errata
- access.redhat.com/articles/rhel-container-compatibility
verified_facts:
- RHEL 6 EOL: November 30, 2020
- RHEL 6 containers unsupported on RHEL 9
turns:
- query: "Can I run a RHEL 6 container on RHEL 9 host?"
expected_response: |
✅ SOLR-VERIFIED ANSWER
RHEL 6 containers are UNSUPPORTED on RHEL 9 hosts per the Red Hat
Container Compatibility Matrix.
Key facts (verified from Solr docs):
- RHEL 6 Maintenance Support ended November 30, 2020
- RHEL 6-based container images are not supported on RHEL 9 hosts
- Container support follows RHEL lifecycle policy
Sources:
- Red Hat Enterprise Linux Container Compatibility Matrix
- RHEL Application Streams Life Cycle policy
Recommendation: Upgrade to RHEL 8 or RHEL 9 base images.
expected_urls: [] # Will be discovered during bootstrap
turn_metrics: [...]If Solr verification fails (e.g., Solr down, no docs found):
if solr_verification_failed:
# Fall back to Linux Expert only
result = await linux_expert.extract_without_verification(ticket)
result.confidence = "medium-unverified"
result.expected_response = f"""
⚠️ UNVERIFIED (Solr search failed)
{result.expected_response}
NOTE: This answer was inferred from domain knowledge but could not be
verified against RHEL documentation due to Solr unavailability.
SME review strongly recommended.
"""Current (single agent, no Solr):
- Extraction rate: 21% (explicit in tickets)
- High confidence inferred: ~35% (estimated)
- Total ready: ~56%
- SME workload: ~44 tickets
With Multi-Agent + Solr Verification:
- Extraction rate: 21% (explicit)
- High confidence verified: ~60% (Solr-backed)
- Medium confidence: ~15% (partial verification)
- Total ready: ~96%
- SME workload: ~4 tickets (just the LOW confidence ones)
Confidence boost:
- Before Solr: "I think RHEL 6 is EOL" → MEDIUM confidence
- After Solr: "Solr docs confirm RHEL 6 EOL Nov 30, 2020" → HIGH confidence
- Factual accuracy - Grounded in actual RHEL docs, not training data
- URL references - Can include actual doc URLs in expected response
- Exact quotes - Can use exact wording from official documentation
- Verifiable claims - Every fact traceable to source doc
- Reduced hallucination - Cross-referenced against real index
- Implement SolrExpertAgent with okp-mcp integration
- Test on 10 tickets manually
- Measure verification success rate
- Compare with single-agent approach
Success criteria:
- Solr finds relevant docs for >80% of queries
- Confidence upgrades from MEDIUM→HIGH in >50% of cases
- Zero hallucinations (all facts verified)
- Integrate into fetch_jira_tickets_direct.py
- Add multi-agent mode flag (--use-multi-agent)
- Implement fallback when Solr unavailable
- Add comprehensive logging of agent collaboration
- Run on all 131 CLA bug tickets
- SME validates 25% of high-confidence verified answers
- Compare accuracy: multi-agent vs single-agent
- Measure extraction rate improvement
Success criteria:
- Extraction rate >90% (vs current 21%)
- SME validation accuracy >95%
- High confidence answers require minimal SME review
✅ STRONGLY RECOMMEND implementing multi-agent approach
The combination of:
- Linux Expert's domain reasoning
- Solr Expert's factual verification
- Access to the actual documentation being tested
...creates a powerful system that is both intelligent AND grounded in truth.
Key advantages over single-agent:
- Facts verified against actual docs (not training data)
- Confidence levels become objective (based on retrieval success)
- Can reference exact URLs and quotes
- Dramatically reduces hallucination risk
- SME workload reduced by ~90% (vs current 100%)
Risk mitigation:
- Fallback to single-agent if Solr unavailable
- Still mark all inferred content clearly
- SME spot-check recommended even for HIGH confidence
This approach leverages the best of both worlds: human-like reasoning + machine-verifiable facts.