feat: Step 2.6 & Phase 3 - Unified Metrics Integration & Abstract Interfaces

Step 2.6: Unified Metrics Integration
- Integrated unified metrics into validation system
  - ValidationMetrics now forwards to UnifiedMetricsCollector
- Integrated unified metrics into RAG system
  - RAG retrieval records metrics (avg_similarity, context_quality, retrieval_time_ms)

Phase 3: Learning & Post-Processing Abstract Interfaces
- Created stillme_core/learning/base.py
  - LearningPipeline abstract interface
  - LearningFetcher abstract interface
  - LearningResult dataclass
- Created stillme_core/postprocessing/base.py
  - PostProcessor abstract interface
  - PostProcessingResult dataclass
- Migrated post-processing components:
  - quality_evaluator.py
  - style_sanitizer.py

All abstract interfaces tested and working.

Next: Complete learning and post-processing migration

Author: anhmtk

docs/PHASE2_3_PROGRESS.md

@@ -0,0 +1,87 @@
+# Phase 2.6 & Phase 3 Progress Report
+
+## ✅ Step 2.6: Integration Completed
+
+### Unified Metrics Integration ✅
+- **Validation System**: Updated `stillme_core/validation/metrics.py` to forward metrics to `UnifiedMetricsCollector`
+- **RAG System**: Integrated unified metrics into `stillme_core/rag/rag_retrieval.py`
+  - Records retrieval events with avg_similarity, context_quality, retrieval_time_ms
+  - Tracks RAG performance metrics
+
+### Integration Status:
+- ✅ Validation metrics → UnifiedMetricsCollector
+- ✅ RAG metrics → UnifiedMetricsCollector
+- ⏳ Learning metrics → UnifiedMetricsCollector (pending - can be done when learning is migrated)
+
+## ✅ Phase 3: Learning & Post-Processing Started
+
+### Step 3.1: Abstract Learning Pipeline ✅
+- Created `stillme_core/learning/base.py` with:
+  - `LearningPipeline` abstract interface
+  - `LearningFetcher` abstract interface
+  - `LearningResult` dataclass
+- Defines contract for learning systems
+
+### Step 3.3: Abstract Post-Processing ✅
+- Created `stillme_core/postprocessing/base.py` with:
+  - `PostProcessor` abstract interface
+  - `PostProcessingResult` dataclass
+- Defines contract for post-processing systems
+
+### Step 3.4: Post-Processing Migration (Partial) ✅
+- Migrated `quality_evaluator.py` → `stillme_core/postprocessing/`
+- Migrated `style_sanitizer.py` → `stillme_core/postprocessing/`
+- Abstract interfaces ready for implementation
+
+## 📊 What Was Created
+
+### New Modules:
+1. **`stillme_core/learning/`** - Abstract learning pipeline
+   - `base.py` - Abstract interfaces
+   
+2. **`stillme_core/postprocessing/`** - Abstract post-processing
+   - `base.py` - Abstract interfaces
+   - `quality_evaluator.py` - Migrated
+   - `style_sanitizer.py` - Migrated
+
+### Integration:
+- ✅ Unified metrics in validation system
+- ✅ Unified metrics in RAG system
+- ✅ Abstract interfaces for learning and post-processing
+
+## ⏳ Remaining Work
+
+### Phase 3.2: Learning Migration (Pending)
+- Migrate `learning_scheduler.py` → `stillme_core/learning/scheduler.py`
+- Migrate fetchers (RSS, arXiv, etc.) → `stillme_core/learning/fetchers/`
+- Migrate `content_curator.py` → `stillme_core/learning/curator.py`
+- Implement `LearningPipeline` interface
+
+### Phase 3.4: Post-Processing Migration (Partial)
+- Migrate remaining post-processing components:
+  - `rewrite_llm.py`
+  - `rewrite_decision_policy.py`
+  - `rewrite_honesty.py`
+  - `rewrite_philosophical_depth.py`
+  - `optimizer.py`
+- Implement `PostProcessor` interface
+
+### Step 2.8: Learning Metrics Integration (Pending)
+- Integrate unified metrics into learning system
+- Record learning cycle metrics
+
+## 🎯 Success Criteria
+
+- ✅ Unified metrics integrated into validation
+- ✅ Unified metrics integrated into RAG
+- ✅ Abstract interfaces created for learning and post-processing
+- ✅ Some post-processing components migrated
+- ⏳ Full learning and post-processing migration (in progress)
+
+## 📝 Next Steps
+
+1. Complete learning migration (Phase 3.2)
+2. Complete post-processing migration (Phase 3.4)
+3. Integrate learning metrics (Step 2.8)
+4. Integration testing
+

stillme_core/learning/__init__.py

@@ -0,0 +1,16 @@
+"""
+StillMe Core Learning System
+
+Provides abstract learning pipeline for continuous knowledge acquisition.
+
+This module will be migrated from backend/learning/ and backend/services/
+"""
+
+from .base import LearningPipeline, LearningFetcher, LearningResult
+
+__all__ = [
+    "LearningPipeline",
+    "LearningFetcher",
+    "LearningResult",
+]
+

stillme_core/learning/base.py

@@ -0,0 +1,114 @@
+"""
+Abstract Learning Pipeline Interface
+
+Defines the interface for learning systems that can:
+- Fetch content from various sources
+- Filter and curate content
+- Add to knowledge base
+- Track learning metrics
+"""
+
+from abc import ABC, abstractmethod
+from typing import List, Dict, Optional, Any
+from dataclasses import dataclass
+from datetime import datetime
+
+
+@dataclass
+class LearningResult:
+    """Result of a learning cycle"""
+    cycle_number: int
+    timestamp: str  # ISO format
+    entries_fetched: int
+    entries_added: int
+    entries_filtered: int
+    sources: Dict[str, int]  # source -> entries_fetched
+    duration_seconds: float
+    error: Optional[str] = None
+    metadata: Optional[Dict[str, Any]] = None
+
+
+class LearningFetcher(ABC):
+    """
+    Abstract interface for content fetchers
+    
+    Implementations should fetch content from specific sources
+    (RSS, arXiv, CrossRef, Wikipedia, etc.)
+    """
+    
+    @abstractmethod
+    def fetch(self, limit: Optional[int] = None) -> List[Dict[str, Any]]:
+        """
+        Fetch content from source
+        
+        Args:
+            limit: Maximum number of entries to fetch (None = no limit)
+            
+        Returns:
+            List of content entries (dicts with title, content, source, etc.)
+        """
+        pass
+    
+    @abstractmethod
+    def get_source_name(self) -> str:
+        """Get name of the source"""
+        pass
+
+
+class LearningPipeline(ABC):
+    """
+    Abstract learning pipeline interface
+    
+    Defines the contract for learning systems that can:
+    1. Fetch content from multiple sources
+    2. Filter and curate content
+    3. Add curated content to knowledge base
+    4. Track metrics
+    """
+    
+    @abstractmethod
+    def run_learning_cycle(self) -> LearningResult:
+        """
+        Run a single learning cycle
+        
+        Returns:
+            LearningResult with cycle metrics
+        """
+        pass
+    
+    @abstractmethod
+    def get_fetchers(self) -> List[LearningFetcher]:
+        """
+        Get list of content fetchers
+        
+        Returns:
+            List of LearningFetcher instances
+        """
+        pass
+    
+    @abstractmethod
+    def filter_content(self, entries: List[Dict[str, Any]]) -> tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
+        """
+        Filter content entries
+        
+        Args:
+            entries: List of content entries to filter
+            
+        Returns:
+            Tuple of (accepted_entries, filtered_entries)
+        """
+        pass
+    
+    @abstractmethod
+    def add_to_knowledge_base(self, entries: List[Dict[str, Any]]) -> int:
+        """
+        Add entries to knowledge base
+        
+        Args:
+            entries: List of content entries to add
+            
+        Returns:
+            Number of entries successfully added
+        """
+        pass
+

stillme_core/postprocessing/__init__.py

@@ -0,0 +1,15 @@
+"""
+StillMe Core Post-Processing System
+
+Provides abstract post-processing pipeline for response quality improvement.
+
+This module will be migrated from backend/postprocessing/
+"""
+
+from .base import PostProcessor, PostProcessingResult
+
+__all__ = [
+    "PostProcessor",
+    "PostProcessingResult",
+]
+

stillme_core/postprocessing/base.py

@@ -0,0 +1,63 @@
+"""
+Abstract Post-Processing Interface
+
+Defines the interface for post-processing systems that can:
+- Evaluate response quality
+- Rewrite responses if needed
+- Sanitize style and content
+- Track post-processing metrics
+"""
+
+from abc import ABC, abstractmethod
+from typing import Optional, Dict, Any
+from dataclasses import dataclass
+
+
+@dataclass
+class PostProcessingResult:
+    """Result of post-processing"""
+    processed_text: str
+    quality_score: float
+    rewrite_attempted: bool = False
+    rewrite_successful: bool = False
+    processing_time_ms: float = 0.0
+    metadata: Optional[Dict[str, Any]] = None
+
+
+class PostProcessor(ABC):
+    """
+    Abstract post-processing interface
+    
+    Defines the contract for post-processing systems that can:
+    1. Evaluate response quality
+    2. Rewrite responses if needed
+    3. Sanitize style and content
+    """
+    
+    @abstractmethod
+    def process(self, text: str, context: Optional[Dict[str, Any]] = None) -> PostProcessingResult:
+        """
+        Process and improve response text
+        
+        Args:
+            text: Original response text
+            context: Optional context (question, validation results, etc.)
+            
+        Returns:
+            PostProcessingResult with processed text and metrics
+        """
+        pass
+    
+    @abstractmethod
+    def evaluate_quality(self, text: str) -> float:
+        """
+        Evaluate quality of response text
+        
+        Args:
+            text: Response text to evaluate
+            
+        Returns:
+            Quality score (0.0-1.0)
+        """
+        pass
+