feat: Phase 3 Complete - Learning & Post-Processing Migration

Phase 3.1: Abstract Learning Pipeline
- Created stillme_core/learning/base.py with LearningPipeline, LearningFetcher interfaces
- LearningResult dataclass for cycle results

Phase 3.2: Learning Components Migration
- Migrated content_curator.py → stillme_core/learning/curator.py
- Migrated learning_scheduler.py → stillme_core/learning/scheduler.py
- Created stillme_core/learning/fetchers/ with 4 fetchers (RSS, arXiv, CrossRef, Wikipedia)
- Updated backend/services/learning_scheduler.py to use stillme_core components
- Integrated unified metrics into learning scheduler

Phase 3.3: Abstract Post-Processing
- Created stillme_core/postprocessing/base.py with PostProcessor interface
- PostProcessingResult dataclass

Phase 3.4: Post-Processing Migration
- Migrated all 7 post-processing components to stillme_core/postprocessing/
- Updated exports in __init__.py files
- Created backward compatibility adapters

Step 2.8: Learning Metrics Integration
- Integrated unified metrics into learning scheduler
- Records cycle_number, entries_fetched, entries_added, entries_filtered, sources, duration, errors

Backward Compatibility:
- backend/services/learning_scheduler.py uses stillme_core components
- backend/services/content_curator.py forwards from stillme_core
- backend/postprocessing/__init__.py forwards from stillme_core
- All existing imports continue to work

All imports tested and working. No breaking changes.

Author: anhmtk

VALIDATION_CHAIN_DIAGRAM_PROMPT_V2.txt

@@ -0,0 +1,163 @@
+=== CRITICAL: COPY EXACT TEXT BELOW - DO NOT MODIFY OR INTERPRET ===
+
+Create a professional flowchart diagram with EXACT text as specified below.
+
+=== TITLE ===
+"StillMe Validation Chain: Multi-Layer Hallucination Reduction"
+Subtitle: "12+ Validators | Validation Pass Rate 93.9% | Zero Hallucination in Custom Tests"
+
+=== START ===
+Grey box: "Start: RAG Response"
+Arrow down
+
+=== MAIN VALIDATION BLOCK ===
+Grey container: "StillMe Validation Chain (12+ Validators)"
+
+LEFT COLUMN - Core Validators (GREEN boxes, exact text):
+
+1. Box text:
+   "LanguageValidator"
+   "Check language match"
+
+2. Box text:
+   "CitationRequired"
+   "Ensure citations are present"
+
+3. Box text:
+   "CitationRelevance"
+   "Validate citation quality"
+
+4. Box text:
+   "NumericUnitsBasic"
+   "Validate numeric/units claims"
+
+5. Box text:
+   "ConfidenceValidator"
+   "Detect when to express uncertainty"
+
+6. Box text:
+   "FactualHallucinationValidator"
+   "Detect factual errors"
+
+7. Box text:
+   "ReligiousChoiceValidator"
+   "Reject religion choice in responses"
+
+RIGHT COLUMN - Conditional Validators (GREEN boxes, exact text):
+
+8. Box text:
+   "EvidenceOverlap"
+   "Check content matches context"
+   "(Only when has context)"
+
+9. Box text:
+   "SourceConsensusValidator"
+   "Detect contradictions between sources"
+   "(Only when has 2+ sources)"
+
+10. Box text:
+    "EgoNeutralityValidator"
+    "Catch 'Hallucination of Experience'"
+    "(Only when has context)"
+
+11. Box text:
+    "IdentityCheckValidator"
+    "Prevent anthropomorphism"
+    "(If enabled)"
+
+12. Box text:
+    "PhilosophicalDepthValidator"
+    "Validate philosophical depth"
+    "(Only for philosophical questions)"
+
+13. Box text:
+    "EthicsAdapter"
+    "Ethical content filtering"
+
+=== ARROWS ===
+Connect validators with arrows as shown in flow.
+
+=== DECISION POINT 1 ===
+YELLOW diamond: "All Validations Passed?"
+
+IF YES → RIGHT arrow to:
+GREEN box: "Final Response with Citations + Transparency Metadata"
+Inside box (3 lines):
+- "93.9% Validation Pass Rate"
+- "Zero Hallucination in Custom Tests"
+- "91.1% Citation Rate (full) / 100% (subset)"
+
+IF NO → DOWN arrow to:
+
+=== DECISION POINT 2 ===
+YELLOW diamond: "Critical Failure?"
+
+IF YES → RIGHT arrow to:
+RED box: "Fallback Handler"
+Inside box (2 lines):
+- "Epistemic Fallback Response"
+- "Honest 'I don't know' answer"
+
+IF NO → DOWN arrow to:
+ORANGE box: "Response with Warning"
+Inside box (2 lines):
+- "Uncertainty + Validator Flags"
+- "Non-critical warnings"
+
+=== SPELLING CHECKLIST - VERIFY EVERY WORD ===
+✓ "LanguageValidator" (not "Language Validator" or "LanguageValidator")
+✓ "CitationRequired" (not "Citation Required" or "CitationRequire")
+✓ "CitationRelevance" (not "CitationReletance" or "Citation Relevance")
+✓ "NumericUnitsBasic" (not "NumericUnits" or "Numeric Units Basic")
+✓ "ConfidenceValidator" (not "Confidence Validator" or "Facfdence Validator")
+✓ "FactualHallucinationValidator" (not "FactualHallucination" or "Reject HalluchationValidator" or "Halluchation")
+✓ "ReligiousChoiceValidator" (not "Religious Choice Validator")
+✓ "EvidenceOverlap" (not "Evidence Overlap" or "EvidenceOverlapValidator")
+✓ "SourceConsensusValidator" (not "SourceConsneusValidator" or "Source Consensus Validator")
+✓ "EgoNeutralityValidator" (not "EgoNeutrality" or "Ego Neutrality Validator")
+✓ "IdentityCheckValidator" (not "IdentityCheriity Validator" or "Identity Check Validator")
+✓ "PhilosophicalDepthValidator" (not "PhilospocicalDepth Validator" or "Philosophical Depth Validator")
+✓ "EthicsAdapter" (not "EthicsAdapter" or "Ethics oment filterkng")
+
+✓ "anthropomorphism" (not "antrhoporrgrism" or "anthropophism")
+✓ "hallucination" (not "hailucination" or "Halluccination" or "Halluchation" or "Hallucation")
+✓ "uncertainty" (not "uncertianty" or "Uncertimity")
+✓ "Epistemic" (not "Epistettic")
+✓ "sources" (not "sontext" or "sourtest")
+✓ "philosophical" (not "phliosocnical")
+✓ "YES" (not "YÓ")
+✓ "NO" (not "NÓ")
+✓ "are present" (not "are presnt" or "are ptiee" or "arepresent")
+✓ "matches" (not "matthes")
+✓ "content" (not "conten" or "contet")
+
+=== EXACT PHRASES TO USE ===
+- "Check language match" (not "language are presnt")
+- "Ensure citations are present" (not "Ensure citations are ptiee")
+- "Validate citation quality" (not "validates numeric/units claims" - that's for NumericUnitsBasic)
+- "Validate numeric/units claims" (for NumericUnitsBasic only)
+- "Detect when to express uncertainty" (not "uncertianty")
+- "Detect factual errors" (not "Detect factual erros")
+- "Catch 'Hallucination of Experience'" (not "Hallucation" or "Halluchation")
+- "Prevent anthropomorphism" (not "antrhoporrgrism")
+- "Detect contradictions between sources" (not "between sontext" or "sourtest")
+- "Check content matches context" (not "Vallect content matthes context")
+- "Validate philosophical depth" (not "phliosocnical")
+- "Ethical content filtering" (not "Ethics oment filterkng")
+
+=== FINAL VERIFICATION ===
+Before generating, check:
+1. All validator names are EXACTLY as listed above
+2. All descriptions use EXACT phrases above
+3. NO typos in: hallucination, anthropomorphism, uncertainty, epistemic, philosophical, sources, matches, content
+4. Decision points say "YES" and "NO" (not "YÓ" or "NÓ")
+5. All percentages are correct: 93.9%, 91.1%, 100%
+6. "12+ Validators" (not "11" or "12 Validators")
+
+=== DESIGN ===
+- Professional, clean flowchart
+- Clear top-to-bottom flow
+- Color coding: GREY (start/container), GREEN (validators/success), YELLOW (decisions), RED (failures), ORANGE (warnings)
+- Arrows show flow direction clearly
+- Text is readable and properly spaced
+

arxiv/main_2.pdf

@@ -1,11 +1,33 @@
 """
 Post-processing modules for StillMe output normalization and quality enforcement
+
+⚠️ MIGRATION NOTE: This module is being migrated to stillme_core.postprocessing.
+During migration, imports are forwarded from stillme_core.postprocessing for backward compatibility.
 """
 
-from backend.postprocessing.style_sanitizer import StyleSanitizer
-from backend.postprocessing.quality_evaluator import QualityEvaluator
-from backend.postprocessing.rewrite_llm import RewriteLLM
-from backend.postprocessing.optimizer import PostProcessingOptimizer
+# During migration: Forward imports from stillme_core.postprocessing
+try:
+    from stillme_core.postprocessing import (
+        PostProcessor,
+        PostProcessingResult,
+        QualityEvaluator,
+        StyleSanitizer,
+        RewriteLLM,
+        PostProcessingOptimizer,
+    )
+except ImportError:
+    # Fallback to local imports if stillme_core is not available yet
+    from .style_sanitizer import StyleSanitizer
+    from .quality_evaluator import QualityEvaluator
+    from .rewrite_llm import RewriteLLM
+    from .optimizer import PostProcessingOptimizer
 
-__all__ = ['StyleSanitizer', 'QualityEvaluator', 'RewriteLLM', 'PostProcessingOptimizer']
+__all__ = [
+    'PostProcessor',
+    'PostProcessingResult',
+    'StyleSanitizer',
+    'QualityEvaluator',
+    'RewriteLLM',
+    'PostProcessingOptimizer'
+]
 

backend/postprocessing/__init__.py

@@ -3,8 +3,18 @@
 Prioritizes and optimizes learning content based on quality and effectiveness
 Includes Pre-Filter mechanism to reduce costs by filtering before embedding
 Now includes ReviewAdapter for simulated peer review evaluation
+
+⚠️ MIGRATION NOTE: This module is being migrated to stillme_core.learning.curator.
+During migration, imports are forwarded from stillme_core.learning.curator for backward compatibility.
 """
 
+# During migration: Forward imports from stillme_core.learning.curator
+try:
+    from stillme_core.learning.curator import ContentCurator
+except ImportError:
+    # Fallback to local implementation if stillme_core is not available yet
+    import os
+
 from typing import List, Dict, Any, Optional, Tuple
 import logging
 from datetime import datetime

backend/services/content_curator.py

@@ -2,6 +2,9 @@
 Learning Scheduler Service for StillMe
 Automatically runs learning cycles every N hours to fetch content and add to RAG
 CRITICAL: This is a core feature - must always be enabled and running
+
+⚠️ MIGRATION NOTE: This module is being migrated to stillme_core.learning.scheduler.
+During migration, this file serves as fallback. New code should use stillme_core.learning.scheduler.
 """
 
 import asyncio
@@ -64,12 +67,12 @@ def __init__(
 
         if source_integration is None:
             from backend.services.source_integration import SourceIntegration
-            from backend.services.content_curator import ContentCurator
+            from stillme_core.learning.curator import ContentCurator
             content_curator = content_curator or ContentCurator()
             source_integration = SourceIntegration(content_curator=content_curator)
 
         if content_curator is None:
-            from backend.services.content_curator import ContentCurator
+            from stillme_core.learning.curator import ContentCurator
             content_curator = ContentCurator()
 
         if rss_fetch_history is None:
@@ -234,6 +237,22 @@ async def run_learning_cycle(self) -> Dict[str, Any]:
         except Exception as e:
             logger.error(f"❌ Error in learning cycle #{cycle_number}: {e}", exc_info=True)
 
+            # Record error in unified metrics
+            try:
+                from stillme_core.monitoring import get_metrics_collector, MetricCategory
+                unified_metrics = get_metrics_collector()
+                unified_metrics.record_learning_cycle(
+                    cycle_number=cycle_number,
+                    entries_fetched=0,
+                    entries_added=0,
+                    entries_filtered=0,
+                    sources={},
+                    duration_seconds=(datetime.now() - start_time).total_seconds(),
+                    error=str(e)
+                )
+            except Exception as metrics_error:
+                logger.debug(f"Failed to record learning error metrics: {metrics_error}")
+            
             # Don't increment cycle count on error
             return {
                 "cycle_number": 0,  # 0 indicates error

backend/services/learning_scheduler.py

@@ -0,0 +1,16 @@
+"""
+Adapter for LearningScheduler - Backward Compatibility
+
+⚠️ MIGRATION NOTE: LearningScheduler has been migrated to stillme_core.learning.scheduler.
+This adapter forwards imports for backward compatibility.
+"""
+
+# Forward import from stillme_core
+try:
+    from stillme_core.learning.scheduler import LearningScheduler
+except ImportError:
+    # Fallback to local import if stillme_core is not available
+    from backend.services.learning_scheduler import LearningScheduler
+
+__all__ = ["LearningScheduler"]
+