[FEATURE] Enhance Error Context Propagation with Structured Logging and User-Friendly Error Messages

[smirk-dev]

Problem Statement

While exploring the CocoIndex codebase, I noticed that error handling and propagation from the Rust core to Python SDK could be significantly improved to provide better developer experience, especially for debugging data transformation pipelines. Currently, when errors occur during flow execution, users may not receive sufficient context about:

1. Which transformation step failed in the dataflow pipeline
2. What input data caused the error (especially important for batch processing)
3. Stack traces from both Rust and Python sides for hybrid errors
4. Actionable suggestions for resolving common errors

Proposed Solution

I propose implementing a comprehensive error context enhancement system with the following components:

1. Structured Error Context in Rust Core

• Add a ErrorContext struct that captures:
  • Flow name and step identifier
  • Source/function/target being executed
  • Input data sample (sanitized for PII)
  • Timestamp and execution phase
• Implement context propagation through all operator chains
• Add breadcrumb trail for nested transformations

2. Enhanced Python SDK Error Classes

Create specialized exception classes:

class CocoIndexError(Exception):
    """Base exception with enhanced context"""
    def __init__(self, message, context=None, suggestions=None):
        super().__init__(message)
        self.context = context or {}
        self.suggestions = suggestions or []
    
    def __str__(self):
        # Format with context and suggestions
        pass

class TransformationError(CocoIndexError):
    """Error during data transformation"""
    pass

class SourceError(CocoIndexError):
    """Error reading from data source"""
    pass

class TargetError(CocoIndexError):
    """Error writing to target"""
    pass

3. Rust-Python Error Bridge

• Implement anyhow::Context wrapper for all user-facing operations
• Add JSON serialization for error context across FFI boundary
• Preserve Rust stack traces in Python exceptions when possible
• Map common Rust error types to Python exception types

4. User-Friendly Error Messages

Provide actionable error messages:

TransformationError: Failed to embed text in chunk 1523 of document "user_manual.md"

Context:
  • Flow: TextEmbedding
  • Step: chunk["embedding"] = chunk["text"].transform(SentenceTransformerEmbed(...))
  • Input text (truncated): "Getting started with CocoIndex..."
  • Timestamp: 2025-10-04T14:23:45Z

Possible causes:
  1. Model not found - ensure 'sentence-transformers/all-MiniLM-L6-v2' is installed
  2. Out of memory - try reducing batch size
  3. Network timeout - check your internet connection

Suggestions:
  • Run: pip install sentence-transformers
  • Check available memory: cocoindex doctor --memory
  • See docs: https://cocoindex.io/docs/ai/embeddings#troubleshooting

5. Diagnostic CLI Command

Add a cocoindex doctor command to help diagnose common issues:

• Check system resources (memory, disk space)
• Verify dependencies and versions
• Test connectivity to common sources/targets
• Validate configuration files

Implementation Plan

Phase 1: Core Infrastructure (Week 1-2)

• Define ErrorContext struct in Rust
• Implement context propagation in core engine
• Add unit tests for error context

Phase 2: Python Integration (Week 2-3)

• Create enhanced exception classes
• Implement Rust-Python error bridge
• Add integration tests

Phase 3: User Experience (Week 3-4)

• Design error message templates
• Implement suggestion engine for common errors
• Create error code catalog
• Add documentation

Phase 4: Diagnostics (Week 4-5)

• Implement cocoindex doctor command
• Add system checks
• Create troubleshooting guide

Benefits

1. Improved Developer Experience: Developers can quickly identify and fix issues in their data pipelines
2. Reduced Support Burden: Clear error messages with actionable suggestions reduce support requests
3. Better Production Debugging: Enhanced context helps diagnose issues in production environments
4. Onboarding Friendly: New users get helpful guidance when they encounter errors
5. Aligns with Project Goals: Supports "Exceptional developer velocity" and "Production-ready at day 0"

References to Existing Issues

This proposal addresses and extends several existing issues:

• Enhance error message population from Rust side to Python SDK #1060 - Enhance error message population from Rust side to Python SDK
• Attach context in the error population chain (context() or with_context()) whenever a user-configured op (source, function, target) is involved #1061 - Attach context in error population chain
• Differentiate API error and internal error (caused by internal components), always carry Rust stack trace for internal error #1062 - Differentiate API error and internal error

Technical Considerations

• Performance: Error context should have minimal overhead in the happy path
• Privacy: Sanitize PII from error context (especially data samples)
• I18n: Design with internationalization in mind for future multi-language support
• Backward Compatibility: Maintain compatibility with existing error handling

Example Usage

After implementation, developers would see:

try:
    flow_builder.run()
except cocoindex.TransformationError as e:
    print(e)  # User-friendly formatted error with context
    print(e.context)  # Structured context dict for programmatic access
    print(e.suggestions)  # List of actionable suggestions
    
    # Log structured error for monitoring
    logger.error("Flow failed", extra=e.context)

Related Work

• Rust: anyhow, thiserror, eyre crates for error handling patterns
• Python: rich library for beautiful error formatting
• Similar projects: dbt (data build tool) has excellent error messages we can learn from

Questions for Maintainers

1. Would you prefer using an existing Rust error crate like eyre with custom hooks, or build a custom solution?
2. Should error context be configurable (e.g., verbose vs. minimal modes)?
3. Are there specific error scenarios you see frequently that should be prioritized?

---

I'm eager to work on this enhancement for Hacktoberfest! I have experience with both Rust and Python error handling systems and would love to contribute to CocoIndex's developer experience.

Please let me know if you'd like me to adjust the scope or focus on specific aspects of this proposal. I'm happy to start with Phase 1 and iterate based on feedback.

⭐ Starred the repo - excited to contribute to this amazing project!