Phase 3: Evaluation Tooling & Benchmark Suite

[elbruno]

Overview

Build comprehensive evaluation and benchmarking infrastructure to measure conversion quality, compare chunking strategies, and track performance metrics. Enable data-driven optimization decisions.

Goals

• Create BenchmarkCorpus with diverse test documents
• Implement strategy comparison and ranking system
• Track citation coverage and accuracy metrics
• Measure performance (latency, memory, throughput)
• Generate exportable evaluation reports

Acceptance Criteria

1. BenchmarkCorpus (New Package: ElBruno.MarkItDotNet.Benchmarking)

• Purpose: Centralized collection of test documents with ground truth
• Structure:
  • src/benchmarks/corpus/: File storage
  • Categories: PDFs, Word docs, spreadsheets, HTML, mixed-format files
  • Size distribution: small (1KB), medium (1MB), large (10MB+)
  • Metadata: expected output samples, known issues, quality baseline
• Content (50+ files minimum):
  • 10 PDFs (text, scanned, mixed)
  • 10 DOCX files (simple, complex formatting, tables, images)
  • 10 XLSX files (simple, complex formulas, charts)
  • 5 HTML files (various structures, CSS)
  • 5 CSV/JSON/YAML files
  • 5 mixed-format documents (EPUB, RTF, etc.)
  • 5+ edge cases (password-protected, corrupted headers, encoding issues)
• Corpus Manifest (JSON):

  {
    "documents": [
      {
        "id": "pdf-001",
        "filename": "sample.pdf",
        "format": "pdf",
        "size": 2048576,
        "expectedMarkdownLines": 250,
        "expectedHeadings": 5,
        "tags": ["text-extraction", "tables"],
        "groundTruth": "corpus/pdf-001-expected.md"
      }
    ]
  }

2. StrategyComparisonReport

• Class: StrategyComparisonEvaluator
• Compares: HeadingBasedChunker vs ParagraphBasedChunker vs TokenAwareChunker
• Metrics per Strategy:
  • Chunk count
  • Avg chunk size (tokens, characters)
  • Semantic coherence score (0-1)
  • Overlap percentage
  • Processing time (ms)
• Output (Report):

  public record StrategyComparisonReport(
      string DocumentId,
      Dictionary<string, StrategyMetrics> Results,
      string RecommendedStrategy,
      double ConfidenceScore
  );
  
  public record StrategyMetrics(
      int ChunkCount,
      double AvgChunkSizeTokens,
      double AvgChunkSizeChars,
      double SemanticCoherence,
      double OverlapPercent,
      long ProcessingTimeMs
  );

• Ranking: Best strategy by composite score (coherence + efficiency - overlap)

3. CitationCoverageEvaluator

• Purpose: Track citation accuracy and retention through pipeline
• Metrics:
  • Citation retention rate: (chunks with citations / total chunks) * 100%
  • Citation accuracy: (correct source references / total citations) * 100%
  • Citation propagation through chunking
  • Lost citations analysis (before/after)
• Output:

  public record CitationCoverageReport(
      string DocumentId,
      int TotalCitations,
      int ChunksWithCitations,
      double RetentionRate,
      double AccuracyRate,
      List<LostCitation> LostCitations
  );
  
  public record LostCitation(
      string SourceReference,
      string Reason // "trimmed_by_chunker", "converted_incorrectly", etc.
  );

4. PerformanceMetrics

• Class: PerformanceMetricsCollector
• Metrics (per document):
  • Latency: Conversion time (ms), chunking time (ms), total time (ms)
  • Memory: Peak RSS, GC collections, heap allocations (bytes)
  • Throughput: Documents/second, MB/second
  • Quality: Citation retention, semantic coherence, heading density
• Aggregations (across corpus):
  • Percentiles: p50, p95, p99 latency
  • Memory high-water marks
  • Format-specific baselines
• Output Format:

  {
    "timestamp": "2026-06-10T14:30:00Z",
    "corpus": "main-50",
    "results": [
      {
        "documentId": "pdf-001",
        "format": "pdf",
        "latencyMs": 1234,
        "memoryMb": 45.2,
        "throughputMbPerSec": 2.1,
        "qualityScore": 0.92
      }
    ],
    "aggregates": {
      "avgLatencyMs": 1100,
      "p95LatencyMs": 2500,
      "avgMemoryMb": 40
    }
  }

• Export Formats: JSON, CSV (for Excel/analysis)

5. EvaluationRunner

• Class: BenchmarkSuiteRunner
• Workflow:
  1. Load corpus manifest
  2. For each document:
     • Convert using all chunking strategies
     • Collect metrics (latency, memory, quality)
     • Evaluate citation coverage
  3. Generate comparison report
  4. Export results to JSON/CSV
• Configuration:

  var runner = new BenchmarkSuiteRunner(new()
  {
      CorpusPath = "benchmarks/corpus",
      EnableMemoryProfiling = true,
      TimeoutSeconds = 300,
      ExportFormats = new[] { "json", "csv" }
  });
  var report = await runner.RunAsync();

6. Baseline Tracking

• File: benchmarks/baselines.json
• Content:

  {
    "version": "0.7.1",
    "date": "2026-06-10",
    "metrics": {
      "pdf-001": {
        "latencyMs": 1200,
        "memoryMb": 42,
        "citationRetention": 0.98
      }
    }
  }

• Compare Current vs Baseline: Flag regressions (>5% variance)

7. Tests

• File Structure: src/tests/ElBruno.MarkItDotNet.Benchmarking.Tests/
• Coverage: 30+ tests
  • Corpus loading and validation
  • Strategy comparison logic
  • Citation coverage calculation
  • Metrics aggregation and percentiles
  • Report generation and export
  • Regression detection

8. Integration with CI

• New Workflow: .github/workflows/benchmark.yml (optional, triggered manually or nightly)
• Steps:
  1. Run BenchmarkSuiteRunner
  2. Export CSV results
  3. Compare against baseline
  4. Warn if regressions detected
  5. Commit updated baseline to repo

Sample Reports

• benchmarks/reports/latest.json (after each run)
• benchmarks/reports/report-2026-06-10.csv (timestamped exports)

Related Packages

• ElBruno.MarkItDotNet.Chunking (strategies to compare)
• ElBruno.MarkItDotNet.Citations (coverage tracking)
• ElBruno.MarkItDotNet.Quality (quality scoring)

Labels

Phase 3, evaluation, benchmarking, quality, metrics