Phase 3: End-to-End Samples & Integration

[elbruno]

Overview

Build comprehensive sample applications and updated documentation that demonstrate Phase 3 features (connectors, security, evaluation) in realistic end-to-end workflows. Enable users to quickly integrate the full pipeline.

Goals

• Demonstrate multi-source document ingestion
• Show security policy application
• Illustrate evaluation and benchmarking
• Provide copy-paste-ready code samples
• Update core walkthroughs with Phase 3 features

Acceptance Criteria

1. ConnectorsDemo Sample

• Location: src/samples/ConnectorsDemo/
• Purpose: Show FileSystemConnector and AzureBlobConnector usage
• Workflow:
  1. Define FileSystemConnector (scan C:\Documents)
  2. Define AzureBlobConnector (connect to Azure Blob Storage)
  3. Merge both sources
  4. Convert documents from both sources
  5. Display summary (count, formats, output paths)
• Code Structure:

  // 1. Setup connectors
  var fsConnector = new FileSystemConnector(new()
  {
      RootPath = "./documents",
      Patterns = ["*.pdf", "*.docx"],
      MaxDepth = 2
  });
  
  var blobConnector = new AzureBlobConnector(new()
  {
      AccountName = "myaccount",
      ContainerName = "documents"
  });
  
  // 2. Enumerate and convert
  var services = new ServiceCollection()
      .AddMarkItDotNetCore()
      .AddMarkItDotNetConnectors()
      .BuildServiceProvider();
  
  var converter = services.GetRequiredService<IMarkdownConverter>();
  var processed = 0;
  
  await foreach (var doc in fsConnector)
  {
      var result = await converter.ConvertAsync(doc.Content, doc.MimeType);
      File.WriteAllText($"output/{doc.Id}.md", result.Markdown);
      processed++;
  }
  
  Console.WriteLine($"Processed {processed} documents");

• Features:
  • Command-line args for paths, patterns
  • Progress reporting
  • Error handling and logging
  • Output validation

2. SecurityPoliciesDemo Sample

• Location: src/samples/SecurityPoliciesDemo/
• Purpose: Demonstrate PII detection, content policies, guardrails
• Workflow:
  1. Load a document with sensitive data
  2. Apply PII detection (find emails, phone, SSN)
  3. Apply content policy (block certain keywords)
  4. Apply guardrails (size/page limits)
  5. Export redacted markdown
  6. Generate audit log
• Code Structure:

  var converter = services.GetRequiredService<IMarkdownConverter>();
  var policyChain = new SecurityPolicyChain(
      new PiiDetector(new()
      {
          RedactionStrategy = RedactionStrategy.Mask
      }),
      new ContentPolicyEngine(new()
      {
          DenyKeywords = ["confidential", "secret"]
      }),
      new GuardrailsPolicy(new()
      {
          MaxFileSize = 50_000_000,
          MaxPageCount = 100
      })
  );
  
  // Convert then evaluate policies
  var result = await converter.ConvertAsync(doc);
  var policyResult = await policyChain.EvaluateAsync(result.Markdown);
  
  if (!policyResult.Passed)
  {
      Console.WriteLine($"Policy violations: {policyResult.Violations.Count}");
      foreach (var v in policyResult.Violations)
          Console.WriteLine($"  - {v.RuleName}: {v.Message}");
      
      // Use redacted content instead
      result = result with { Markdown = policyResult.RedactedContent };
  }
  
  File.WriteAllText("output/redacted.md", result.Markdown);

• Features:
  • Multiple policy chains
  • Before/after comparison
  • Violation report
  • Audit log generation

3. EvaluationDemo Sample

• Location: src/samples/EvaluationDemo/
• Purpose: Run benchmarks and compare chunking strategies
• Workflow:
  1. Load benchmark corpus
  2. Run all chunking strategies on sample documents
  3. Collect metrics (latency, memory, quality)
  4. Generate comparison report
  5. Export results to CSV
  6. Print recommendations
• Code Structure:

  var runner = new BenchmarkSuiteRunner(new()
  {
      CorpusPath = "./benchmarks/corpus",
      EnableMemoryProfiling = true,
      TimeoutSeconds = 300,
      ExportFormats = new[] { "json", "csv" }
  });
  
  var report = await runner.RunAsync();
  
  Console.WriteLine($"Evaluated {report.Results.Count} documents");
  Console.WriteLine($"\nRecommended strategy: {report.RecommendedStrategy}");
  
  foreach (var (strategy, metrics) in report.Results.First().Value)
  {
      Console.WriteLine($"\n{strategy}:");
      Console.WriteLine($"  Chunks: {metrics.ChunkCount}");
      Console.WriteLine($"  Latency: {metrics.ProcessingTimeMs}ms");
      Console.WriteLine($"  Coherence: {metrics.SemanticCoherence:P}");
  }
  
  // Export for analysis
  await report.ExportAsync("reports/comparison.csv", ExportFormat.Csv);

• Features:
  • Corpus validation
  • Progress reporting
  • Detailed metrics output
  • Export to multiple formats

4. IngestionWorkflow Sample

• Location: src/samples/IngestionWorkflow/
• Purpose: Full end-to-end pipeline combining all Phase 3 components
• Workflow:
  1. Setup multi-source connectors (FileSystem + Azure Blob)
  2. Apply security policies (PII detection, guardrails)
  3. Convert documents
  4. Chunk with best strategy (from evaluation)
  5. Generate citations
  6. Upload to Azure Search
  7. Report metrics
• Code Structure:

  // Phase 1: Connectors
  var connectors = new[]
  {
      new FileSystemConnector(/* ... */),
      new AzureBlobConnector(/* ... */)
  };
  
  // Phase 2: Security
  var policies = new SecurityPolicyChain(/* ... */);
  
  // Phase 3: Conversion & Chunking
  var converter = services.GetService<IMarkdownConverter>();
  var chunker = new TokenAwareChunker(new() { TokenLimit = 512 });
  
  // Phase 4: Citations & Quality
  var citations = new CitationBuilder();
  
  // Phase 5: Upload
  var searchUploader = services.GetService<SearchIndexUploader>();
  
  // Execute pipeline
  int processed = 0, failed = 0;
  var sw = Stopwatch.StartNew();
  
  foreach (var connector in connectors)
  {
      await foreach (var doc in connector)
      {
          try
          {
              // Convert
              var converted = await converter.ConvertAsync(doc.Content, doc.MimeType);
  
              // Apply policies
              var policyResult = await policies.EvaluateAsync(converted.Markdown);
              if (!policyResult.Passed)
              {
                  converted = converted with { Markdown = policyResult.RedactedContent };
              }
  
              // Chunk
              var chunks = await chunker.ChunkAsync(converted.Markdown);
  
              // Add citations
              foreach (var chunk in chunks)
              {
                  chunk.Metadata["citations"] = citations.BuildFromContent(chunk.Content);
              }
  
              // Upload
              await searchUploader.UploadAsync(chunks);
  
              processed++;
          }
          catch (Exception ex)
          {
              Console.WriteLine($"Error processing {doc.Id}: {ex.Message}");
              failed++;
          }
      }
  }
  
  sw.Stop();
  Console.WriteLine($"Pipeline complete: {processed} succeeded, {failed} failed in {sw.ElapsedMilliseconds}ms");

• Features:
  • Error handling and recovery
  • Progress tracking
  • Timing metrics
  • Dry-run mode (no uploads)
  • Configurable via appsettings.json

5. Documentation Updates

docs/ingestion-workflows.md (New/Updated)

• Sections:
  1. Quick Start: Basic multi-source ingestion
  2. Security Best Practices: Policy chains, audit logging
  3. Strategy Comparison: Choosing chunking strategy
  4. Integration Patterns: With Azure Search, Copilot
  5. Troubleshooting: Common issues and recovery
• Code Examples: Copy-paste-ready snippets from samples above

docs/connectors-guide.md (New)

• Using FileSystemConnector
• Using AzureBlobConnector
• Creating custom connectors (extending IDocumentSource)
• Error handling and retries

docs/security-policies-guide.md (New)

• PII detection and redaction patterns
• Content policy rules and conditions
• Guardrails configuration
• Audit logging and compliance

docs/evaluation-guide.md (New)

• Running benchmarks
• Comparing strategies
• Interpreting metrics
• Exporting results
• Baseline management

6. Tests

• File Structures:
  • src/samples/ConnectorsDemo.Tests/
  • src/samples/SecurityPoliciesDemo.Tests/
  • src/samples/EvaluationDemo.Tests/
  • src/samples/IngestionWorkflow.Tests/
• Coverage:
  • Sample execution end-to-end
  • Configuration variations
  • Error scenarios
  • Output validation

7. README Updates

• Root README.md:
  • Add Phase 3 feature highlights
  • Links to new samples
  • Quick start for end-to-end pipeline
• Sample README files:
  • Each sample has dedicated README with setup, config, execution steps

Integration Checklist

• All samples run successfully from command line
• All samples produce expected output
• Documentation examples match sample code
• Error messages are helpful
• Configuration via appsettings.json works
• Dry-run modes work (no side effects)
• Logging output is clear and actionable

Related Packages

• ElBruno.MarkItDotNet (core)
• ElBruno.MarkItDotNet.Connectors (Phase 3)
• ElBruno.MarkItDotNet.Security.Policies (Phase 3)
• ElBruno.MarkItDotNet.Benchmarking (Phase 3)
• ElBruno.MarkItDotNet.Sync (orchestration)
• ElBruno.MarkItDotNet.AzureSearch (destination)

Labels

Phase 3, samples, documentation, integration, end-to-end