diff --git a/.github/README.md b/.github/README.md index ab87e86..8f7d9e4 100644 --- a/.github/README.md +++ b/.github/README.md @@ -1,469 +1,22 @@ -# GitHub Repository Configuration +# GitHub Automation and Community Config -This directory contains all GitHub-specific configuration files including workflows, templates, and automation rules. +This directory holds repository automation and contribution metadata for StreamForge. ---- +User-facing docs live here instead: +- [../README.md](../README.md) for the project overview +- [../docs/index.md](../docs/index.md) for the published documentation site -## 📁 Directory Structure +## What Lives Here -``` -.github/ -├── workflows/ # GitHub Actions workflows -│ ├── ci.yml # Main CI pipeline (tests, builds, security) -│ ├── docker.yml # Docker image builds -│ ├── release.yml # Release automation -│ ├── pr-checks.yml # Pull request validation checks -│ ├── auto-label.yml # Automatic labeling -│ └── stale.yml # Stale issue/PR management -├── ISSUE_TEMPLATE/ # Issue templates -│ ├── bug_report.md # Bug report template -│ └── feature_request.md # Feature request template -├── CODEOWNERS # Code ownership and review assignment -├── labeler.yml # File-based auto-labeling rules -├── pull_request_template.md # PR template -├── BRANCH_PROTECTION.md # Branch protection setup guide -└── README.md # This file -``` +- `workflows/` - CI, Docker, release, PR checks, and Pages deploy +- `ISSUE_TEMPLATE/` - issue intake templates +- `CODEOWNERS` - review ownership rules +- `labeler.yml` - file-based label mapping +- `pull_request_template.md` - PR checklist and structure +- `BRANCH_PROTECTION.md` - branch policy notes ---- +## Maintainer Notes -## 🔧 Workflows - -### CI/CD Workflows - -#### `ci.yml` - Continuous Integration -**Triggers:** Push to `main`, Pull requests to `main` - -**Jobs:** -- **Rust Tests** - Run all tests with coverage -- **Rust Build** - Multi-platform builds (Linux, macOS, Windows) -- **Rust Security** - Security audit with cargo-audit -- **Rust Benchmarks** - Build and validate benchmarks -- **Operator Tests** - Kubernetes operator tests -- **UI Tests** - Frontend tests and linting -- **Helm Validation** - Chart linting and templating - -**Status:** ✅ Required for merge - -#### `docker.yml` - Docker Builds -**Triggers:** Push to `main`, tags - -**Jobs:** -- Build and push Docker images -- Multi-architecture support (amd64, arm64) - -**Status:** ⚠ī¸ Optional (for releases) - -#### `release.yml` - Release Automation -**Triggers:** Push tags (`v*`) - -**Jobs:** -- Create GitHub release -- Build release artifacts -- Publish to crates.io -- Push Docker images - -**Status:** ⚠ī¸ Release only - -### Pull Request Workflows - -#### `pr-checks.yml` - PR Validation -**Triggers:** Pull request events - -**Checks:** -1. **PR Metadata** - Validates PR title follows conventional commits -2. **PR Size** - Warns on large PRs (>500 lines), fails on very large (>1500 lines) -3. **Dependency Review** - Scans for vulnerable dependencies -4. **Label Requirements** - Ensures PR has required labels -5. **Documentation Check** - Warns if code changes without doc updates -6. **Changelog Check** - Warns if CHANGELOG.md not updated -7. **Breaking Changes** - Flags breaking changes for extra review -8. **Security Review** - Flags security-sensitive code changes - -**Status:** ✅ Required for merge (most checks) - -#### `auto-label.yml` - Automatic Labeling -**Triggers:** PR/Issue opened or updated - -**Functions:** -- Labels PRs based on changed files (using `labeler.yml`) -- Adds size labels (XS, S, M, L, XL) -- Labels issues based on content -- Welcomes first-time contributors - -**Status:** ℹī¸ Informational - -#### `stale.yml` - Stale Management -**Triggers:** Daily schedule, manual - -**Functions:** -- Marks inactive issues stale after 60 days -- Closes stale issues after 14 days -- Marks inactive PRs stale after 30 days -- Closes stale PRs after 7 days -- Respects exempt labels (pinned, security, etc.) - -**Status:** ℹī¸ Maintenance - ---- - -## 📋 Templates - -### Issue Templates - -#### Bug Report (`ISSUE_TEMPLATE/bug_report.md`) -**Use When:** Reporting a bug or unexpected behavior - -**Includes:** -- Expected vs actual behavior -- Steps to reproduce -- Environment information -- Configuration details - -#### Feature Request (`ISSUE_TEMPLATE/feature_request.md`) -**Use When:** Suggesting new features or enhancements - -**Includes:** -- Problem description -- Proposed solution -- Alternatives considered -- Implementation ideas - -### Pull Request Template - -**Location:** `pull_request_template.md` - -**Sections:** -- Description and type of change -- Related issues -- Changes made -- Testing performed -- Configuration examples -- Checklist (style, tests, docs) -- Performance impact - -**Required Fields:** -- Type of change (bug fix, feature, etc.) -- Testing checklist -- Documentation updates - ---- - -## đŸ‘Ĩ Code Ownership - -### CODEOWNERS File - -Automatically requests reviews from designated owners when PRs touch specific files. - -**Current Owners:** -- **Global:** @rahulbsw (all files) -- **Critical Files:** Security, configuration, CI/CD -- **Components:** Core, Kafka, DSL, Operator, UI - -**How It Works:** -1. PR touches files in owned paths -2. GitHub automatically requests review from owner -3. PR cannot merge without owner approval (if required) - -**Adding Owners:** -``` -/path/to/code @username @org/team-name -``` - ---- - -## 🏷ī¸ Labels - -### Automatic Labels - -Configured in `labeler.yml`, automatically applied based on: - -#### File-Based Labels -- `area/*` - Code area (core, kafka, dsl, config, security) -- `component/*` - Component (operator, ui, helm, docker) -- `documentation` - Doc changes -- `dependencies` - Dependency updates -- `ci/cd` - CI/CD changes -- `tests` - Test changes -- `benchmarks` - Benchmark changes - -#### Size Labels -- `size/XS` - < 10 lines -- `size/S` - < 50 lines -- `size/M` - < 200 lines -- `size/L` - < 500 lines -- `size/XL` - 500+ lines - -#### Branch-Based Labels -- `version/patch` - Bug fixes (fix/*, hotfix/*) -- `version/minor` - Features (feature/*, feat/*) -- `version/major` - Breaking changes (breaking/*, major/*) - -### Manual Labels (Require Reviewer Action) - -#### Type Labels (Required - at least one) -- `bug` - Bug fixes -- `enhancement` - New features -- `documentation` - Documentation only -- `maintenance` - Refactoring, cleanup -- `dependencies` - Dependency updates - -#### Priority Labels -- `priority/critical` - Production incidents, security issues -- `priority/high` - Important, blocking work -- `priority/medium` - Normal priority -- `priority/low` - Nice to have, backlog - -#### Status Labels -- `needs-review` - Awaiting review -- `needs-changes` - Changes requested -- `approved` - Ready to merge -- `on-hold` - Blocked or paused -- `wip` - Work in progress (don't merge) - -#### Special Labels -- `breaking-change` - Breaking API/config changes -- `security` - Security-related changes -- `performance` - Performance improvements -- `good-first-issue` - Good for newcomers -- `help-wanted` - Looking for contributors -- `pinned` - Never mark as stale -- `blocked` - Waiting on external dependency -- `skip-changelog` - Don't require changelog update - ---- - -## 🛡ī¸ Branch Protection - -**See:** [`BRANCH_PROTECTION.md`](BRANCH_PROTECTION.md) for complete setup guide - -### Quick Summary - -#### Main Branch (`main`) - -**Required Checks:** -- ✅ At least 1 approval -- ✅ All CI tests pass -- ✅ Conversations resolved -- ✅ Signed commits -- ✅ Linear history -- ✅ Up-to-date with base -- ✅ Code owner review - -**Restrictions:** -- ❌ No force pushes -- ❌ No deletions -- ❌ No bypassing rules - -**Allowed Merge Types:** -- ✅ Squash merge (recommended) -- ✅ Rebase merge -- ❌ Merge commits (disabled) - -### Protected Patterns - -| Pattern | Protection | Use | -|---------|-----------|-----| -| `main` | Full | Production code | -| `release/*` | Full | Release branches | -| `hotfix/*` | Medium | Emergency fixes | - ---- - -## 🚀 Workflow Usage - -### For Contributors - -#### Creating a Pull Request - -1. **Branch Naming:** - ```bash - feature/add-new-filter # New features - fix/kafka-connection # Bug fixes - docs/update-readme # Documentation - perf/optimize-dsl # Performance - ``` - -2. **Commit Messages:** - ```bash - # Follow conventional commits - feat: add regex filter support - fix: resolve kafka connection timeout - docs: update configuration guide - perf: optimize filter evaluation - ``` - -3. **Creating PR:** - - Fill out PR template completely - - Link related issues - - Add appropriate labels (automated + manual) - - Request reviews from code owners - - Ensure all checks pass - -#### During Review - -1. **Address feedback:** - - Mark conversations as resolved - - Push new commits (don't force push) - - Re-request review when ready - -2. **Keep PR updated:** - - Rebase or merge main regularly - - Resolve conflicts promptly - -3. **Before merging:** - - All checks green ✅ - - Approved by required reviewers ✅ - - Conversations resolved ✅ - - CHANGELOG updated (or skip-changelog label) ✅ - -### For Maintainers - -#### Reviewing Pull Requests - -1. **Automated checks:** - - Wait for CI to complete - - Review any warnings/failures - - Check automated labels are correct - -2. **Code review:** - - Check code quality - - Verify tests added/updated - - Review documentation changes - - Check for breaking changes - -3. **Approval:** - - Approve if satisfactory - - Request changes if needed - - Comment for discussion - -4. **Merging:** - - Use **Squash and merge** (default) - - Edit commit message if needed - - Ensure linear history maintained - -#### Managing Issues - -1. **Triage:** - - Review new issues daily - - Add appropriate labels - - Assign to team members - - Link to milestones if applicable - -2. **Label management:** - - Add priority labels - - Add type labels - - Add component labels - - Use good-first-issue for newcomers - -3. **Close/archive:** - - Close resolved issues - - Close duplicates with reference - - Let stale bot handle inactive issues - ---- - -## 🔍 Monitoring - -### Check Workflow Status - -```bash -# View recent workflow runs -gh run list - -# View specific workflow -gh run view - -# Watch workflow in real-time -gh run watch -``` - -### Common Issues - -#### CI Failing on PR - -1. **Check logs:** - ```bash - gh run view --log - ``` - -2. **Common causes:** - - Tests failing - - Clippy warnings - - Formatting issues - - Security vulnerabilities - -3. **Fix locally:** - ```bash - cargo test - cargo clippy --fix - cargo fmt - cargo audit - ``` - -#### PR Checks Not Running - -1. **Verify triggers:** - - Check workflow `on:` conditions - - Ensure PR targets correct branch - -2. **Re-run checks:** - - Push new commit - - Or use GitHub UI to re-run - -3. **Check permissions:** - - Verify Actions enabled - - Check workflow permissions - ---- - -## 📚 Additional Resources - -### Documentation -- [GitHub Actions Documentation](https://docs.github.com/en/actions) -- [Branch Protection Rules](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches) -- [CODEOWNERS Documentation](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners) - -### Project Documentation -- [Contributing Guide](../docs/CONTRIBUTING.md) -- [Security Policy](../SECURITY.md) -- [Code of Conduct](../CODE_OF_CONDUCT.md) - -### Support -- [Open an Issue](https://github.com/rahulbsw/streamforge/issues/new/choose) -- [Discussions](https://github.com/rahulbsw/streamforge/discussions) -- [Support Guide](../SUPPORT.md) - ---- - -## 🔄 Maintenance - -### Regular Tasks - -- **Weekly:** Review open PRs and issues -- **Monthly:** Review stale items -- **Quarterly:** Update workflows and check for Action updates -- **Annually:** Review and update branch protection rules - -### Updating Workflows - -1. **Test changes:** - - Create branch - - Modify workflow - - Test on PR before merging - -2. **Version updates:** - - Keep Action versions current - - Test after updating - - Check for breaking changes - -3. **Monitor:** - - Check workflow success rates - - Review execution times - - Optimize slow jobs - ---- - -**Last Updated:** April 2026 -**Maintainer:** @rahulbsw -**Questions:** See [SUPPORT.md](../SUPPORT.md) +- Keep this directory focused on GitHub behavior, not product positioning. +- If public messaging changes, update the root `README.md` and `docs/` first. +- If GitHub Pages output changes, review `.github/workflows/pages.yml`. diff --git a/docs/USAGE.md b/docs/USAGE.md index 43c3c1a..47e6a8c 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -5,698 +5,142 @@ nav_order: 3 # Usage Guide -Complete guide covering various use cases for StreamForge implementation. +StreamForge is strongest when you want to move a selected subset of Kafka data into downstream systems that need a different shape, a lower-trust payload, or a different topic layout. -## Table of Contents +This guide focuses on the high-value usage patterns that match the current product position: selective replication, shaping, redaction, and routing. It does not treat StreamForge as a full active-active mirroring tool or a general stream processing engine. -- [Basic Usage](#basic-usage) -- [Use Cases](#use-cases) - - [Simple Cross-Cluster Mirroring](#use-case-1-simple-cross-cluster-mirroring) - - [Content-Based Routing](#use-case-2-content-based-routing) - - [Data Validation Pipeline](#use-case-3-data-validation-pipeline) - - [Multi-Environment Deployment](#use-case-4-multi-environment-deployment) - - [Event Streaming Platform](#use-case-5-event-streaming-platform) - - [Data Lake Ingestion](#use-case-6-data-lake-ingestion) - - [Real-time Analytics](#use-case-7-real-time-analytics) - - [Microservices Integration](#use-case-8-microservices-integration) -- [Configuration Patterns](#configuration-patterns) -- [Troubleshooting](#troubleshooting) +## Start with a Validated Example -## Basic Usage +If you want a working starting point instead of building from scratch: -### Installation +- [QUICKSTART.md](QUICKSTART.md) for the five-minute local demo +- [EXAMPLES.md](EXAMPLES.md) for validated example packs +- [YAML_CONFIGURATION.md](YAML_CONFIGURATION.md) for the configuration structure +- [ADVANCED_DSL_GUIDE.md](ADVANCED_DSL_GUIDE.md) for filter and transform expressions -```bash -# Clone the repository -git clone -cd streamforge +## Core Usage Patterns -# Build the application -cargo build --release +### Filtered Replication to Analytics or Lake -# Binary location -./target/release/streamforge -``` +Use StreamForge when an operational topic contains more fields than downstream analytics systems should receive. -### Running +Typical flow: +- consume an application topic +- filter to the event types you actually want downstream +- project only the analytics-safe fields +- publish to a dedicated analytics topic or cluster -```bash -# Using default config location -CONFIG_FILE=config.json ./target/release/streamforge +Good fit: +- CDC or event streams feeding a warehouse or lake +- domain event cleanup before analytics ingestion +- splitting operational and analytical contracts -# With logging -RUST_LOG=info CONFIG_FILE=config.json ./target/release/streamforge - -# With debug logging -RUST_LOG=debug CONFIG_FILE=config.json ./target/release/streamforge -``` +See: +- [examples/production/cdc-to-datalake.yaml](../examples/production/cdc-to-datalake.yaml) +- [QUICKSTART.md](QUICKSTART.md) -### Docker +### PII-Safe Replication Across Trust Boundaries -```bash -# Build image -docker build -t streamforge:latest . - -# Run with config -docker run -d \ - --name mirrormaker \ - -v $(pwd)/config.json:/app/config/config.json:ro \ - -e RUST_LOG=info \ - streamforge:latest -``` +Use StreamForge when the destination system should receive business events but not raw identifiers or sensitive fields. -## Use Cases +Typical flow: +- keep only approved fields +- hash, mask, or drop sensitive values +- publish to a downstream topic with a tighter contract -### Use Case 1: Simple Cross-Cluster Mirroring +Good fit: +- staging or partner environments +- lower-trust analytics consumers +- internal topics that should not expose raw customer data -**Scenario**: Mirror all messages from a topic in Cluster A to Cluster B without modification. +See: +- [examples/production/pii-redaction.yaml](../examples/production/pii-redaction.yaml) +- [SECURITY_CONFIGURATION.md](SECURITY_CONFIGURATION.md) -**Configuration:** +### Topic Fan-Out with Consumer-Specific Shapes -```json -{ - "appid": "simple-mirror", - "bootstrap": "cluster-a:9092", - "target_broker": "cluster-b:9092", - "input": "source-topic", - "output": "destination-topic", - "offset": "latest", - "threads": 4, - "compression": { - "compression_type": "raw", - "compression_algo": "gzip" - } -} -``` +Use one input topic with multiple outputs when different consumers need different subsets or payload shapes. -**Use When:** -- Disaster recovery (DR) setup -- Data center replication -- Cluster migration -- Backup/archive purposes - -**Performance Tuning:** -```json -{ - "threads": 8, - "consumer_properties": { - "fetch.min.bytes": "1048576", - "fetch.wait.max.ms": "500" - }, - "producer_properties": { - "batch.size": "65536", - "linger.ms": "10", - "compression.type": "gzip" - } -} -``` +Typical flow: +- read one source topic +- apply per-destination filters +- publish consumer-specific payloads to separate topics -### Use Case 2: Content-Based Routing - -**Scenario**: Route messages to different topics based on content (e.g., route by event type, user tier, or region). - -**Configuration:** - -```json -{ - "appid": "content-router", - "bootstrap": "kafka:9092", - "target_broker": "kafka:9092", - "input": "events", - "routing": { - "routing_type": "content", - "path": "/eventType", - "destinations": [ - { - "name": "user-events", - "output": "users", - "filter": "REGEX:/eventType,^user\\.", - "transform": "/payload", - "partition": "/userId" - }, - { - "name": "order-events", - "output": "orders", - "filter": "REGEX:/eventType,^order\\.", - "transform": "CONSTRUCT:orderId=/orderId:amount=/amount:status=/status", - "partition": "/orderId" - }, - { - "name": "payment-events", - "output": "payments", - "filter": "REGEX:/eventType,^payment\\.", - "transform": "/payload", - "partition": "/paymentId" - }, - { - "name": "audit-all", - "output": "audit-log", - "transform": "CONSTRUCT:eventType=/eventType:timestamp=/timestamp:userId=/userId" - } - ] - } -} -``` +Good fit: +- one source topic feeding operations, analytics, and audits +- service-specific integration topics +- separating high-volume raw streams from narrower downstream contracts -**Use When:** -- Event-driven architectures -- Domain-specific topic routing -- Service-to-service communication -- Multi-tenant applications - -### Use Case 3: Data Validation Pipeline - -**Scenario**: Validate incoming data and route to valid/invalid topics for processing or DLQ. - -**Configuration:** - -```json -{ - "appid": "validator", - "bootstrap": "kafka:9092", - "input": "raw-data", - "routing": { - "routing_type": "content", - "destinations": [ - { - "name": "valid-emails", - "output": "validated-users", - "filter": "AND:REGEX:/email,^[\\w\\.-]+@[\\w\\.-]+\\.\\w{2,}$:/name,!=,:/age,>,0", - "transform": "CONSTRUCT:id=/id:email=/email:name=/name:age=/age", - "partition": "/id" - }, - { - "name": "invalid-email-format", - "output": "validation-errors", - "filter": "NOT:REGEX:/email,^[\\w\\.-]+@[\\w\\.-]+\\.\\w{2,}$", - "transform": "CONSTRUCT:error=email_format_invalid:record=/", - "partition": "/id" - }, - { - "name": "missing-required-fields", - "output": "validation-errors", - "filter": "OR:/name,==,:/email,==,:/age,<=,0", - "transform": "CONSTRUCT:error=missing_required_fields:record=/", - "partition": "/id" - } - ] - } -} -``` +### Cross-Cluster Replication with Shaping -**Use When:** -- Data quality enforcement -- ETL pipelines -- Input sanitization -- Compliance checking - -### Use Case 4: Multi-Environment Deployment - -**Scenario**: Mirror production data to staging/testing environments with data masking. - -**Configuration:** - -```json -{ - "appid": "prod-to-staging", - "bootstrap": "prod-kafka:9092", - "target_broker": "staging-kafka:9092", - "input": "production-events", - "routing": { - "routing_type": "content", - "destinations": [ - { - "name": "staging-safe-data", - "output": "staging-events", - "filter": "NOT:/sensitive,==,true", - "transform": "CONSTRUCT:id=/id:type=/type:timestamp=/timestamp:data=/nonSensitiveData", - "partition": "/id" - }, - { - "name": "test-sample", - "output": "test-events", - "filter": "AND:NOT:/sensitive,==,true:/testFlag,==,true", - "transform": "/", - "partition": "/id" - } - ] - } -} -``` +Use StreamForge when you need to move data between clusters but do not want to mirror whole topics unchanged. -**Use When:** -- Testing with production-like data -- Development environment setup -- QA validation -- Performance testing - -### Use Case 5: Event Streaming Platform - -**Scenario**: Build a central event bus that distributes events to multiple downstream consumers. - -**Configuration:** - -```json -{ - "appid": "event-bus", - "bootstrap": "central-kafka:9092", - "input": "event-stream", - "routing": { - "routing_type": "content", - "destinations": [ - { - "name": "analytics", - "output": "analytics-events", - "filter": "ARRAY_ANY:/tags,/value,==,analytics", - "transform": "CONSTRUCT:eventId=/eventId:type=/type:userId=/userId:timestamp=/timestamp:metrics=/metrics" - }, - { - "name": "notifications", - "output": "notification-queue", - "filter": "OR:/priority,==,high:/priority,==,urgent", - "transform": "CONSTRUCT:userId=/userId:message=/message:type=/notificationType" - }, - { - "name": "billing", - "output": "billing-events", - "filter": "REGEX:/type,^(usage|subscription|payment)", - "transform": "CONSTRUCT:userId=/userId:amount=/amount:type=/type:timestamp=/timestamp" - }, - { - "name": "audit", - "output": "audit-trail", - "filter": "REGEX:/action,^(create|update|delete)", - "transform": "/" - }, - { - "name": "ml-features", - "output": "ml-training", - "filter": "/mlRelevant,==,true", - "transform": "ARRAY_MAP:/features,/value" - } - ] - } -} -``` +Good fit: +- regional or environment replication with filtering +- topic migrations where downstream contracts are changing +- Redpanda or Kafka targets that only need a selected portion of the source stream -**Use When:** -- Event-driven microservices -- Real-time data distribution -- CQRS pattern implementation -- Event sourcing - -### Use Case 6: Data Lake Ingestion - -**Scenario**: Ingest data into data lake while maintaining real-time processing streams. - -**Configuration:** - -```json -{ - "appid": "data-lake-ingestion", - "bootstrap": "kafka:9092", - "input": "application-events", - "routing": { - "routing_type": "content", - "destinations": [ - { - "name": "raw-archive", - "output": "datalake-raw", - "transform": "/", - "comment": "Archive everything" - }, - { - "name": "processed-metrics", - "output": "datalake-metrics", - "filter": "/metrics,>,0", - "transform": "CONSTRUCT:timestamp=/timestamp:source=/source:metrics=/metrics" - }, - { - "name": "realtime-high-priority", - "output": "realtime-processing", - "filter": "AND:/priority,==,high:/processingTime,<,1000", - "transform": "CONSTRUCT:id=/id:priority=/priority:data=/data" - }, - { - "name": "anomaly-detection", - "output": "anomaly-queue", - "filter": "OR:/errorRate,>,0.1:/responseTime,>,5000", - "transform": "CONSTRUCT:source=/source:metric=/metricName:value=/metricValue:threshold=/threshold" - } - ] - } -} -``` +See: +- [COMPATIBILITY.md](COMPATIBILITY.md) +- [examples/redpanda/README.md](../examples/redpanda/README.md) -**Use When:** -- Big data analytics -- Long-term storage -- Historical analysis -- Compliance/retention - -### Use Case 7: Real-time Analytics - -**Scenario**: Calculate metrics and route aggregated data for real-time dashboards. - -**Configuration:** - -```json -{ - "appid": "realtime-analytics", - "bootstrap": "kafka:9092", - "input": "raw-metrics", - "routing": { - "routing_type": "content", - "destinations": [ - { - "name": "revenue-total", - "output": "revenue-metrics", - "filter": "/order/status,==,completed", - "transform": "ARITHMETIC:ADD,/order/amount,/order/tax", - "partition": "/order/customerId" - }, - { - "name": "conversion-rate", - "output": "conversion-metrics", - "filter": "AND:/visits,>,0:/conversions,>,0", - "transform": "ARITHMETIC:DIV,/conversions,/visits", - "partition": "/campaignId" - }, - { - "name": "user-engagement", - "output": "engagement-metrics", - "filter": "ARRAY_ANY:/sessions,/duration,>,300", - "transform": "CONSTRUCT:userId=/userId:sessions=ARRAY_MAP:/sessions,/duration", - "partition": "/userId" - }, - { - "name": "error-rates", - "output": "error-metrics", - "filter": "AND:/requests,>,0:/errors,>,0", - "transform": "ARITHMETIC:DIV,/errors,/requests", - "partition": "/serviceId" - } - ] - } -} -``` +## Build a Pipeline -**Use When:** -- Real-time dashboards -- KPI monitoring -- Business intelligence -- Alerting systems - -### Use Case 8: Microservices Integration - -**Scenario**: Connect multiple microservices through event-driven communication. - -**Configuration:** - -```json -{ - "appid": "microservices-hub", - "bootstrap": "kafka:9092", - "input": "service-events", - "routing": { - "routing_type": "content", - "destinations": [ - { - "name": "user-service", - "output": "user-service-events", - "filter": "REGEX:/aggregate,^User", - "transform": "CONSTRUCT:aggregateId=/aggregateId:eventType=/eventType:payload=/payload", - "partition": "/aggregateId" - }, - { - "name": "order-service", - "output": "order-service-events", - "filter": "REGEX:/aggregate,^Order", - "transform": "CONSTRUCT:aggregateId=/aggregateId:eventType=/eventType:payload=/payload", - "partition": "/aggregateId" - }, - { - "name": "inventory-service", - "output": "inventory-service-events", - "filter": "REGEX:/aggregate,^(Inventory|Product)", - "transform": "CONSTRUCT:aggregateId=/aggregateId:eventType=/eventType:payload=/payload", - "partition": "/aggregateId" - }, - { - "name": "notification-service", - "output": "notification-events", - "filter": "ARRAY_ANY:/tags,/value,==,notify", - "transform": "CONSTRUCT:userId=/userId:message=/message:channels=ARRAY_MAP:/channels,/type" - }, - { - "name": "cross-service-saga", - "output": "saga-coordinator", - "filter": "/sagaId,!=,", - "transform": "CONSTRUCT:sagaId=/sagaId:step=/step:status=/status:data=/data" - } - ] - } -} -``` +### 1. Choose the Source and Destinations -**Use When:** -- Event-driven microservices -- Saga pattern -- Domain events -- Service choreography - -## Configuration Patterns - -### Pattern 1: Broadcast with Transformation - -Send the same message to multiple topics with different transformations: - -```json -{ - "destinations": [ - { - "name": "full-archive", - "output": "archive", - "transform": "/" - }, - { - "name": "minimal-log", - "output": "logs", - "transform": "CONSTRUCT:id=/id:timestamp=/timestamp:level=/level" - }, - { - "name": "error-only", - "output": "errors", - "filter": "REGEX:/level,^(ERROR|FATAL)", - "transform": "CONSTRUCT:id=/id:error=/error:stackTrace=/stackTrace" - } - ] -} -``` +Define the input topic and decide whether you are publishing to: +- one destination topic +- multiple destination topics in the same cluster +- a different Kafka-compatible target cluster -### Pattern 2: Conditional Routing - -Route based on complex conditions: - -```json -{ - "destinations": [ - { - "name": "premium-fast-lane", - "output": "premium-queue", - "filter": "AND:/user/tier,==,premium:/priority,==,high", - "transform": "/" - }, - { - "name": "standard-processing", - "output": "standard-queue", - "filter": "NOT:AND:/user/tier,==,premium:/priority,==,high", - "transform": "/" - } - ] -} -``` +### 2. Add Selection Logic -### Pattern 3: Data Enrichment +Use filters when only part of the source stream should move downstream. -Add calculated fields: +Common selectors: +- event type +- region or tenant +- presence or value of a field +- metadata in key or headers -```json -{ - "destinations": [ - { - "name": "with-total", - "output": "enriched-orders", - "transform": "CONSTRUCT:orderId=/orderId:subtotal=/subtotal:tax=ARITHMETIC:MUL,/subtotal,0.08:total=ARITHMETIC:MUL,/subtotal,1.08" - } - ] -} -``` +### 3. Shape the Payload -Note: Nested transforms not yet supported; apply sequentially. - -### Pattern 4: Filter Pipeline - -Progressive filtering: - -```json -{ - "destinations": [ - { - "name": "stage1-valid-format", - "output": "stage1", - "filter": "REGEX:/email,^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$" - }, - { - "name": "stage2-corporate", - "output": "stage2", - "filter": "AND:REGEX:/email,^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$:REGEX:/email,@company\\.com$" - }, - { - "name": "stage3-active", - "output": "stage3", - "filter": "AND:REGEX:/email,^[\\w\\.-]+@[\\w\\.-]+\\.\\w+$:REGEX:/email,@company\\.com$:/active,==,true" - } - ] -} -``` +Use transforms to: +- keep only downstream-safe fields +- rename or restructure fields +- construct smaller consumer-specific payloads +- hash or drop sensitive data -## Troubleshooting +### 4. Validate Before Running -### Common Issues +Prefer validated YAML configs over ad hoc inline examples. -#### 1. Messages Not Being Routed - -**Symptoms**: Messages consumed but not sent to any destination. - -**Check:** ```bash -# Enable debug logging -RUST_LOG=debug CONFIG_FILE=config.json ./streamforge -``` - -**Common Causes:** -- Filter not matching (check filter logic) -- Transform error (check field paths) -- All destinations filtered out - -**Solution:** -```json -{ - "destinations": [ - { - "name": "catchall", - "output": "unrouted", - "comment": "No filter = accepts all" - } - ] -} -``` - -#### 2. Performance Issues - -**Symptoms**: Low throughput, high latency. - -**Check:** -- Thread count -- Batch size -- Network latency - -**Solution:** -```json -{ - "threads": 8, - "consumer_properties": { - "fetch.min.bytes": "1048576", - "max.poll.records": "500" - }, - "producer_properties": { - "batch.size": "65536", - "linger.ms": "10" - } -} +cargo run --quiet --bin streamforge-validate -- path/to/config.yaml ``` -#### 3. Memory Usage High - -**Symptoms**: Process uses excessive memory. - -**Check:** -- Array sizes in messages -- Number of destinations -- Batch sizes - -**Solution:** -```json -{ - "consumer_properties": { - "max.poll.records": "100", - "fetch.max.bytes": "52428800" - } -} -``` - -#### 4. Connection Failures - -**Symptoms**: Cannot connect to Kafka. - -**Check:** -```bash -# Test connectivity -nc -zv kafka-broker 9092 - -# Check DNS -nslookup kafka-broker - -# Check config -cat config.json | grep bootstrap -``` - -**Solution:** -- Verify bootstrap servers -- Check network/firewall rules -- Verify authentication config - -#### 5. Filter Not Working - -**Symptoms**: Expected messages not matching filter. - -**Debug:** -```json -{ - "destinations": [ - { - "name": "debug-all", - "output": "debug-topic", - "transform": "/", - "comment": "Send all to debug topic" - } - ] -} -``` +### 5. Deploy in the Right Mode -**Check:** -- Field paths (case-sensitive) -- Value types (string vs number) -- Regex escaping (`\\` for special chars) +Use the standalone binary when you want the lightest operational path. Use the operator and Helm chart when you want pipelines managed as Kubernetes resources. -### Debugging Tips +See: +- [DEPLOYMENT.md](DEPLOYMENT.md) +- [KUBERNETES.md](KUBERNETES.md) -1. **Start Simple**: Begin with no filter, add complexity gradually -2. **Use Debug Logs**: `RUST_LOG=debug` shows filter evaluation -3. **Test Regex Separately**: Use online regex testers -4. **Validate JSON Paths**: Check field names and nesting -5. **Monitor Metrics**: Watch filtered vs completed counts +## When StreamForge Is the Wrong Tool -### Getting Help +Do not use StreamForge as your primary answer for: +- MirrorMaker 2 active-active replication +- consumer offset synchronization across clusters +- general SQL stream processing +- joins and broad stateful event computation -- Check `ADVANCED_DSL_GUIDE.md` for DSL syntax -- See `PERFORMANCE.md` for tuning tips -- Review example configs in `config*.example.json` -- Enable debug logging for detailed troubleshooting +That boundary is intentional. StreamForge is the selective replication and shaping layer. Heavier stateful analytics belongs in tools built for that purpose. -## Next Steps +## Recommended Reading Order -- [PERFORMANCE.md](PERFORMANCE.md) - Performance optimization -- [CONTRIBUTING.md](CONTRIBUTING.md) - Contributing guide -- [ADVANCED_DSL_GUIDE.md](ADVANCED_DSL_GUIDE.md) - Complete DSL reference +1. [QUICKSTART.md](QUICKSTART.md) +2. [EXAMPLES.md](EXAMPLES.md) +3. [YAML_CONFIGURATION.md](YAML_CONFIGURATION.md) +4. [ADVANCED_DSL_GUIDE.md](ADVANCED_DSL_GUIDE.md) +5. [COMPATIBILITY.md](COMPATIBILITY.md) diff --git a/docs/_config.yml b/docs/_config.yml index 13c11ab..5bcc88c 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -12,9 +12,36 @@ render_with_liquid: false # Exclude non-documentation files exclude: - "*.sh" - - "development/internal" + - "archive" + - "benchmarks" + - "development" + - "superpowers" + - "CONFIG_SCHEMA.json" - Gemfile - Gemfile.lock + - "ADVANCED_FILTERS.md" + - "AT_LEAST_ONCE_AND_CACHE_BACKENDS.md" + - "DSL_ARCHITECTURE.md" + - "DSL_FEATURES.md" + - "DSL_SIMPLIFICATION_INVENTORY.md" + - "DSL_SPEC.md" + - "DSL_V2_FUNCTION_SYNTAX.md" + - "ENVELOPE_FEATURE_DESIGN.md" + - "ENVELOPE_MIGRATION_GUIDE.md" + - "ERROR_HANDLING.md" + - "FUNCTION_STYLE_DSL_IMPLEMENTATION.md" + - "HASH_AND_CACHE.md" + - "IMPLEMENTATION_NOTES.md" + - "IMPLEMENTATION_STATUS.md" + - "OBSERVABILITY_IMPLEMENTATION_SUMMARY.md" + - "OBSERVABILITY_METRICS_DESIGN.md" + - "PARSER_REFACTOR_PLAN.md" + - "PERFORMANCE_TESTING.md" + - "PROJECT_SUMMARY.md" + - "QUICK_REFERENCE.md" + - "TEST_COVERAGE.md" + - "TYPED_ENVELOPE_DESIGN.md" + - "V1_GUARANTEES.md" # just-the-docs configuration color_scheme: dark # options: default (light), dark @@ -22,6 +49,41 @@ color_scheme: dark # options: default (light), dark # Navigation nav_sort: case_insensitive +defaults: + - scope: + path: "USAGE.md" + values: + has_children: true + - scope: + path: "ADVANCED_DSL_GUIDE.md" + values: + parent: "Usage Guide" + - scope: + path: "YAML_CONFIGURATION.md" + values: + parent: "Usage Guide" + - scope: + path: "DEPLOYMENT.md" + values: + title: Deployment + nav_order: 7 + has_children: true + - scope: + path: "DELIVERY_GUARANTEES.md" + values: + title: Delivery Guarantees + nav_order: 8 + - scope: + path: "OPERATIONS.md" + values: + title: Operations Runbook + parent: "Deployment" + - scope: + path: "TROUBLESHOOTING.md" + values: + title: Troubleshooting + parent: "Deployment" + # Search search_enabled: true search_tokenizer_separator: /[\s/]+/